CN113420006A - Data migration method and device, electronic equipment and computer readable medium - Google Patents
Data migration method and device, electronic equipment and computer readable medium Download PDFInfo
- Publication number
- CN113420006A CN113420006A CN202010725946.XA CN202010725946A CN113420006A CN 113420006 A CN113420006 A CN 113420006A CN 202010725946 A CN202010725946 A CN 202010725946A CN 113420006 A CN113420006 A CN 113420006A
- Authority
- CN
- China
- Prior art keywords
- data
- migrated
- relational database
- migration
- fragment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application provides a data migration method and device, electronic equipment and a computer readable medium, and relates to the field of databases. Wherein the method comprises the following steps: determining data fragments to be migrated in a relational database instance in a distributed relational database; determining a target relational database instance used for storing user data in the data fragment to be migrated in the distributed relational database; generating a migration task for migrating the user data in the data fragment to be migrated based on the data fragment to be migrated and the target relational database instance; and submitting the migration task to the data migration system, so that the data migration system migrates the user data in the data fragment to be migrated to the target relational database instance based on the migration task. By the embodiment of the application, the distributed relational database has the capability of overall transverse expansion.
Description
Technical Field
The embodiment of the application relates to the field of databases, in particular to a data migration method, a data migration device, electronic equipment and a computer readable medium.
Background
In recent years, with the rapid increase of data volume, distributed database technology has been rapidly developed, and the traditional relational database is beginning to develop from centralized type to distributed type. Distributed databases refer in particular to a logically unified database formed by connecting physically distributed database instances using a high-speed computer network. The basic idea of the distributed database is to store user data in the original centralized database to a plurality of database instances connected through a network in a scattered manner, so as to obtain larger storage capacity and higher concurrent access amount.
In the running process of a distributed Relational Database (distributed Relational Database), along with the increase of the data scale of a user, a Relational Database instance where a single data fragment is located can reach the upper limit of the capacity, so that the capacity bottleneck occurs to a rapidly growing user. For distributed relational databases, the storage of user data is one of the key technologies. Therefore, how to effectively expand the data storage capacity of the distributed relational database becomes a technical problem to be solved urgently at present.
Disclosure of Invention
The present application aims to provide a data migration method, an apparatus, an electronic device, and a computer-readable medium, which are used to solve the technical problem in the prior art that how to effectively extend the data storage capacity of a distributed relational database is currently in urgent need of solution.
According to a first aspect of embodiments of the present application, a data migration method is provided. The method comprises the following steps: determining data fragments to be migrated in a relational database instance in a distributed relational database; determining a target relational database instance used for storing user data in the data fragment to be migrated in the distributed relational database; generating a migration task for migrating the user data in the data fragment to be migrated based on the data fragment to be migrated and the target relational database instance; and submitting the migration task to the data migration system, so that the data migration system migrates the user data in the data fragment to be migrated to the target relational database instance based on the migration task.
According to a second aspect of embodiments of the present application, there is provided a data migration apparatus. The device comprises: the system comprises a first determining module, a second determining module and a third determining module, wherein the first determining module is used for determining data fragments to be migrated in a relational database instance in a distributed relational database; a second determining module, configured to determine a target relational database instance in the distributed relational database, where the target relational database instance is used to store the user data in the data segment to be migrated; a generating module, configured to generate a migration task for migrating user data in the data shards to be migrated, based on the data shards to be migrated and the target relational database instance; and the submitting module is used for submitting the migration task to the data migration system, so that the data migration system migrates the user data in the data fragment to be migrated to the target relational database instance based on the migration task.
According to a third aspect of embodiments of the present application, there is provided an electronic apparatus, including: one or more processors; a computer readable medium configured to store one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the data migration method as described in the first aspect of the embodiments above.
According to a fourth aspect of embodiments of the present application, there is provided a computer-readable medium, on which a computer program is stored, which when executed by a processor, implements the data migration method as described in the first aspect of the embodiments above.
By the data migration scheme provided by the embodiment of the application, the data fragment to be migrated in the relational database instance in the distributed relational database is determined, the target relational database instance for storing the user data in the data fragment to be migrated in the distributed relational database is determined, the migration task for migrating the user data in the data fragment to be migrated is generated based on the data fragment to be migrated and the target relational database instance, and then the migration task is submitted to the data migration system, so that the data migration system migrates the user data in the data fragment to be migrated to the target relational database instance based on the migration task, compared with the existing other modes, the migration task for migrating the user data in the data fragment to be migrated is generated based on the data fragment to be migrated and the target relational database instance, and submitting the migration task to a data migration system, so that the data migration system migrates the user data in the data fragments to be migrated to the target relational database instance based on the migration task, and on the premise of minimizing the influence on the user data, the smooth migration of the user data in the data fragments can be effectively ensured, and the distributed relational database can have the capability of overall transverse expansion.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:
FIG. 1A is a flowchart illustrating steps of a data migration method according to an embodiment of the present disclosure;
FIG. 1B is a diagram illustrating a data slice migration process according to an embodiment of the present disclosure;
fig. 1C is a schematic diagram of a data slicing switching process according to an embodiment of the present application;
FIG. 1D is a diagram illustrating a data slicing and cleaning process according to an embodiment of the present disclosure;
fig. 1E is a schematic diagram of a data slice whole migration process according to an embodiment of the present application;
FIG. 2A is a flowchart illustrating steps of a data migration method according to a second embodiment of the present application;
fig. 2B is a schematic diagram of a data slicing rollback process according to the second embodiment of the present application;
FIG. 3 is a schematic structural diagram of a data migration apparatus according to a third embodiment of the present application;
FIG. 4 is a schematic structural diagram of a data migration apparatus according to a fourth embodiment of the present application;
FIG. 5 is a schematic structural diagram of a data migration apparatus according to a fifth embodiment of the present application;
fig. 6 is a schematic structural diagram of an electronic device according to a sixth embodiment of the present application;
fig. 7 is a hardware structure of an electronic device according to a seventh embodiment of the present application.
Detailed Description
The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings.
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
Referring to fig. 1A, a flowchart of steps of a data migration method according to a first embodiment of the present application is shown.
Specifically, the data migration method provided in this embodiment includes the following steps:
in step S101, a data slice to be migrated in a relational database instance in a distributed relational database is determined.
In the embodiment of the present application, a relational database may be understood as a database that organizes data by using a relational model, and stores data in the form of rows and columns so as to be easily understood by a user. A user retrieves data in a database by a query, which is an executable code that defines certain areas in the database. The relational model can be simply understood as a two-dimensional table model, and a relational database is a data organization composed of two-dimensional tables and relations between them. The distributed relational database may be understood to be a middleware-based distributed relational database system service provided. The distributed relational database can be used for constructing a database system supporting mass data storage and access based on a transverse expansion mode of a common server, so that infinite capacity expansion and elastic expansion are realized. Compared with a single-machine relational database, the distributed relational database provides a higher-specification storage query rate, and meets the continuously-increased mass data storage requirement of a user and the continuously-increased user request pressure. In addition, a plurality of relational database instances may be deployed on the database server (physical node) of the distributed relational database, and a plurality of relational databases (sub-databases) may be deployed on the relational database instances, where a relational database is a logical organization form of a relational database table, and one relational database may include a plurality of tables. A data table may be understood as a data storage unit. The data slice to be migrated may be understood as a data slice of user data to be migrated. The data shards may be understood as a vault in a relational database instance. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In some optional embodiments, when determining a data shard to be migrated in a relational database instance in a distributed relational database, the list of data shards to be migrated is generated based on a data capacity of the data shard in the relational database instance. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In a specific example, when the list of the data shards to be migrated is generated based on the data capacity of the data shards in the relational database instance, when the data capacity of the data shards in the relational database instance is greater than a preset data capacity threshold, determining that the data shards are the data shards to be migrated; and when the data capacity of the data fragment in the relational database instance is smaller than or equal to a preset data capacity threshold value, determining that the data fragment is not the data fragment to be migrated. The preset data capacity threshold may be set by a person skilled in the art according to actual needs, and the embodiment of the present application is not limited in any way. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In some optional embodiments, after generating the list of data shards to be migrated, the method further comprises: receiving an adjustment operation aiming at the list of the data fragments to be migrated; and adjusting the data fragments to be migrated in the list based on the adjusting operation. Therefore, the data fragments to be migrated in the list can be dynamically adjusted according to requirements through the adjustment operation. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In a specific example, a user clicks a data fragmentation adjustment control on a DRDS (Distributed Relational Database Services) console, the DRDS console sends a data fragmentation adjustment request to a DRDS management and control server, and the DRDS management and control server adjusts data fragments to be migrated in a list based on the data fragmentation adjustment request. The DRDS control server is a server added between the console and the relational database instances, and mainly provides the capability of relational database fragmentation routing for the console, SQL instructions input by users can be routed to one of the relational database instances according to a fragmentation algorithm of the DRDS control server, and the users can conveniently manage and operate a plurality of relational database instances through the DRDS control server. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In step S102, a target relational database instance in the distributed relational database for storing the user data in the data segment to be migrated is determined.
In the embodiment of the present application, the user data may be commodity order data, commodity transaction data, and the like. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In some optional embodiments, upon determining a target relational database instance in the distributed relational database for storing user data in the data shards to be migrated, receiving a selection operation for the target relational database instance; selecting the target relational database instance based on the selecting operation. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In a specific example, after the user inputs the number and the identification information of the target relational database instances on the DRDS console, the user clicks a selection control of the target relational database instances, the DRDS console sends a selection request to the DRDS management and control server, and the DRDS management and control server selects the target relational database instances based on the selection request. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In step S103, a migration task for migrating user data in the to-be-migrated data segment is generated based on the to-be-migrated data segment and the target relational database instance.
In the embodiment of the application, when a migration task for migrating user data in the data fragment to be migrated is generated, a migration task for migrating user data in the data fragment to be migrated is generated based on the identification information of the data fragment to be migrated and the identification information of the target relational database instance. The identification information of the data fragment to be migrated may be a code or a name of the data fragment to be migrated, and the identification information of the target relational database instance may be a code or a name of the target relational database instance. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In step S104, submitting the migration task to the data migration system, so that the data migration system migrates the user data in the data segment to be migrated to the target relational database instance based on the migration task.
In the embodiments of the present application, the data migration system may be understood as a computer program system for data migration. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In a specific example, as shown in fig. 1B, a user selects a target relational database instance on the DRDS console and clicks a selection control on the DRDS console, the DRDS console sends a selection request to the DRDS management and control server, and the DRDS management and control server selects the target relational database instance based on the selection request. Then, the DRDS management and control server evaluates the data capacity of the data shards in the relational database instance to generate a list of the data shards to be migrated. After the list of the data fragments to be migrated is generated, the DRDS management and control server receives an adjustment request for the data fragments in the list of the data fragments to be migrated, which is sent by the DRDS console, and adjusts the data fragments to be migrated in the list based on the adjustment request. After the data fragments to be migrated in the list are adjusted, the DRDS management and control server generates a migration task based on the adjusted data fragments and the target relational database instance, and submits the migration task to the data migration system. And the data migration system migrates the user data in the data fragment to be migrated to the target data fragment in the target relational database instance based on the migration task. In the data migration process, a user can check the migration state of user data in the data fragment to be migrated through the DRDS console. In addition, in the process of migrating data, a bit of read pressure may be applied to the data fragments to be migrated, and at this time, the pressure may be controlled by adjusting the speed of migrating data by the data migration system. No impact is caused to the user data. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In some optional embodiments, after submitting the migration task to the data migration system, the method further comprises: receiving a migration state of user data in the data fragment to be migrated, which is sent by the data migration system; and displaying the migration state of the user data in the data fragment to be migrated. The migration status may be understood as a data migration progress, for example, that user data in the target data fragment already ties up user data in the data fragment to be migrated, that user data in the target data fragment does not tie up user data in the data fragment to be migrated, and the like. Therefore, the user can know the migration state of the user data in the data fragment to be migrated by showing the migration state of the user data in the data fragment to be migrated. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In some optional embodiments, after submitting the migration task to the data migration system, the method further comprises: when the migration state of the user data in the data fragment to be migrated is a tie-up state, receiving switching operation aiming at the data fragment to be migrated within a data low peak time period; and locking the distributed relational database based on the switching operation, and modifying the fragment metadata of the data fragment to be migrated during the locking period of the distributed relational database so as to switch the data fragment to be migrated into the target data fragment in the target relational database instance. Therefore, the fragment metadata of the data fragment to be migrated is modified during the database locking period of the distributed relational database, the data fragment to be migrated can be switched to the target data fragment in the target relational database instance, and the data consistency of the data fragment to be migrated after data migration can be further effectively ensured. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In a specific example, the tie-down state may be understood as a state in which user data in a target data fragment in a target relational database instance is the same as user data in a data fragment to be migrated, or a state in which user data in a target data fragment in a target relational database instance is close to the same as user data in a data fragment to be migrated. The data low-peak time period may be understood as a time period when the speed of writing the user data into the distributed relational database is less than a preset speed threshold, or a time period when the data amount of the user data written into the distributed relational database in unit time is less than a preset data amount threshold. The preset speed threshold and the preset data amount threshold may be set by those skilled in the art according to actual needs, and this is not limited in this embodiment of the present application. The lock bank operation may be understood as an operation that stops the writing of user data to the distributed relational database. The shard metadata may be understood as data that describes a mapping relationship between a data shard and a relational database instance in which the data shard is located. When the fragment metadata of the data fragment to be migrated is modified, modifying the data describing the mapping relationship between the data fragment to be migrated and the source relational database instance into the data describing the mapping relationship between the target data fragment and the target relational database instance. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In some optional embodiments, before modifying the shard metadata of the data shards to be migrated, the method further comprises: checking whether the incremental user data in the data fragment to be migrated and the incremental user data in the target data fragment are leveled; and if the incremental user data in the data fragment to be migrated is flush with the incremental user data in the target data fragment, executing the step of modifying the fragment metadata of the data fragment to be migrated. Therefore, when the incremental user data in the data fragment to be migrated and the incremental user data in the target data fragment are leveled, the step of modifying the fragment metadata of the data fragment to be migrated is executed, and the data consistency of the data fragment to be migrated after data migration can be further ensured. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In a specific example, when verifying whether incremental user data in the to-be-migrated data slice and incremental user data in the target data slice level, verifying whether incremental log data of a log file of the to-be-migrated data slice in a preset time period is the same as incremental log data of a log file of the target data slice in the preset time period, so as to verify whether incremental user data in the to-be-migrated data slice and incremental user data in the target data slice level, where the log file is used to record an SQL statement for updating user data in data slices. Specifically, the log file may be a log file in a binary format, and is used to record SQL statements for updating user data in the data fragment, for example, the SQL statements for changing the database table and the content may be recorded in the log file in the binary format, but the query for the content such as the database table may not be recorded. The log file in binary format is mainly used for master-slave copy of the relational database and incremental recovery of data. The log file (binlog) in binary format supports three data formats, the corresponding parameter is binlog _ format, the selectable values are ROW, STATEMENT and MIXED respectively, the corresponding default value is MIXED, and the ROW format needs to be adjusted. The data format is characterized in that: each added and deleted SQL statement records the values of the rows affected by the SQL statement before and after modification in a log file in binary format. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In a specific example, as shown in fig. 1C, a user views the migration status of user data in a data slice to be migrated through the DRDS console. When a user finds that the migration state of user data in the data fragments to be migrated is a leveling state through the DRDS console, the user selects to click a data fragment switching control at the DRDS console in a data low-peak time period, and the DRDS console sends a data fragment switching request to the DRDS management and control server based on data fragment switching operation. The DRDS management and control server locks the distributed relational database based on the data fragment switching request, checks whether incremental user data in the data fragments to be migrated and incremental user data in the target data fragments are leveled or not during the locking period of the distributed relational database, and modifies fragment metadata of the data fragments to be migrated to switch the data fragments to be migrated to the target data fragments in the target relational database example if the incremental user data in the data fragments to be migrated and the incremental user data in the target data fragments are leveled. After switching the data fragment to be migrated to the target data fragment in the target relational database instance, the user can check the switching state of the data fragment through the DRDS console. When the switching state of the data fragment is successful, the user can check that the physical topology of the distributed relational database has changed through the DRDS console, and the user flow is injected into the target data fragment in the target relational database instance. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In some optional embodiments, after switching the data shard to be migrated to a target data shard in the target relational database instance, the method further comprises: receiving a cleaning operation aiming at the data fragments to be migrated; and sending a data fragment cleaning request to the relational database instance where the data fragments to be migrated are located based on the cleaning operation, so that the relational database instance where the data fragments to be migrated are located cleans the data fragments to be migrated based on the data fragment cleaning request. Therefore, the data fragments to be migrated are cleared through the clearing operation, the whole capacity resource occupation ratio of the relational database instance where the data fragments to be migrated are located can be reduced, and the capacity resources occupied by the data fragments to be migrated are released. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In a specific example, as shown in fig. 1D, after switching the data fragment to be migrated to the target data fragment in the target relational database instance, the user may click a data fragment cleaning control on the DRDS console, and the DRDS console sends a data fragment cleaning request to the DRDS management and control server based on the data fragment cleaning operation. And the DRDS management and control server sends a data fragment cleaning request to the relational database instance where the data fragments to be migrated are located based on the data fragment switching request. And cleaning the data fragments to be migrated based on the data fragment cleaning request by the relational database example in which the data fragments to be migrated are positioned. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In a specific example, as shown in fig. 1E, the subbanks 3 and 4 in the relational database instance 1 are data fragments to be migrated, the relational database instance 2 is a target relational database instance, and the subbanks 3 and 4 in the relational database instance 2 are target data fragments. Specifically, according to the data migration method provided in the embodiment of the present application, the user data in the sub-library 3 and the user data in the sub-library 4 in the relational database instance 1 may be migrated to the sub-library 3 and the sub-library 4 in the relational database instance 2, respectively. After the data migration is completed, the mapping relation between the subbases 3 and 4 in the relational database instance 1 and the relational database instance 1 can be modified into the mapping relation between the subbases 3 and 4 in the relational database instance 2 and the relational database instance 2. After the modification is completed, the sub-libraries 3 and 4 in the relational database instance 1 can be cleaned up, so as to obtain the final result as shown in fig. 1E. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
By the data migration method provided by the embodiment of the application, the data fragment to be migrated in the relational database instance in the distributed relational database is determined, the target relational database instance for storing the user data in the data fragment to be migrated in the distributed relational database is determined, the migration task for migrating the user data in the data fragment to be migrated is generated based on the data fragment to be migrated and the target relational database instance, and then the migration task is submitted to the data migration system, so that the data migration system migrates the user data in the data fragment to be migrated to the target relational database instance based on the migration task, compared with the existing other methods, the migration task for migrating the user data in the data fragment to be migrated is generated based on the data fragment to be migrated and the target relational database instance, and submitting the migration task to a data migration system, so that the data migration system migrates the user data in the data fragments to be migrated to the target relational database instance based on the migration task, and on the premise of minimizing the influence on the user data, the smooth migration of the user data in the data fragments can be effectively ensured, and the distributed relational database can have the capability of overall transverse expansion.
The data migration method provided by the present embodiment may be executed by any suitable device with data processing capability, including but not limited to: a camera, a terminal, a mobile terminal, a PC, a server, an in-vehicle device, an entertainment device, an advertising device, a Personal Digital Assistant (PDA), a tablet computer, a notebook computer, a handheld game console, smart glasses, a smart watch, a wearable device, a virtual display device, a display enhancement device, or the like.
Referring to fig. 2A, a flowchart illustrating steps of a data migration method according to a second embodiment of the present application is shown.
Specifically, the data migration method provided in this embodiment includes the following steps:
in step S201, a data slice to be migrated in a relational database instance in the distributed relational database is determined.
Since the embodiment of step S201 is similar to that of step S101, it is not described herein again.
In step S202, a target relational database instance in the distributed relational database for storing the user data in the data slice to be migrated is determined.
Since the embodiment of step S202 is similar to that of step S102, it is not repeated herein.
In step S203, a migration task for migrating user data in the to-be-migrated data segment is generated based on the to-be-migrated data segment and the target relational database instance.
Since the embodiment of step S203 is similar to that of step S103, it is not described herein again.
In step S204, the migration task is submitted to the data migration system, so that the data migration system migrates the user data in the data segment to be migrated to the target relational database instance based on the migration task.
Since the embodiment of step S204 is similar to that of step S104, it is not repeated herein.
In step S205, a rollback operation for the migration task is received.
In the embodiment of the application, when the user selects the data fragment to be migrated incorrectly or selects the target relational database instance incorrectly, the user may click the migration task rollback control on the DRDS console, and the DRDS console may send the migration task cancellation request to the DRDS management and control server based on the migration task rollback operation. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In step S206, based on the rollback operation, a migration task cancellation request is sent to the data migration system, so that the data migration system cancels the migration task based on the migration task cancellation request.
In the embodiment of the application, the DRDS management and control server sends a migration task cancellation request to the data migration system based on the received migration task cancellation request, so that the data migration system cancels the migration task based on the migration task cancellation request. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In some optional embodiments, after sending a migration task cancellation request to the data migration system based on the rollback operation, the method further comprises: and sending a data fragment deleting request to the target relational database instance, so that the target relational database instance deletes the target data fragment in the target relational database instance based on the data fragment deleting request. Therefore, the target data fragments in the target relational database instance are deleted through the data fragment deleting request, and the data storage capacity in the distributed relational database can be expanded transversely again. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In a specific example, in the data fragment migration process, if the data fragment to be migrated is selected incorrectly or the target relational database instance is selected incorrectly, and the migration task is already submitted, rollback of the migration task may be selected at this time, but if the data fragment to be migrated is successfully switched, rollback of the migration task may not be performed, and only if the data fragment to be migrated is not successfully switched, rollback of the migration task may be performed, and the rollback process may delete the target data fragment in the target relational database instance. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
In a specific example, as shown in fig. 2B, if the data fragment to be migrated is selected incorrectly or the target relational database instance is selected incorrectly, and the migration task has been submitted, the user may click a migration task rollback control on the DRDS console, the DRDS console sends a migration task cancellation request to the DRDS management and control server based on the migration task rollback operation, the DRDS management and control server sends a migration task cancellation request to the data migration system based on the migration task cancellation request, and the data migration system cancels the migration task based on the migration task cancellation request. Then, the DRDS management and control server sends a data fragment deletion request to the target relational database instance, and the target relational database instance deletes the target data fragment in the target relational database instance based on the data fragment deletion request. After deleting the target data fragment in the target relational database instance, the user can check the rollback state of the migration task through the DRDS console. And the user sees that the whole migration task is cancelled on the DRDS console, and the data migration can be carried out again. It should be understood that the above description is only exemplary, and the embodiments of the present application are not limited in this respect.
On the basis of the first embodiment of the application, a rollback operation for a migration task is received, and a migration task cancellation request is sent to a data migration system based on the rollback operation, so that the data migration system cancels the migration task based on the migration task cancellation request.
The data migration method provided by the present embodiment may be executed by any suitable device with data processing capability, including but not limited to: a camera, a terminal, a mobile terminal, a PC, a server, an in-vehicle device, an entertainment device, an advertising device, a Personal Digital Assistant (PDA), a tablet computer, a notebook computer, a handheld game console, smart glasses, a smart watch, a wearable device, a virtual display device, a display enhancement device, or the like.
Referring to fig. 3, a schematic structural diagram of a data migration apparatus in a third embodiment of the present application is shown.
The data migration apparatus provided in this embodiment includes: a first determining module 301, configured to determine a data fragment to be migrated in a relational database instance in a distributed relational database; a second determining module 302, configured to determine a target relational database instance in the distributed relational database, where the target relational database instance is used to store the user data in the data segment to be migrated; a generating module 303, configured to generate a migration task for migrating user data in the to-be-migrated data segment based on the to-be-migrated data segment and the target relational database instance; a submitting module 304, configured to submit the migration task to the data migration system, so that the data migration system migrates user data in the data segment to be migrated to the target relational database instance based on the migration task.
The data migration apparatus provided in this embodiment is used to implement the corresponding data migration method in the foregoing multiple method embodiments, and has the beneficial effects of the corresponding method embodiment, which are not described herein again.
Referring to fig. 4, a schematic structural diagram of a data migration apparatus in the fourth embodiment of the present application is shown.
The data migration apparatus provided in this embodiment includes: a first determining module 401, configured to determine a data fragment to be migrated in a relational database instance in a distributed relational database; a second determining module 402, configured to determine a target relational database instance in the distributed relational database, where the target relational database instance is used to store the user data in the data segment to be migrated; a generating module 403, configured to generate a migration task for migrating user data in the data shards to be migrated, based on the data shards to be migrated and the target relational database instance; a submitting module 404, configured to submit the migration task to the data migration system, so that the data migration system migrates user data in the data segment to be migrated to the target relational database instance based on the migration task.
Optionally, the first determining module 401 includes: the generating sub-module 4011 is configured to generate a list of the data shards to be migrated based on the data capacity of the data shards in the relational database instance.
Optionally, after the generating the sub-module 4011, the first determining module 401 further includes: the receiving submodule 4012 is configured to receive an adjustment operation for the list of the data fragments to be migrated; and the adjusting submodule 4013 is configured to adjust the data fragment to be migrated in the list based on the adjusting operation.
Optionally, after the submitting module 404, the apparatus further includes: a first receiving module 405, configured to receive a migration state of user data in the data fragment to be migrated, where the migration state is sent by the data migration system; a displaying module 406, configured to display a migration state of the user data in the data fragment to be migrated.
Optionally, after the submitting module 404, the apparatus further includes: a fourth receiving module 407, configured to receive a rollback operation for the migration task; a rollback module 408, configured to send a migration task cancellation request to the data migration system based on the rollback operation, so that the data migration system cancels the migration task based on the migration task cancellation request.
Optionally, after the rollback module 408, the apparatus further includes: a deleting module 409, configured to send a data fragment deleting request to the target relational database instance, so that the target relational database instance deletes the target data fragment in the target relational database instance based on the data fragment deleting request.
The data migration apparatus provided in this embodiment is used to implement the corresponding data migration method in the foregoing multiple method embodiments, and has the beneficial effects of the corresponding method embodiment, which are not described herein again.
Referring to fig. 5, a schematic structural diagram of a data migration apparatus in the fifth embodiment of the present application is shown.
The data migration apparatus provided in this embodiment includes: a first determining module 501, configured to determine a data fragment to be migrated in a relational database instance in a distributed relational database; a second determining module 502, configured to determine a target relational database instance in the distributed relational database, where the target relational database instance is used to store the user data in the data segment to be migrated; a generating module 503, configured to generate a migration task for migrating user data in the data shards to be migrated, based on the data shards to be migrated and the target relational database instance; a submitting module 504, configured to submit the migration task to the data migration system, so that the data migration system migrates user data in the data segment to be migrated to the target relational database instance based on the migration task.
Optionally, after the submitting module 504, the apparatus further includes: a second receiving module 505, configured to receive, when a migration state of user data in the data fragment to be migrated is a tie-up state, a switching operation for the data fragment to be migrated in a data low peak time period; a switching module 508, configured to lock the distributed relational database based on the switching operation, and modify the fragment metadata of the data fragment to be migrated during the locking of the distributed relational database, so as to switch the data fragment to be migrated to a target data fragment in the target relational database instance.
Optionally, before the switching module 508, the apparatus further includes: a checking module 506, configured to check whether incremental user data in the to-be-migrated data segment and incremental user data in the target data segment are leveled; an executing module 507, configured to execute the step of modifying the fragment metadata of the data fragment to be migrated if the incremental user data in the data fragment to be migrated is flush with the incremental user data in the target data fragment.
Optionally, the verification module 506 is specifically configured to: and verifying whether incremental log data of the log file of the data fragment to be migrated in a preset time period is the same as incremental log data of the log file of the target data fragment in the preset time period, so as to verify whether incremental user data in the data fragment to be migrated and incremental user data in the target data fragment are leveled, wherein the log file is used for recording SQL statements for updating the user data in the data fragment.
Optionally, after the switching module 508, the apparatus further includes: a third receiving module 509, configured to receive a cleaning operation for the to-be-migrated data segment; a cleaning module 510, configured to send a data fragment cleaning request to the relational database instance where the data fragment to be migrated is located based on the cleaning operation, so that the relational database instance where the data fragment to be migrated is located cleans the data fragment to be migrated based on the data fragment cleaning request.
The data migration apparatus provided in this embodiment is used to implement the corresponding data migration method in the foregoing multiple method embodiments, and has the beneficial effects of the corresponding method embodiment, which are not described herein again.
Fig. 6 is a schematic structural diagram of an electronic device according to a sixth embodiment of the present application; the electronic device may include:
one or more processors 601;
a computer-readable medium 602, which may be configured to store one or more programs,
when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the data migration method as described in the first embodiment or the second embodiment.
Fig. 7 is a hardware structure of an electronic device according to a seventh embodiment of the present application; as shown in fig. 7, the hardware structure of the electronic device may include: a processor 701, a communication interface 702, a computer-readable medium 703 and a communication bus 704;
wherein the processor 701, the communication interface 702, and the computer-readable medium 703 are in communication with each other via a communication bus 704;
alternatively, the communication interface 702 may be an interface of a communication module, such as an interface of a GSM module;
the processor 701 may be specifically configured to: determining data fragments to be migrated in a relational database instance in a distributed relational database; determining a target relational database instance used for storing user data in the data fragment to be migrated in the distributed relational database; generating a migration task for migrating the user data in the data fragment to be migrated based on the data fragment to be migrated and the target relational database instance; and submitting the migration task to the data migration system, so that the data migration system migrates the user data in the data fragment to be migrated to the target relational database instance based on the migration task.
The Processor 701 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; but may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The computer-readable medium 703 may be, but is not limited to, a Random Access Memory (RAM), a Read-Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Read-Only Memory (EPROM), an electrically Erasable Read-Only Memory (EEPROM), and the like.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code configured to perform the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication section, and/or installed from a removable medium. The computer program, when executed by a Central Processing Unit (CPU), performs the above-described functions defined in the method of the present application. It should be noted that the computer readable medium described herein can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access storage media (RAM), a read-only storage media (ROM), an erasable programmable read-only storage media (EPROM or flash memory), an optical fiber, a portable compact disc read-only storage media (CD-ROM), an optical storage media piece, a magnetic storage media piece, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
Computer program code configured to carry out operations for the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may operate over any of a variety of networks: including a Local Area Network (LAN) or a Wide Area Network (WAN) -to the user's computer, or alternatively, to an external computer (e.g., through the internet using an internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions configured to implement the specified logical function(s). In the above embodiments, specific precedence relationships are provided, but these precedence relationships are only exemplary, and in particular implementations, the steps may be fewer, more, or the execution order may be modified. That is, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present application may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a first determination module, a second determination module, a generation module, and a submission module. Where the names of these modules do not in some cases constitute a limitation on the module itself, for example, the first determining module may also be described as a "module that determines a data shard to be migrated in a relational database instance in a distributed relational database".
As another aspect, the present application also provides a computer-readable medium on which a computer program is stored, the program, when executed by a processor, implementing the data migration method as described in the first or second embodiment.
As another aspect, the present application also provides a computer-readable medium, which may be contained in the apparatus described in the above embodiments; or may be present separately and not assembled into the device. The computer readable medium carries one or more programs which, when executed by the apparatus, cause the apparatus to: determining data fragments to be migrated in a relational database instance in a distributed relational database; determining a target relational database instance used for storing user data in the data fragment to be migrated in the distributed relational database; generating a migration task for migrating the user data in the data fragment to be migrated based on the data fragment to be migrated and the target relational database instance; and submitting the migration task to the data migration system, so that the data migration system migrates the user data in the data fragment to be migrated to the target relational database instance based on the migration task.
The expressions "first", "second", "said first" or "said second" used in various embodiments of the present disclosure may modify various components regardless of order and/or importance, but these expressions do not limit the respective components. The above description is only configured for the purpose of distinguishing elements from other elements. For example, the first user equipment and the second user equipment represent different user equipment, although both are user equipment. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of the present disclosure.
When an element (e.g., a first element) is referred to as being "operably or communicatively coupled" or "connected" (operably or communicatively) to "another element (e.g., a second element) or" connected "to another element (e.g., a second element), it is understood that the element is directly connected to the other element or the element is indirectly connected to the other element via yet another element (e.g., a third element). In contrast, it is understood that when an element (e.g., a first element) is referred to as being "directly connected" or "directly coupled" to another element (a second element), no element (e.g., a third element) is interposed therebetween.
The above description is only a preferred embodiment of the application and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention herein disclosed is not limited to the particular combination of features described above, but also encompasses other arrangements formed by any combination of the above features or their equivalents without departing from the spirit of the invention. For example, the above features may be replaced with (but not limited to) features having similar functions disclosed in the present application.
Claims (13)
1. A method of data migration, the method comprising:
determining data fragments to be migrated in a relational database instance in a distributed relational database;
determining a target relational database instance used for storing user data in the data fragment to be migrated in the distributed relational database;
generating a migration task for migrating the user data in the data fragment to be migrated based on the data fragment to be migrated and the target relational database instance;
and submitting the migration task to the data migration system, so that the data migration system migrates the user data in the data fragment to be migrated to the target relational database instance based on the migration task.
2. The method of claim 1, wherein the determining a data shard to be migrated in a relational database instance in a distributed relational database comprises:
and generating the list of the data fragments to be migrated based on the data capacity of the data fragments in the relational database instance.
3. The method of claim 2, wherein after the generating the list of data shards to be migrated, the method further comprises:
receiving an adjustment operation aiming at the list of the data fragments to be migrated;
and adjusting the data fragments to be migrated in the list based on the adjusting operation.
4. The method of claim 1, wherein after the submitting the migration task to the data migration system, the method further comprises:
receiving a migration state of user data in the data fragment to be migrated, which is sent by the data migration system;
and displaying the migration state of the user data in the data fragment to be migrated.
5. The method of claim 1, wherein after the submitting the migration task to the data migration system, the method further comprises:
when the migration state of the user data in the data fragment to be migrated is a tie-up state, receiving switching operation aiming at the data fragment to be migrated within a data low peak time period;
and locking the distributed relational database based on the switching operation, and modifying the fragment metadata of the data fragment to be migrated during the locking period of the distributed relational database so as to switch the data fragment to be migrated into the target data fragment in the target relational database instance.
6. The method of claim 5, wherein prior to said modifying shard metadata of said data shards to be migrated, said method further comprises:
checking whether the incremental user data in the data fragment to be migrated and the incremental user data in the target data fragment are leveled;
and if the incremental user data in the data fragment to be migrated is flush with the incremental user data in the target data fragment, executing the step of modifying the fragment metadata of the data fragment to be migrated.
7. The method of claim 6, wherein the verifying whether the incremental user data in the data slice to be migrated is flush with the incremental user data in the target data slice comprises:
and verifying whether incremental log data of the log file of the data fragment to be migrated in a preset time period is the same as incremental log data of the log file of the target data fragment in the preset time period, so as to verify whether incremental user data in the data fragment to be migrated and incremental user data in the target data fragment are leveled, wherein the log file is used for recording SQL statements for updating the user data in the data fragment.
8. The method of claim 5, wherein after switching the data shard to be migrated to a target data shard in the target relational database instance, the method further comprises:
receiving a cleaning operation aiming at the data fragments to be migrated;
and sending a data fragment cleaning request to the relational database instance where the data fragments to be migrated are located based on the cleaning operation, so that the relational database instance where the data fragments to be migrated are located cleans the data fragments to be migrated based on the data fragment cleaning request.
9. The method of claim 1, wherein after the submitting the migration task to the data migration system, the method further comprises:
receiving a rollback operation for the migration task;
and sending a migration task cancellation request to the data migration system based on the rollback operation, so that the data migration system cancels the migration task based on the migration task cancellation request.
10. The method of claim 9, wherein after sending a migration task cancellation request to the data migration system based on the rollback operation, the method further comprises:
and sending a data fragment deleting request to the target relational database instance, so that the target relational database instance deletes the target data fragment in the target relational database instance based on the data fragment deleting request.
11. A data migration apparatus, the apparatus comprising:
the system comprises a first determining module, a second determining module and a third determining module, wherein the first determining module is used for determining data fragments to be migrated in a relational database instance in a distributed relational database;
a second determining module, configured to determine a target relational database instance in the distributed relational database, where the target relational database instance is used to store the user data in the data segment to be migrated;
a generating module, configured to generate a migration task for migrating user data in the data shards to be migrated, based on the data shards to be migrated and the target relational database instance;
and the submitting module is used for submitting the migration task to the data migration system, so that the data migration system migrates the user data in the data fragment to be migrated to the target relational database instance based on the migration task.
12. An electronic device, comprising:
one or more processors;
a computer readable medium configured to store one or more programs,
when executed by the one or more processors, cause the one or more processors to implement a data migration method as claimed in any one of claims 1-10.
13. A computer-readable medium, on which a computer program is stored which, when being executed by a processor, carries out the data migration method according to any one of claims 1 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010725946.XA CN113420006A (en) | 2020-07-24 | 2020-07-24 | Data migration method and device, electronic equipment and computer readable medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010725946.XA CN113420006A (en) | 2020-07-24 | 2020-07-24 | Data migration method and device, electronic equipment and computer readable medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113420006A true CN113420006A (en) | 2021-09-21 |
Family
ID=77711534
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010725946.XA Pending CN113420006A (en) | 2020-07-24 | 2020-07-24 | Data migration method and device, electronic equipment and computer readable medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113420006A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114138714A (en) * | 2022-02-08 | 2022-03-04 | 上海金仕达软件科技有限公司 | Data migration method and device, storage medium and electronic equipment |
-
2020
- 2020-07-24 CN CN202010725946.XA patent/CN113420006A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114138714A (en) * | 2022-02-08 | 2022-03-04 | 上海金仕达软件科技有限公司 | Data migration method and device, storage medium and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9990224B2 (en) | Relaxing transaction serializability with statement-based data replication | |
| CN108280080B (en) | Data synchronization method and device and electronic equipment | |
| CN102629247B (en) | Method, device and system for data processing | |
| CN109508326B (en) | Method, device and system for processing data | |
| CN110402441A (en) | Quote accesses control list | |
| US20230099501A1 (en) | Masking shard operations in distributed database systems | |
| US11429311B1 (en) | Method and system for managing requests in a distributed system | |
| CN113420006A (en) | Data migration method and device, electronic equipment and computer readable medium | |
| US20180011897A1 (en) | Data processing method having structure of cache index specified to transaction in mobile environment dbms | |
| US11308058B1 (en) | Building and using combined multi-type sub-indices to search NoSQL databases | |
| WO2020192663A1 (en) | Data management method and related device | |
| US9460137B2 (en) | Handling an increase in transactional data without requiring relocation of preexisting data between shards | |
| US9607021B2 (en) | Loading data with complex relationships | |
| US20230153300A1 (en) | Building cross table index in relational database | |
| CN115981559A (en) | Distributed data storage method and device, electronic equipment and readable medium | |
| US11580128B2 (en) | Preventing DBMS deadlock by eliminating shared locking | |
| CN113419896A (en) | Data recovery method and device, electronic equipment and computer readable medium | |
| CN114238390A (en) | Data warehouse optimization method, device, equipment and storage medium | |
| US20220027335A1 (en) | Active-active system index management | |
| US20200104391A1 (en) | Ensuring integrity of records in a not only structured query language database | |
| CN112084141A (en) | Full-text retrieval system capacity expansion method, device, equipment and medium | |
| US11995102B2 (en) | Bidirectional data replication with wait for data function | |
| US11816088B2 (en) | Method and system for managing cross data source data access requests | |
| US20220405263A1 (en) | Increasing Index Availability in Databases | |
| CN107169098A (en) | Data method for carrying, data handling device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 40058774 Country of ref document: HK |