CN116226078A - Data migration method, device, equipment and storage medium - Google Patents
Data migration method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN116226078A CN116226078A CN202211564060.7A CN202211564060A CN116226078A CN 116226078 A CN116226078 A CN 116226078A CN 202211564060 A CN202211564060 A CN 202211564060A CN 116226078 A CN116226078 A CN 116226078A
- Authority
- CN
- China
- Prior art keywords
- migration
- migrated
- data
- state
- batch
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013508 migration Methods 0.000 title claims abstract description 429
- 230000005012 migration Effects 0.000 title claims abstract description 416
- 238000000034 method Methods 0.000 title claims abstract description 89
- 238000003860 storage Methods 0.000 title claims abstract description 15
- 238000012545 processing Methods 0.000 claims abstract description 35
- 238000012795 verification Methods 0.000 claims description 17
- 238000011161 development Methods 0.000 abstract description 4
- 230000008569 process Effects 0.000 description 32
- 238000010586 diagram Methods 0.000 description 5
- 238000012546 transfer Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000013461 design Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000010923 batch production Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000013524 data verification Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a data migration method, a data migration device, data migration equipment and a storage medium. The main technical scheme comprises the following steps: acquiring data to be migrated of a first system; the data to be migrated comprises service type information; carrying out batch processing on the data to be migrated according to the business type information to obtain batch data to be migrated of different batches; acquiring preset time set for batch data to be migrated, and configuring a migration label for the batch data to be migrated according to the preset time to obtain label data to be migrated; the method comprises the steps of executing batch migration of label data to be migrated from a first system to a second system. According to the method and the device, the data to be migrated are batched, and meanwhile, the migration label is configured based on the preset time, so that migration of new and old system data is completed batched, rapidly and accurately at any time under the condition that normal service development is guaranteed and customer experience is not affected.
Description
Technical Field
The present disclosure relates to the field of data migration technologies, and in particular, to a data migration method, apparatus, device, and storage medium.
Background
In recent years, the internet finance industry rapidly develops, the transaction number of daily living users is also continuously increased, the requirement of massive data processing on the performance of a server system is also higher and higher, the long-term iteration of service requirements causes disordered code structures and unclear layers, and the system lacks of integral design and cannot rapidly develop new services, so that the system needs to be correspondingly reconfigured.
However, after the system is reconfigured, the data of the old system needs to be migrated to the new system, and in the conventional implementation, the data of the old system is generally migrated to the new system as a whole, however, the following problems may occur: because of the error occurring in the process of data migration, the task amount is very large in the process of investigation, material resources and manpower are consumed, and the process of unified migration generally needs to stop receiving new service requests, thereby bringing bad experience to clients.
Disclosure of Invention
Based on the above, the application provides a data migration method, a device, equipment and a storage medium, which can realize quick and accurate migration of new and old system data in batches at any time under the condition of ensuring normal development of business and not affecting customer experience.
In a first aspect, a data migration method is applied to a data migration system, where the data migration system includes a first system and a second system, and the method includes:
acquiring data to be migrated of a first system; the data to be migrated comprises service type information;
carrying out batch processing on the data to be migrated according to the business type information to obtain batch data to be migrated of different batches; acquiring preset time set for batch data to be migrated, and configuring a migration label for the batch data to be migrated according to the preset time to obtain label data to be migrated;
the method comprises the steps of executing batch migration of label data to be migrated from a first system to a second system.
According to an implementation manner in the embodiment of the present application, batch processing is performed on data to be migrated according to service type information to obtain batch data to be migrated of different batches, including:
classifying the data to be migrated according to the service type information to obtain classified data to be migrated of different types;
and taking each type of the classified data to be migrated as a batch to obtain batch data to be migrated of different batches.
According to an implementation manner in the embodiments of the present application, the data migration system further includes a front-end service system, the tag data to be migrated includes a migration identification information set, and performing batch migration of the tag data to be migrated from the first system to the second system includes:
Uniformly setting initial migration states corresponding to each migration identification information in the migration identification information set to be non-migration states;
acquiring service transaction data sent by a front-end service system, and modifying the non-migration state of each migration identification information into a corresponding current migration state according to the service transaction data;
and executing batch migration of the label data to be migrated from the first system to the second system according to the current migration state corresponding to each migration identification information.
According to one implementation manner in the embodiments of the present application, the current migration state includes an undelivered state and/or an in-migration state; performing batch migration of tag data to be migrated from a first system to a second system according to the current migration state corresponding to each migration identification information, including:
when the migration identification information corresponds to the current migration state to be the non-migration state, locking the label data to be migrated;
if the locking is successful, modifying the non-migrated state into a migrated state; performing batch migration of the label data to be migrated from the first system to the second system;
if the locking fails, the non-migration state is kept, and the next migration is waited; and/or
And executing batch migration of the label data to be migrated from the first system to the second system when the migration identification information corresponds to the current migration state as the migration in-process state.
According to one implementation manner in the embodiments of the present application, the current migration state further includes a state in the processing of the first system and/or a migrated state; performing batch migration of tag data to be migrated from a first system to a second system according to the current migration state corresponding to each migration identification information, including:
when the current migration state corresponding to the migration identification information is the state in the first system processing, acquiring counter value information corresponding to the migration identification information in real time;
when the counter value information is zero, modifying the state in the first system processing to be an un-migrated state; locking the label data to be migrated;
if the locking is successful, modifying the non-migrated state into a migrated state; performing batch migration of the label data to be migrated from the first system to the second system;
if the locking fails, the non-migration state is kept, and the next migration is waited; and/or
And when the current migration state corresponding to the migration identification information is the migrated state, skipping the migration execution step of the label data to be migrated corresponding to the migration identification information.
According to one implementation manner in the embodiment of the present application, after performing the step of migrating the tag data to be migrated from the first system to the second system in batches, the method further includes:
Unlocking the label data to be migrated corresponding to the migration identification information;
if the unlocking is successful, modifying the migration identification information corresponding to the in-migration state into a migrated state;
if the unlocking fails, the state in migration is kept, and the next migration is waited.
According to an implementation manner in the embodiments of the present application, when the migration state corresponding to the migration identification information is a migrated state, the method further includes:
acquiring to-be-migrated tag data of a first system and to-be-migrated tag data of a second system corresponding to migration identification information;
verifying the label data to be migrated of the first system and the label data to be migrated of the second system;
if the verification is passed, modifying the migration label of the label data to be migrated into a migrated label;
if the verification is not passed, the original migration label of the label data to be migrated is maintained, and the data migration step of the label data to be migrated is executed in a returning mode.
In a second aspect, a data migration apparatus is provided, applied to a data migration system, where the data migration system includes a first system and a second system, and the data migration apparatus is disposed in the data migration system, and the apparatus includes:
the acquisition module is used for acquiring data to be migrated of the first system; the data to be migrated comprises service type information;
The batch module is used for carrying out batch processing on the data to be migrated according to the business type information to obtain batch data to be migrated of different batches;
the configuration module is used for acquiring preset time set for batch data to be migrated, and configuring a migration tag for the batch data to be migrated according to the preset time to obtain tag data to be migrated;
and the migration module is used for executing batch migration of the label data to be migrated from the first system to the second system.
In a third aspect, there is provided a computer device comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,,
the memory stores computer instructions executable by the at least one processor to enable the at least one processor to perform the method referred to in the first aspect above.
In a fourth aspect, there is provided a computer readable storage medium having stored thereon computer instructions for causing a computer to perform the method referred to in the first aspect above.
According to the technical content provided by the embodiment of the application, the data to be migrated of the first system is obtained; the data to be migrated comprises service type information; carrying out batch processing on the data to be migrated according to the business type information to obtain batch data to be migrated of different batches; acquiring preset time set for batch data to be migrated, and configuring a migration label for the batch data to be migrated according to the preset time to obtain label data to be migrated; the method comprises the steps of executing batch migration of label data to be migrated from a first system to a second system. According to the operation, the data to be migrated is batched, and the migration label is configured based on the preset time, so that the migration of new and old system data can be rapidly and accurately completed in batches at any time under the condition that normal development of business is guaranteed and customer experience is not affected.
Drawings
FIG. 1 is a system frame diagram of a data migration method in one embodiment;
FIG. 2 is a flow chart illustrating a method of data migration according to one embodiment;
FIG. 3 is a flow chart illustrating step 207 of a data migration method according to one embodiment;
FIG. 4 is a flowchart illustrating a data migration method according to an embodiment of step 305;
FIG. 5 is a second flow chart of a data migration method according to one embodiment;
FIG. 6 is a block diagram of a data migration apparatus in one embodiment;
fig. 7 is a schematic structural diagram of a computer device in one embodiment.
Detailed Description
The present application will be described in further detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.
For ease of understanding, a system architecture to which the present application applies will first be described. The data migration method provided by the application can be applied to a system architecture shown in fig. 1. The system architecture comprises: front-end business system 101, front-end system 103, first system 105 and second system 107, front-end system 103 processes the data to be migrated of first system 105 in batches, and combines business transaction data of front-end business system 101 to realize batch migration of the data to be migrated of first system 105 to second system 107, and under the condition that current business transaction service is not suspended, batch migration of the data to be migrated of first system 105 to second system 107 is completed, so that user experience is not affected, and user satisfaction is enhanced. The first system 105 may be an old system, the second system 107 may be a new system, the new system is an updated version of the old system, and when the data to be migrated of the first system 105 is migrated to the second system 107 in batches, dual-write and dual-use of the new system and the old system may be implemented, so as to further achieve migration accuracy of the data to be migrated.
Fig. 2 is a flowchart of a data migration method according to an embodiment of the present application, where the method may be performed by the front-end system 103 in the system shown in fig. 1. As shown in fig. 2, the method may include the steps of:
step 201: acquiring data to be migrated of a first system; the data to be migrated includes service type information.
Here, the front-end system is provided with a data migration state table, and the data migration state table is used for recording the data migration state in the data migration process. Specifically, a front-end system acquires data to be migrated of a first system; the data to be migrated includes service type information.
Step 203: and carrying out batch processing on the data to be migrated according to the business type information to obtain batch data to be migrated of different batches.
The data to be migrated is batched according to the business type information, namely, the data to be migrated is divided into different batches. When the data to be migrated is divided into different batches, the processing type of the data to be migrated needs to be considered, if the query processing service is the query processing service, the data to be migrated is not affected by the data migration process, and the data to be migrated is integrally migrated according to the general operation. In order to avoid the inconsistent problem of migration data caused in the data migration process, the accounting processing business needs to divide the data to be migrated into different batches to obtain batch data to be migrated of different batches, thereby realizing batch-to-batch migration of the batch data to be migrated of the first system to the second system.
Step 205: acquiring preset time set for batch data to be migrated, and configuring a migration label for the batch data to be migrated according to the preset time to obtain the label data to be migrated.
Wherein, the preset time can be provided with a plurality of.
Specifically, a preset time set by a worker for batch data to be migrated according to business requirements is obtained, and a migration label is configured for the batch data to be migrated according to the preset time, so that the label data to be migrated is obtained. The configuration tag at this time is referred to as a migration tag, that is, when the preset time expires, the migration tag is configured for the batch data to be migrated to obtain the batch data to be migrated, that is, the batch data to be migrated can be subjected to data migration operation at this time.
Step 207: the method comprises the steps of executing batch migration of label data to be migrated from a first system to a second system.
Here, after the front-end system configures the in-migration tag for the batch data to be migrated, the front-end system performs an operation of migrating the data of the tags to be migrated of all the configured in-migration tags from the first system to the second system.
It can be seen that, in the embodiment of the present application, data to be migrated of the first system is obtained; the data to be migrated comprises service type information; carrying out batch processing on the data to be migrated according to the business type information to obtain batch data to be migrated of different batches; acquiring preset time set for batch data to be migrated, and configuring a migration label for the batch data to be migrated according to the preset time to obtain label data to be migrated; the method comprises the steps of executing batch migration of label data to be migrated from a first system to a second system. According to the operation, the data to be migrated is batched, and the migration label is configured based on the preset time, so that the migration of new and old system data can be rapidly and accurately completed in batches at any time under the condition that normal development of business is guaranteed and customer experience is not affected.
The following describes the step 203 in detail, namely, "batch processing the data to be migrated according to the service type information to obtain batch data to be migrated of different batches" with reference to the embodiment.
Classifying the data to be migrated according to the service type information to obtain classified data to be migrated of different types; and taking each type of the classified data to be migrated as a batch to obtain batch data to be migrated of different batches.
Here, since the accounting-type service type information includes, but is not limited to, asset service, liability service, intermediate service, and off-meter service, data to be migrated may be classified into asset service data to be migrated, liability service data to be migrated, intermediate service data to be migrated, off-meter service data to be migrated, and the like. Taking each type of classified data to be migrated as one batch to obtain batch data to be migrated of different batches, namely taking the asset business data to be migrated as one batch, taking the liability business data to be migrated as another batch, and so on to obtain batch data to be migrated of different batches.
Here, it should be emphasized that the data to be migrated may be classified not only according to the service type information, but also based on the product type, the product number, the public service or the private service, etc., and the limitation is not made herein.
Through the operation, the data to be migrated is classified to obtain batch data to be migrated, so that the batch data to be migrated is conveniently migrated from the first system to the second system in batches, and the effect of rapidly completing data migration of the first system and the second system in batches at any time is achieved.
As shown in fig. 3, the above step 207, i.e., performing batch-wise migration of tag data to be migrated from the first system to the second system, is described in detail below in connection with an embodiment.
Step 301, uniformly setting initial migration states corresponding to each migration identification information in the migration identification information set to an un-migrated state.
The label data to be migrated comprises a migration identification information set, the migration identification information set comprises at least one migration identification information, each migration identification information penetrates through the whole data migration process, and the migration identification information comprises but is not limited to account information, client number information, order number information and the like.
Here, the migration state corresponding to each migration identification information is set to the non-migration state in a unified manner, that is, the initial state of each migration identification information is defaulted to the non-migration state.
Step 303, obtaining service transaction data sent by the front-end service system, and modifying the non-migration state of each migration identification information to correspond to the current migration state according to the service transaction data.
The business transaction data comprises transaction identification information, wherein the transaction identification information is migration identification information, and the names of the transaction identification information are different at different stages, but the corresponding specific information is consistent, and the business transaction data comprises account information, client number information, order number information and the like.
Here, since the front-end service system still transmits the service transaction data in real time in the migration process of the data, it is required to ensure that the service transaction can be performed normally and that the label data to be migrated can be migrated normally, and therefore, the non-migrated state of each migration identification information needs to be modified in real time. And the front-end system is internally provided with a counter, the numerical value information of the counter defaults to 0, when business transaction occurs, the numerical value information of the counter is increased by 1, and when the business transaction is completed, the numerical value information of the counter is decreased by 1.
In one implementation manner, since the business transaction data includes transaction identification information, when the transaction identification information exists in the migration identification information set, which indicates that the transaction is currently occurring on the tag data to be migrated corresponding to the transaction identification information, the counter value information is increased by 1, and at this time, the tag data to be migrated cannot be migrated. Therefore, the transaction identification information is required to be changed into a state in the processing of the first system corresponding to the current migration state, and the first system is adopted to process the business transaction data, namely the old system is adopted to process the current business transaction data.
In another implementation manner, when the transaction identification information does not exist in the migration identification information set, it is indicated that the service transaction data corresponding to the transaction identification information at this time is a first created service transaction, the initial state of the first created service transaction is defaulted to be a migrated state, and the first created service transaction data is directly processed by adopting the second system, that is, the new system is adopted to process the service transaction data.
In yet another implementation, the current migration state corresponding to other migration identification information in the migration identification information set other than the transaction identification information remains in the original non-migration state.
Here, according to the current migration state corresponding to each migration identification information, the label data to be migrated is migrated from the first system to the second system in batches.
Through the operation, the initial migration state of each migration identification information is modified by combining the service transaction data sent by the front-end service system to obtain the corresponding current migration state, and the operation of migrating the label data to be migrated from the first system to the second system in batches is performed based on the current migration state, so that the migration operation of the label data to be migrated is completed under the condition that the service of the service transaction is not suspended.
In one embodiment, as shown in fig. 4, the step 305 of performing batch-wise migration of the tag data to be migrated from the first system to the second system according to the current migration status corresponding to each migration identification information is described in detail.
The current migration state comprises an un-migration state, a first system processing state and a migrated state.
Executing step 3051 when the migration identification information corresponds to the current migration state as the non-migration state; executing step 3053 when the current migration state corresponding to the migration identification information is the in-migration state; executing step 3055 when the current migration state corresponding to the migration identification information is the state in the first system processing; when the current migration state corresponding to the migration identification information is the migrated state, step 3057 is executed.
Here, when the migration identification information corresponds to the current migration state being the non-migration state, the locking operation is performed on the tag data to be migrated, but the locking operation is not necessarily directly successful, because the service transaction is performed in real time, and the service transaction may occur or a problem occurs in the migration identification information at this time. It should be noted that, only when the migration identification information corresponds to the current migration state as the non-migration state, the locking operation is performed.
If the locking is successful, the non-migration state corresponding to the migration identification information can be modified into a migration state, which indicates that the label data to be migrated can be migrated at this time, that is, the operation of migrating the label data to be migrated from the first system to the second system in batches is performed. At this time, if a service transaction occurs, the front-end system directly reports errors and refuses to receive the current service transaction data.
Here, if the locking fails, the current migration state of the migration identification information is kept to be the non-migration state for any reason, that is, the locking operation is indicated to be problematic, and after the problem is solved, the next migration needs to be waited. And at this time, if the business transaction occurs, the first system is still adopted to process the business transaction data, and the label data to be migrated, which is not successfully migrated, is cleared when waiting for the next migration.
Here, when the migration identification information corresponds to the current migration state being the in-migration state, the operation of migrating the tag data to be migrated from the first system to the second system in batches may be directly performed.
Here, when the migration identification information corresponds to the current migration state being the state in which the first system is in process, it is indicated that the current business transaction occurs, and the first system is in process, therefore, the counter value information corresponding to the migration identification information needs to be acquired in real time, since the counter is decremented by 1 after the business transaction is completed, when the counter value information is zero, it can be indicated that no business transaction occurs at this time, the step of modifying the state in process of the first system into the non-migration state can be executed, after that, the step of locking the tag data to be migrated when the current migration state is the non-migration state is executed, and if the locking is successful, the non-migration state is modified into the in-process state; and performing an operation of transferring the tag data to be transferred from the first system to the second system in batches; if the locking fails, the locking process is indicated to have a problem, and the migration needs to be terminated, the migration is still in an undelivered state, and after the problem is solved, the next migration is performed.
Step 3057, skipping the migration execution step of the to-be-migrated tag data corresponding to the migration identification information.
Here, when the current migration state corresponding to the migration identification information is the migrated state, the step of executing migration on the tag data to be migrated corresponding to the migration identification information is skipped. It should be noted that, the migrated state may have two cases, one is that the migration identification information is the first created business transaction, and then the migrated state is directly defaulted, at this time, if a new business transaction occurs, then the second system is directly invoked to process the new business transaction data; another case is that the migration identification information corresponds to the to-be-migrated tag data being migrated, that is, migration is not needed. At this time, if a new business transaction occurs, the first system is called to process the business transaction data, and after the processing is successful, the second system is called to process the business transaction data so as to realize double writing, thereby ensuring the consistency of migration data of the first system and the second system and facilitating the use of the subsequent data verification and rollback.
It should be noted that, after the step of migrating the tag data to be migrated from the first system to the second system in batches, the method further includes the following steps:
Step 307, performing unlocking operation on the label data to be migrated corresponding to the migration identification information; if the unlocking is successful, step 3071 is executed, and if the unlocking is failed, step 3073 is executed.
Here, the unlocking operation is performed on the tag data to be migrated corresponding to the migration identification information, and under normal conditions, the unlocking can be successfully performed after the migration operation is performed, but if a problem occurs in the migration process, the unlocking failure is caused.
Step 3071, modifying the migration identification information to the migrated state corresponding to the in-migration state.
Here, if the unlocking is successful, it indicates that the migration of the tag data to be migrated corresponding to the migration identification information is completed, the state in migration corresponding to the migration identification information may be modified to be a migrated state, which indicates that the migration operation is completed. It should be noted that, only when the non-migrated state corresponding to the migration identification information is modified to the in-migration state, the unlocking operation is performed.
Here, if the unlocking fails, it is indicated that an error occurs during the data migration, and at this time, the state in migration corresponding to the migration identification information is maintained, and after the problem is solved, the next migration is waited.
It should be emphasized that the migration operation is a process performed in a circulating manner, and it may be indicated that the migration operation of the batch is completed until the current migration state corresponding to each migration identifier in the label data to be migrated of the current batch is the migrated state.
Through the operation, the operation executed is different according to the different corresponding current migration states of the migration identification information, and meanwhile, the system for calling and executing the business transaction data is also different, so that the migration operation of the label data to be migrated is completed under the condition that the business transaction is not suspended. After the step of transferring the label data to be transferred from the first system to the second system in batches is executed, the state in the transfer corresponding to the transfer identification information can be modified to be the transferred state based on the unlocking condition, so that the transfer is completed, and repeated transfer is avoided.
As shown in fig. 5, in one embodiment, when the migration status corresponding to the migration identification information is the migrated status, the method further includes the following steps:
step 401, obtaining to-be-migrated tag data of the first system and to-be-migrated tag data of the second system corresponding to the migration identification information.
Specifically, when the migration state corresponding to the migration identification information is the migrated state, the to-be-migrated tag data of the first system and the to-be-migrated tag data of the second system corresponding to the migration identification information are obtained, so that consistency of the to-be-migrated tag data of the first system and the to-be-migrated tag data of the second system is determined.
Step 403, verifying the label data to be migrated of the first system and the label data to be migrated of the second system; if the verification is passed, step 405 is executed; if the verification is not passed, step 407 is performed.
Specifically, the label data to be migrated of the first system and the label data to be migrated of the second system are verified by triggering a full-quantity query operation, and a verification result is returned to the front-end system so as to screen out migration abnormal data.
Step 405, modifying the migration tag of the tag data to be migrated.
Specifically, if the verification is passed, the migration label of the migration label data is modified to be a migrated label, where it needs to be explained that when the batch data to be migrated is not configured, the migration label corresponding to the label data to be migrated is a label in migration after being configured, and when the verification is passed, the migration label is modified to be a migrated label, that is, the data migration operation about the batch is successfully completed, and all subsequent business transactions of the label data to be migrated corresponding to the batch are processed by the second system.
Step 407, if the verification is not passed, the original migration label of the label data to be migrated is maintained, and the data migration step of the label data to be migrated is executed again.
Specifically, if the verification is not passed, the original migration label of the label data to be migrated, namely the label in migration, is maintained, and meanwhile, the reason of verification failure is searched: 1) If part of the tag data to be migrated is wrong, checking whether the tag data to be migrated of the first system is problematic or the tag data to be migrated of the second system is problematic, and returning to execute the data migration step of the tag data to be migrated after the problem is regulated; 2) If the batch data to be migrated is configured once, a plurality of groups of label data to be migrated are obtained, and part of the groups of label data to be migrated are misplaced, the label data to be migrated corresponding to the misplaced groups can be removed, and the data migration step of the label data to be migrated corresponding to the groups is executed in a returning mode; 3) If the label data to be migrated has more errors and is scattered, the data is not easy to adjust in a short time, all the label data to be migrated corresponding to the label in migration can be removed, and the data migration operation on all the label data to be migrated is re-executed.
Through the operation, based on the verification result, consistency of the label data to be migrated of the first system and the label data to be migrated of the second system is ensured, so that rapid and accurate data migration is ensured.
By way of example, the migration process of the tag data to be migrated is further described herein, and it is assumed that at this time, there are three batches of data in the first system, that is, P1 batch of data, P2 batch of data, and P3 batch of data, where the P3 batch of data is provided with migrated tag information, and if a service transaction occurs in the later stage, the second system is directly used for processing, and the migration operation is not required; the P1 batch of data is not configured, if business transaction occurs in the later period, the first system is directly adopted for processing, and the batch of data is not required to be migrated temporarily; the P2 batch data is configured based on the service requirement and is configured as a migration tag, and the migration process of the P2 batch data is mainly illustrated below:
assuming that the tag data to be migrated is P2 batch data, at this time, migration identification information contained in the P2 batch data is an identification a, an identification B and an identification C respectively, initial migration states corresponding to the identification a, the identification B and the identification C are all defaulted to be non-migrated states, and the tag data is assumed to be represented by 1, and at this time, if a service transaction occurs, the tag data is directly executed by adopting a first system, namely an old system, and specifically represented by the following table form:
Acquiring service transaction data sent by a front-end service system, wherein the service transaction data comprises service identification information, and the service identification information is assumed to be an identification B, an identification C and an identification D, so that the identification A does not currently carry out service transaction, and the counter value information is 0; assuming that a business transaction occurs in the identifier B at this time, the counter value information is 1, locking fails when the data corresponding to the identifier B is migrated, and meanwhile, the current migration state corresponding to the identifier B is modified to be the state in the processing of the first system, wherein the state is denoted by 2, and the first system is called to process the current business transaction data; by analogy, if 3 services occur in the identifier C at this time, the counter value information is 3, the current migration state corresponding to the identifier C needs to be modified to be 2, and meanwhile, the first system is also called to process the current transaction; the fact that the identifier D does not exist in the previous migration identifier information set indicates that the service transaction is created for the first time, the current migration state is set as a migrated state by default, the condition is indicated by 4, and the second system is directly called to process the service transaction data corresponding to the identifier D. The table is expressed as follows:
Acquiring the current migration state corresponding to each migration identification information in the label data to be migrated, if the current migration state corresponding to the identification A is 1 at the moment, locking the label data, and if the locking is successful, modifying the current migration state corresponding to the identification A into a migration state, wherein the current migration state is assumed to be represented by 3; at this time, the current migration states corresponding to the identifier B and the identifier C are both 2, so that the counter value information corresponding to the identifier B and the identifier C needs to be obtained in real time, and when the counter value information corresponding to the identifier B and the identifier C is 0, the current migration states corresponding to the identifier B and the identifier C are modified to be 1. Because the current state corresponding to the identifier D is 4 at this time, when the migration data corresponding to the identifier D is migrated, the migration is directly skipped, and the migration is not needed. The table is expressed as follows:
and because the current migration state corresponding to the identifier A is 3 at this time, the operation of migrating the label data to be migrated of the first system to the second system is executed, and after the execution is finished, unlocking operation is carried out on the identifier A, if the business transaction occurs, an error is reported at this time, after the unlocking is successful, the current migration state 3 corresponding to the identifier A is modified to be a migrated state 4, and the first system and the second system are simultaneously called to process the label data to be migrated corresponding to the identifier A. Since the migration states of the markers B and C are both 1 and the counter value information is 0 at this time, the same operation as the above-described change from the non-migration state 1 to the in-migration state 3 is performed. The table is expressed as follows:
Because the current migration states corresponding to the identifier B and the identifier C are 3 at this time, the operation of migrating the tag data to be migrated of the first system to the second system needs to be executed on the identifier B and the identifier C, and after the execution is completed, the unlocking operation is performed on the identifier B and the identifier C, at this time, if a service transaction occurs, an error is reported, after the unlocking is successful, the current migration state 3 corresponding to the identifier B and the identifier C is modified to be a migrated state 4, and the first system and the second system are simultaneously called to process the tag data to be migrated corresponding to the identifier B and the identifier C, namely the double-write operation. The table is expressed as follows:
after migration of the label data to be migrated corresponding to the P2 batch is completed, verification is needed, if the verification is passed, the migration label of the label data to be migrated corresponding to the P2 batch is modified to be a migrated label, which indicates that the migration operation of the label data to be migrated corresponding to the P2 batch is completed and the migration is successful, and then if business transaction occurs again, the second system, namely the new system, is directly adopted to process the identification data to be migrated. The table is expressed as follows:
and the like, the migration operation of all batches of label data to be migrated, namely the migration operation of all data to be migrated is realized.
It should be understood that, although the steps in the flowcharts of fig. 2-5 are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly stated in the present application, and the steps may be executed in other orders. Moreover, at least some of the steps in fig. 2-5 may include multiple sub-steps or stages that are not necessarily performed at the same time, but may be performed at different times, nor does the order in which the sub-steps or stages are performed necessarily occur in sequence, but may be performed alternately or alternately with at least a portion of the other steps or sub-steps of other steps.
The method embodiment described above may be applied to a variety of application scenarios, which may include, but are not limited to, the following application scenarios:
by adding the front-end system, the function difference of the first system and the second system can be avoided, the data migration progress can be controlled, the data migration operation can be realized according to the current migration state of the data to be migrated, and the migration of the business transaction data can be guaranteed to be completed under the condition of normal operation of the first system and the second system.
Fig. 6 is a schematic structural diagram of a data migration apparatus according to an embodiment of the present application, where the apparatus may be disposed in a front-end system of the system shown in fig. 1, so as to perform the method flows shown in fig. 2 to 5. As shown in fig. 6, the apparatus may include: an acquisition module 501, a batching module 503, a configuration module 505 and a migration module 507. The main functions of each component module are as follows:
an obtaining module 501, configured to obtain data to be migrated of a first system; the data to be migrated comprises service type information;
a batch module 503, configured to batch-process the data to be migrated according to the service type information, to obtain batch data to be migrated of different batches;
the configuration module 505 is configured to obtain a preset time set for batch data to be migrated, configure a migration tag for the batch data to be migrated according to the preset time, and obtain data of the migration tag;
the migration module 507 is configured to perform batch-wise migration of tag data to be migrated from the first system to the second system.
In one embodiment, the batch module 503 is further configured to:
classifying the data to be migrated according to the service type information to obtain classified data to be migrated of different types;
And taking each type of the classified data to be migrated as a batch to obtain batch data to be migrated of different batches.
In one embodiment, the data migration system further includes a front-end service system, the tag data to be migrated includes a migration identification information set, and the migration module 507 is further configured to:
uniformly setting initial migration states corresponding to each migration identification information in the migration identification information set to be non-migration states;
acquiring service transaction data sent by a front-end service system, and modifying the non-migration state of each migration identification information into a corresponding current migration state according to the service transaction data;
and executing batch migration of the label data to be migrated from the first system to the second system according to the current migration state corresponding to each migration identification information.
In one embodiment, the current migration state includes an undelivered state and/or an in-migration state, and the migration module 507 is further configured to:
when the migration identification information corresponds to the current migration state to be the non-migration state, locking the label data to be migrated;
if the locking is successful, modifying the non-migrated state into a migrated state; performing batch migration of the label data to be migrated from the first system to the second system;
If the locking fails, the non-migration state is kept, and the next migration is waited; and/or
And when the current migration state corresponding to the migration identification information is the migration in-process state, executing batch migration of the label data to be migrated from the first system to the second system.
In one embodiment, the current migration state further includes a first system in-process state and/or a migrated state, and the migration module 507 is further configured to:
when the current migration state corresponding to the migration identification information is the state in the first system processing, acquiring numerical value information of a counter corresponding to the migration identification information in real time;
when the counter value information is zero, modifying the state in the first system processing to be an un-migrated state; locking the label data to be migrated;
if the locking is successful, modifying the non-migrated state into a migrated state; performing batch migration of the label data to be migrated from the first system to the second system;
if the locking fails, the non-migration state is kept, and the next migration is waited; and/or
And when the migration state corresponding to the migration identification information is the migrated state, skipping the migration execution step of the label data to be migrated corresponding to the migration identification information.
In one embodiment, the migration module 507 is further configured to:
Unlocking the label data to be migrated corresponding to the migration identification information;
if the unlocking is successful, modifying the migration identification information corresponding to the in-migration state into a migrated state;
if the unlocking fails, the state in migration is kept, and the next migration is waited.
In one embodiment, when the migration status corresponding to the migration identification information is a migrated status, the apparatus is further configured to:
acquiring to-be-migrated tag data of a first system and to-be-migrated tag data of a second system corresponding to migration identification information;
verifying the label data to be migrated of the first system and the label data to be migrated of the second system;
if the verification is passed, modifying the migration label of the label data to be migrated into a migrated label;
if the verification is not passed, the original migration label of the label data to be migrated is maintained, and the data migration step of the label data to be migrated is executed in a returning mode.
According to the specific embodiment provided by the application, the technical scheme provided by the application can have the following advantages:
1) The migration batch is flexible, and the migration batch can be customized according to the requirement; and the method is not influenced by the transaction service time period, and transaction service processing can be performed at any time throughout the day.
2) By adding the function difference of the front-end system shielding new and old systems, the new and old systems can normally operate in the data migration process, the customer experience is not affected, and the economic loss of customers is not caused.
3) And supporting infinite data migration retry operation, and also supporting rollback operation according to migration batches, and performing consistency check after the data migration operation is finished, so as to ensure the consistency of final migration data.
It is to be understood that not necessarily all of the above advantages may be achieved simultaneously in the practice of any one of the methods or articles of manufacture of the present application.
The same and similar parts of the above embodiments are all referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the device embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference is made to the description of the method embodiments in part.
It should be noted that, in the embodiments of the present application, the use of user data may be involved, and in practical applications, user specific personal data may be used in the schemes described herein within the scope allowed by applicable laws and regulations under the condition that the applicable laws and regulations of the country are met (for example, the user explicitly agrees, the user is explicitly notified, the user is explicitly authorized, etc.).
According to embodiments of the present application, there is also provided a computer device, a computer-readable storage medium.
As shown in fig. 7, is a block diagram of a computer device according to an embodiment of the present application. Computer equipment is intended to represent various forms of digital computers or mobile devices. Wherein the digital computer may comprise a desktop computer, a portable computer, a workstation, a personal digital assistant, a server, a mainframe computer, and other suitable computers. The mobile device may include a tablet, a smart phone, a wearable device, etc.
As shown in fig. 7, the apparatus 600 includes a computing unit 601, a ROM 602, a RAM 603, a bus 604, and an input/output (I/O) interface 605, and the computing unit 601, the ROM 602, and the RAM 603 are connected to each other through the bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
The computing unit 601 may perform various processes in the method embodiments of the present application according to computer instructions stored in a Read Only Memory (ROM) 602 or computer instructions loaded from a storage unit 608 into a Random Access Memory (RAM) 603. The computing unit 601 may be a variety of general and/or special purpose processing components having processing and computing capabilities. The computing unit 601 may include, but is not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), as well as any suitable processor, controller, microcontroller, etc. In some embodiments, the methods provided by embodiments of the present application may be implemented as a computer software program tangibly embodied on a computer-readable storage medium, such as storage unit 608.
The RAM 603 may also store various programs and data required for operation of the device 600. Part or all of the computer program may be loaded and/or installed onto the device 600 via the ROM 602 and/or the communication unit 609.
An input unit 606, an output unit 607, a storage unit 608, and a communication unit 609 in the device 600 may be connected to the I/O interface 605. Wherein the input unit 606 may be such as a keyboard, mouse, touch screen, microphone, etc.; the output unit 607 may be, for example, a display, a speaker, an indicator light, or the like. The device 600 is capable of exchanging information, data, etc. with other devices through the communication unit 609.
It should be noted that the device may also include other components necessary to achieve proper operation. It may also include only the components necessary to implement the present application, and not necessarily all the components shown in the figures.
Various implementations of the systems and techniques described here can be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof.
Computer instructions for implementing the methods of the present application may be written in any combination of one or more programming languages. These computer instructions may be provided to a computing unit 601 such that the computer instructions, when executed by the computing unit 601, such as a processor, cause the steps involved in the method embodiments of the present application to be performed.
The computer readable storage medium provided herein may be a tangible medium that may contain, or store, computer instructions for performing the steps involved in the method embodiments of the present application. The computer readable storage medium may include, but is not limited to, storage media in the form of electronic, magnetic, optical, electromagnetic, and the like.
The above embodiments do not limit the scope of the application. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application are intended to be included within the scope of the present application.
Claims (10)
1. A data migration method, applied to a data migration system, the data migration system including a first system and a second system, the method comprising:
Acquiring data to be migrated of a first system; the data to be migrated comprises service type information;
carrying out batch processing on the data to be migrated according to the business type information to obtain batch data to be migrated of different batches;
acquiring preset time set for the batch data to be migrated, and configuring a migration tag for the batch data to be migrated according to the preset time to obtain tag data to be migrated;
performing batch migration of the tag data to be migrated from the first system to the second system.
2. The method according to claim 1, wherein the batch processing the data to be migrated according to the service type information to obtain batch data to be migrated of different batches includes:
classifying the data to be migrated according to the service type information to obtain classified data to be migrated of different types;
and taking each type in the classified data to be migrated as a batch to obtain batch data to be migrated of different batches.
3. The method of claim 1, wherein the data migration system further comprises a front-end business system, the label data to be migrated comprises a migration identification information set, and the performing the batch migration of the label data to be migrated from the first system to the second system comprises:
Uniformly setting initial migration states corresponding to each migration identification information in the migration identification information set to be non-migration states;
acquiring service transaction data sent by a front-end service system, and modifying the non-migration state of each migration identification information to correspond to the current migration state according to the service transaction data;
and executing batch migration of the tag data to be migrated from the first system to the second system according to the current migration state corresponding to each migration identification information.
4. A method according to claim 3, wherein the current migration state comprises an un-migrated state and/or an in-migrated state; performing batch migration of the tag data to be migrated from the first system to the second system according to the current migration state corresponding to each migration identification information, including:
when the migration identification information corresponds to the current migration state to be an un-migrated state, locking the label data to be migrated;
if the locking is successful, modifying the non-migrated state into a migrated state; and performing batch migration of the tag data to be migrated from the first system to the second system;
If the locking fails, the non-migration state is kept, and the next migration is waited; and/or
And executing batch migration of the label data to be migrated from the first system to the second system when the migration identification information corresponds to the current migration state as the migration state.
5. The method of claim 4, wherein the current migration state further comprises a first system in-process state and/or a migrated state; performing batch migration of the tag data to be migrated from the first system to the second system according to the current migration state corresponding to each migration identification information, including:
when the current migration state corresponding to the migration identification information is a state in the first system processing, acquiring counter value information corresponding to the migration identification information in real time;
when the counter value information is zero, modifying the in-process state of the first system to an un-migrated state; locking the label data to be migrated;
if the locking is successful, modifying the non-migrated state into a migrated state; and performing batch migration of the tag data to be migrated from the first system to the second system;
If the locking fails, the non-migration state is kept, and the next migration is waited; and/or
And when the current migration state corresponding to the migration identification information is the migrated state, skipping the migration execution step of the label data to be migrated corresponding to the migration identification information.
6. The method according to any one of claims 1-5, further comprising, after said step of performing batch-wise migration of said tag data to be migrated from said first system to said second system:
unlocking the label data to be migrated corresponding to the migration identification information;
if the unlocking is successful, modifying the migration identification information corresponding to the in-migration state into a migrated state;
if the unlocking fails, the state in migration is kept, and the next migration is waited.
7. The method according to claim 6, wherein when the migration status corresponding to the migration identification information is a migrated status, the method further comprises:
acquiring the label data to be migrated of the first system and the label data to be migrated of the second system corresponding to the migration identification information;
verifying the label data to be migrated of the first system and the label data to be migrated of the second system;
If the verification is passed, modifying the migration label of the label data to be migrated into a migrated label;
if the verification is not passed, the original migration label of the label data to be migrated is maintained, and the data migration step of the label data to be migrated is executed in a returning mode.
8. A data migration apparatus, applied to a data migration system, the data migration system including a first system and a second system, the data migration apparatus being disposed in the data migration system, the apparatus comprising:
the acquisition module is used for acquiring data to be migrated of the first system; the data to be migrated comprises service type information;
the batch module is used for carrying out batch processing on the data to be migrated according to the business type information to obtain batch data to be migrated of different batches;
the configuration module is used for acquiring preset time set for the batch data to be migrated, and configuring a migration tag for the batch data to be migrated according to the preset time to obtain tag data to be migrated;
and the migration module is used for executing batch migration of the label data to be migrated from the first system to the second system.
9. A computer device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,,
the memory stores computer instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.
10. A computer readable storage medium having stored thereon computer instructions for causing a computer to perform the method of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211564060.7A CN116226078A (en) | 2022-12-07 | 2022-12-07 | Data migration method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211564060.7A CN116226078A (en) | 2022-12-07 | 2022-12-07 | Data migration method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116226078A true CN116226078A (en) | 2023-06-06 |
Family
ID=86577476
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211564060.7A Pending CN116226078A (en) | 2022-12-07 | 2022-12-07 | Data migration method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116226078A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116719482A (en) * | 2023-08-08 | 2023-09-08 | 腾讯科技（深圳）有限公司 | Data migration method and device, storage medium and electronic equipment |
-
2022
- 2022-12-07 CN CN202211564060.7A patent/CN116226078A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116719482A (en) * | 2023-08-08 | 2023-09-08 | 腾讯科技（深圳）有限公司 | Data migration method and device, storage medium and electronic equipment |
| CN116719482B (en) * | 2023-08-08 | 2024-06-11 | 腾讯科技（深圳）有限公司 | Data migration method and device, storage medium and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108958714B (en) | Business component type development method and device, computer equipment and storage medium | |
| US10951491B2 (en) | Automatic microservice problem detection in enterprise applications | |
| CN109344154B (en) | Data processing method, device, electronic equipment and storage medium | |
| CN112035230B (en) | Task scheduling file generation method, device and storage medium | |
| CN111666088A (en) | Pod replacement method and device, electronic equipment and computer-readable storage medium | |
| CN107133233B (en) | Processing method and device for configuration data query | |
| CN111367982B (en) | Method, device, computer equipment and storage medium for importing TRRIGA basic data | |
| CN116226078A (en) | Data migration method, device, equipment and storage medium | |
| US20090248186A1 (en) | Methods and Systems for Matching Configurable Manufacturing Capacity Requirements and Availability | |
| CN110428320A (en) | Data processing method and system and electronic equipment and computer-readable medium | |
| US20230214153A1 (en) | Memory device forensics and preparation | |
| CN114816772B (en) | Debugging method, debugging system and computing device for application running based on compatible layer | |
| US20230213573A1 (en) | Method and device for testing wafer, electronic device and storage medium | |
| CN113971074A (en) | Transaction processing method and device, electronic equipment and computer readable storage medium | |
| CN114063982A (en) | Method for automatically constructing functional components based on multi-scenario application | |
| CN116301758B (en) | Rule editing method, device, equipment and medium based on event time points | |
| US20230185549A1 (en) | Automatic Workflow Generation | |
| CN116680277B (en) | Information verification method, device, equipment and storage medium | |
| CN115460077B (en) | Configuration information updating method and device, storage medium and electronic device | |
| CN112347095B (en) | Data table processing method, device and server | |
| US20240036861A1 (en) | Automatic generation of interfaces for optimizing codebases | |
| CN112950138A (en) | Collaborative development state management method and device and server | |
| CN117331956A (en) | Task processing method, device, computer equipment and storage medium | |
| CN115438057A (en) | Data batch processing method, device and equipment and storage medium | |
| CN116069499A (en) | Resource matching method, device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |