CN115982119A - Method and system for adjusting sub-table data and storage medium - Google Patents

Method and system for adjusting sub-table data and storage medium Download PDF

Info

Publication number
CN115982119A
CN115982119A CN202211062652.9A CN202211062652A CN115982119A CN 115982119 A CN115982119 A CN 115982119A CN 202211062652 A CN202211062652 A CN 202211062652A CN 115982119 A CN115982119 A CN 115982119A
Authority
CN
China
Prior art keywords
data
field
field value
routing
migration
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211062652.9A
Other languages
Chinese (zh)
Inventor
关卫国
卫劲波
曹晓橹
李嘉熙
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
WeBank Co Ltd
Original Assignee
WeBank Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by WeBank Co Ltd filed Critical WeBank Co Ltd
Priority to CN202211062652.9A priority Critical patent/CN115982119A/en
Publication of CN115982119A publication Critical patent/CN115982119A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application provides a method, a system and a storage medium for adjusting tabular data, wherein the method comprises the following steps: acquiring a plurality of data sublists to be adjusted, adjustment progress states of the data sublists and a second routing field for redistributing stock data in the data sublists; the multiple data sub-tables are sub-tables obtained by performing horizontal splitting processing on a single database table by taking the first routing field as a dimension; under the condition that the adjustment progress state is in a data migration state, determining a first chart identifier redistributed for each data in the stock data based on the first routing field, the second routing field and the obtained migration log; and migrating each datum to a first data sublist corresponding to the first sublist identifier.

Description

Method and system for adjusting sub-table data and storage medium
Technical Field
The embodiment of the application relates to the technical field of data processing of financial technology (Fintech), and relates to but is not limited to a method, a system and a storage medium for adjusting spreadsheet data.
Background
With the development of computer computing, more and more technologies (such as block chains, cloud computing or big data) are applied in the financial field, the traditional financial industry is gradually changing to financial technology (Fintech), and the database technology is no exception; however, due to the security and real-time requirements of the financial industry, the financial technology also puts higher demands on the technology.
In the field of financial science and technology, in order to reduce the burden of a database and the size of a single database table and improve the data access performance, the single database table is horizontally split according to a certain routing field and routing rules to obtain a plurality of data sub-tables with independent storage spaces, and subsequent operations of increasing, deleting, modifying and checking business data are all positioned in the corresponding data sub-tables for processing. However, as the business develops, the previously designed field may no longer meet the access requirement of the business to the table data, and needs to be adjusted to another field, the system needs to redistribute the stock data in the current table and the newly added data according to the new field, that is, the migration of the stock data and the distribution process of the newly added data are involved.
In the related technology, a shutdown migration scheme or an isomorphic table double-write scheme is adopted for processing the data migration of the sub-tables, the shutdown migration scheme needs to shut down system services, it is guaranteed that no new data is written in, and then stock data in the sub-tables are migrated; the isomorphic table double-write scheme requires the creation of multiple new tables with the same data structure, and then the stock data in the old tables are migrated to the new tables. Therefore, when the sub-table data migration is performed in the related art, at least the problems that the service outage affects the user experience or wastes data resources exist.
Disclosure of Invention
The embodiment of the application provides a method, a system and a storage medium for adjusting tabular data, so as to solve the problem that service outage at least affects user experience or wastes data resources when data migration is performed in the related art.
The technical scheme of the embodiment of the application is realized as follows:
an embodiment of the present application provides a method, including:
acquiring a plurality of data sublists to be adjusted, adjustment progress states of the data sublists and a second routing field for redistributing stock data in the data sublists; the multiple data sub-tables are sub-tables obtained by performing horizontal splitting processing on a single database table by taking a first routing field as a dimension;
under the condition that the adjustment progress state is in a data migration state, determining a first chart identifier redistributed for each data in the stock data based on the first routing field, the second routing field and the obtained migration log;
and migrating each data to a first data sublist corresponding to the first sublist identifier.
An embodiment of the present application provides an apparatus, including:
the acquisition module is used for acquiring a plurality of data sublists to be adjusted, adjustment progress states of the data sublists and second routing fields for redistributing stock data in the data sublists; the multiple data sub-tables are sub-tables obtained by performing horizontal splitting processing on a single database table by taking a first routing field as a dimension;
a determining module, configured to determine, when the adjustment progress state is in a data migration state, a first table identifier reallocated to each piece of data in the inventory data based on the first routing field, the second routing field, and the obtained migration log;
and the processing module is used for migrating each data to a first data sublist corresponding to the first sublist identifier.
An embodiment of the present application provides an adjustment system, including:
a memory for storing executable instructions; a processor, when executing executable instructions stored in the memory, implements the method described above.
The embodiment of the application provides a storage medium, which stores executable instructions and is used for causing a processor to implement the method when executed.
The embodiment of the application has the following beneficial effects: obtaining a plurality of data sub-tables to be adjusted, adjustment progress states of the plurality of data sub-tables, and a second routing field for redistributing stock data in the plurality of data sub-tables; the multiple data sub-tables are sub-tables obtained by performing horizontal splitting processing on a single database table by taking the first routing field as a dimension; under the condition that the adjustment progress state is in a data migration state, determining a first chart identifier redistributed for each data in the stock data based on the first routing field, the second routing field and the obtained migration log; migrating each data to a first data sublist corresponding to the first sublist identifier; that is to say, under the condition of not stopping the system service, the migration of the stock data in the multiple data sub-tables is all performed on the original table by the method that the same data does not need to be constructed based on the migration log and the adjustment progress state, that is, the stock data in the multiple data sub-tables is obtained by performing horizontal splitting processing on a single database table by taking the first route field as a dimension, and the stock data in the multiple data sub-tables is redistributed by taking the first route field as the dimension based on the migration log and the adjustment progress state and is migrated to the redistributed data sub-tables; therefore, dynamic migration in the spreadsheet data is realized, no perception is given to a user, no perception is given to other spreadsheets or systems related to a plurality of data spreadsheets, the code invasion is weak, the risk caused by massive revising of the code is reduced, the robustness and the stability of the system are improved, and meanwhile, the storage resources of the system are saved.
Drawings
Fig. 1 is an alternative architecture diagram of a terminal provided in an embodiment of the present application;
FIG. 2 is a schematic flow chart diagram illustrating an alternative method for adjusting sub-table data according to an embodiment of the present disclosure;
FIG. 3 is a schematic flow chart diagram illustrating an alternative method for adjusting sub-table data according to an embodiment of the present application;
FIG. 4 is a schematic flow chart diagram illustrating an alternative method for adjusting sub-table data according to an embodiment of the present application;
FIG. 5 is a schematic flow chart diagram illustrating an alternative method for adjusting sub-table data according to an embodiment of the present application;
FIG. 6 is a schematic flow chart diagram illustrating an alternative method for adjusting sub-table data according to an embodiment of the present application;
FIG. 7 is a schematic flow chart diagram illustrating an alternative method for adjusting sub-table data according to an embodiment of the present application;
FIG. 8 is a schematic flow chart diagram illustrating an alternative method for adjusting sub-table data according to an embodiment of the present application;
fig. 9 is an alternative flowchart of a method for adjusting sub-table data according to an embodiment of the present application.
Detailed Description
In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be described in further detail with reference to the accompanying drawings, the described embodiments should not be considered as limiting the present application, and all other embodiments obtained by a person of ordinary skill in the art without making creative efforts fall within the protection scope of the present application.
In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. Unless defined otherwise, all technical and scientific terms used in the examples of this application have the same meaning as commonly understood by one of ordinary skill in the art to which the examples of this application belong. The terminology used in the embodiments of the present application is for the purpose of describing the embodiments of the present application only and is not intended to be limiting of the present application.
In order to better understand the method for adjusting the table data provided in the embodiment of the present application, first, a method for adjusting table data in the background art and the related art of the method for adjusting table data is described:
in order to reduce the burden of a database and the size of a single database table and improve the data access performance, the single database table is horizontally split or vertically split according to a certain rule to obtain a plurality of data sub-tables with independent storage spaces, and the follow-up operations of increasing, deleting, modifying and checking the service data are all positioned in the corresponding data sub-tables to be carried out. Here, vertical splitting may be understood as splitting one database table into a plurality of sub-tables according to fields, each sub-table storing a part of the fields; the advantage of vertical splitting is to reduce the probability of locking the table, and to put the fields with high access frequency and the fields with low access frequency in different tables, the operation efficiency of the hot spot data can be fully exerted. The horizontal splitting can be understood as splitting data in one database table into a plurality of sub-tables according to a certain rule, wherein the structure of each sub-table is completely the same; the advantage of horizontal splitting is that the size of the original single table is greatly reduced, and each sub-table only stores a small part of data, thereby improving the data access performance. The present application relates to horizontally split database sub-tables.
In the process of horizontally splitting a database table to obtain sub-tables, firstly, a routing field and a routing rule need to be designed, namely how to distribute the corresponding relation between the business data and the sub-tables, once the corresponding relation is determined, the subsequent operations of increasing, deleting, modifying and checking the business data are positioned to the corresponding sub-tables to be carried out based on the routing field and the routing rule. However, as the service develops, the routing field designed in the earlier stage may no longer meet the access requirement of the service on the table data, and needs to be adjusted to another routing field, at this time, the data in the current table needs to be redistributed according to the new routing field, which involves the migration of stock data and the distribution process of newly added data.
In the first shutdown migration scheme, a machine is shut down firstly, the system is shut down completely to ensure that no new data enters, then a data migration program is used to read the data in each sub-table, a new sub-table is calculated according to a new routing field, the data of the original sub-table is deleted, and the data is inserted into the new sub-table. In the second isomorphic table double-writing scheme, a plurality of new tables with the same data structure are required to be created; secondly, all places of the system needing to write the table are modified, all write operations to the data are written into the old table according to the routing of the old field, and are also written into the new table according to the routing of the new field. Then, writing the read data in the old table into a new table by using a data migration program, judging the data according to the last update time of the data in the writing process, directly writing the data if the data does not exist in the new table, and checking the last update time if the data exists in the new table; if the new data of the old table is written in an overwriting mode, the new data of the new table is abandoned, and meanwhile the data in the old table is deleted.
However, with the first approach, service outages for various types of application systems, especially financial systems, significantly affect user experience. By using the second mode, although the operation can be performed without shutdown, redundant isomorphic data tables with different names from those before need to be created, more database resources are occupied, and meanwhile, the change and the invasiveness of the service codes are large; not only new data needs to be written into two tables at the same time, but also query, modify and delete operations need to be carried out by double-table operation, on one hand, the possibility of transaction overtime is increased, and the response stability of the system is influenced; on the other hand, the same data are distributed in different tables, so that the data migration program is prone to errors, the problems of data duplication or data coverage are caused, and the data are not easy to check. In addition, the migration procedure is also complex to implement, and besides data consistency is considered, data is guaranteed not to be repeated, and external response is not perceived by a user. In addition, if a downstream system depends on an old sub-table, the downstream system may also need to be modified synchronously, and in order not to affect the use of the downstream system, the table name of a new table needs to be renamed to be the old table name after the migration is completed, the data of the old table needs to be cleaned, the storage resource is released, and renaming the table name is a time-consuming operation, and the real-time transaction is affected.
An exemplary application of the adjustment system provided in the embodiment of the present application is described below, and the adjustment system provided in the embodiment of the present application may be implemented as any terminal having an on-screen display function, such as a notebook computer, a tablet computer, a desktop computer, and an intelligent robot, or may be implemented as a server. In the following, an exemplary application will be explained when the adaptation system is implemented as a terminal.
Referring to fig. 1, fig. 1 is a schematic structural diagram of a terminal 100 according to an embodiment of the present application, where the terminal 100 shown in fig. 1 includes: at least one processor 110, at least one network interface 120, a user interface 130, and memory 150. The various components in terminal 100 are coupled together by a bus system 140. It is understood that the bus system 140 is used to enable connected communication between these components. The bus system 140 includes a power bus, a control bus, and a status signal bus in addition to a data bus. For clarity of illustration, however, the various buses are labeled as bus system 140 in fig. 1.
The Processor 110 may be an integrated circuit chip having Signal processing capabilities, such as a general purpose Processor, a Digital Signal Processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like, wherein the general purpose Processor may be a microprocessor or any conventional Processor, or the like.
The user interface 130 includes one or more output devices 131, including one or more speakers and/or one or more visual display screens, that enable the presentation of media content. The user interface 130 also includes one or more input devices 132 including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.
The memory 150 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid state memory, hard disk drives, optical disk drives, and the like. Memory 150 optionally includes one or more storage devices physically located remotely from processor 110. The memory 150 includes volatile memory or nonvolatile memory, and may include both volatile and nonvolatile memory. The nonvolatile Memory may be a Read Only Memory (ROM), and the volatile Memory may be a Random Access Memory (RAM). The memory 150 described in embodiments herein is intended to comprise any suitable type of memory. In some embodiments, memory 150 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or subsets or supersets thereof, as exemplified below.
An operating system 151 including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;
a network communication module 152 for communicating to other computing devices via one or more (wired or wireless) network interfaces 120, exemplary network interfaces 120 including: bluetooth, wireless compatibility authentication (WiFi), and Universal Serial Bus (USB), etc.;
an input processing module 153 for detecting one or more user inputs or interactions from one of the one or more input devices 132 and translating the detected inputs or interactions.
In some embodiments, the apparatus provided in this embodiment of the present application may be implemented in software, and fig. 1 illustrates a table data adjusting apparatus 154 stored in the memory 150, where the table data adjusting apparatus 154 may be a table data adjusting apparatus in the terminal 100, which may be software in the form of programs and plug-ins, and includes the following software modules: the obtaining module 1541, the determining module 1542, the processing module 1543 and the receiving module 1544 are logical and thus may be arbitrarily combined or further divided according to the implemented functions. The functions of the respective modules will be explained below.
In other embodiments, the apparatus provided in this embodiment of the present Application may be implemented in hardware, and for example, the apparatus provided in this embodiment of the present Application may be a processor in the form of a hardware decoding processor, which is programmed to execute the sub-table data adjusting method provided in this embodiment of the present Application, for example, the processor in the form of the hardware decoding processor may employ one or more Application Specific Integrated Circuits (ASICs), DSPs, programmable Logic Devices (PLDs), complex Programmable Logic Devices (CPLDs), field-Programmable Gate arrays (FPGAs), or other electronic components.
The method for adjusting the sub-table data provided by the embodiment of the present application will be described below with reference to an exemplary application and implementation of the terminal 100 provided by the embodiment of the present application. Referring to fig. 2, fig. 2 is an alternative flow chart diagram of a sub-table data adjustment method provided in the embodiment of the present application, applied to an adjustment system, which will be described with reference to the steps shown in fig. 2,
step 201, obtaining a plurality of data sub-tables to be adjusted, adjustment progress states of the plurality of data sub-tables, and a second routing field for redistributing stock data in the plurality of data sub-tables.
The multiple data sub-tables are obtained by performing horizontal splitting processing on a single database table by taking the first routing field as a dimension.
In the embodiment of the present application, a single database table is a data table for storing data generated by a service end. The business end may be various application systems or application programs, for example, the business end is a financial system such as a bank system, transaction data stored in the database table is inventory data in the scheme of the embodiment of the present application, for example, the business end is a shopping system, and order data stored in the database table is inventory data in the scheme of the embodiment of the present application, which is not limited specifically herein.
In the embodiment of the present application, the routing field may be any field name in the database table. For example, in the case of a financial system, referring to table 1, table 1 is a single database table, and the data in table 1 is data in the single database table. The routing field may be any field name in table 1, that is, the routing field may be an identity number, the routing field may also be a nationality, and of course, the routing field may also be a transaction type.
Name (I) Identity card number Nationality Type of transaction Transaction time
Zhang San 1231 Chinese family Transferring accounts 22-01-02
Zhang San 1231 Chinese family Deposit of money 22-01-05
Li Si 1227 Manchu nationality Drawing money 21-07-16
Zhao Wu 1452 Hui nationality Deposit of money 21-08-14
Wang Liu 8945 Mongolian Transfer accounts 22-02-09
TABLE 1
In the embodiment of the application, the number of the data sublists is predetermined, the multiple data sublists take a first routing field as a dimension, a first field value corresponding to the first routing field in each data in a single database table is processed according to a routing rule, a sublist sequence number corresponding to the first field value is determined, and each data is stored in the data sublist corresponding to the sublist sequence number.
In the embodiment of the application, the stock data is all data stored in the multiple data sub-tables.
In this embodiment of the application, the second routing field is any one of all routing fields included in the data sublist, except the first routing field, and the second routing field is used for reallocating the stock data in the multiple data sublists to a new data sublist with the field as a dimension. In the embodiment of the present application, the first routing field is an identification number, and the second routing field is a nation.
In the embodiment of the application, the adjustment progress state is used for representing the migration progress or the stage of stock data in the multiple data sub-tables, and the adjustment progress state can also ensure that the real-time operation of a user is reasonably matched with the migration of the data. The adjustment progress state comprises four states of an initialization state, a data migration state, a migration completed verification data state and a completed state. Wherein, the migration of the initialization state (INIT) representation data is not started, if the stock data is subjected to the operations of increasing, deleting, modifying and checking, the data of the field value corresponding to the first routing field needs to be positioned; the data migration state (SWITCH 1) represents that the migration program starts to redistribute the stock data; the migrated verification data state (SWITCH 2) represents that the data migration task is finished, but additional verification needs to be carried out on the migrated data; the completed state (FINISH) represents that the migration process of the whole data has ended, and if the data storage quantity is subjected to the operations of adding, deleting, changing and checking, the data needs to be located by the field value corresponding to the second routing field.
In other embodiments of the present application, the adjustment progress state may be placed in a Java Virtual Machine (JVM) cache when the adjustment system is started, and when the adjustment progress state changes, the adjustment system may automatically refresh the sensing, thereby well controlling the adjustment progress. Here, the switching between the different adjustment progress states described above has the following rule:
and switching from the initialization state to a data migration state, wherein the switching of the states is manually adjusted by operation and maintenance personnel to inform an adjusting system that the inventory data in the data sub-table can be started to be migrated.
And switching from the data migration state to a migrated and verified data state, wherein the switching of the state is automatically completed by the system, and when the batch migration program completes migration, the progress adjusting state is automatically modified so that operation and maintenance personnel can judge that the data migration is completed through the state, and further sample and check the migrated data.
And switching from the state of the migrated verification data to the state of the migrated verification data, wherein the state is manually adjusted by operation and maintenance personnel, and the operation and maintenance personnel perform sampling check on the migrated data and call the state into a FINISH state after determining that no problem exists completely so as to inform an adjustment system that all subsequent database operations can be calculated according to the second routing field without additional judgment.
And step 202, under the condition that the adjustment progress state is in a data migration state, determining a first chart identifier redistributed for each piece of data in the stock data based on the first routing field, the second routing field and the obtained migration log.
In the embodiment of the present application, the migration log is used to record migration information or migration records of migrated data. Wherein, the migration information includes but is not limited to: an nth field value corresponding to the first routing field, an mth field value corresponding to the second routing field, and a primary key identifier for marking the first data. Illustratively, the migration log is shown with reference to Table 2,
field value Whether or not it can be empty Main key Remarks for note
Value of 1 st field Whether or not Primary key identification 1 The value of the first routing field
Value of 2 nd field Whether or not Primary key identification 1 The value of the field corresponding to the second routing field
Time of operation Is that Time of operation
TABLE 2
In the embodiment of the application, after the adjustment system obtains a plurality of data sublists to be adjusted and adjustment progress states of the data sublists and a second routing field for redistributing stock data in the data sublists, whether the adjustment progress states of the data sublists are in a data migration state is judged, and when the adjustment progress states are in the data migration state, a first branch table identifier redistributed for each piece of data in the stock data is determined based on a first routing field for horizontally splitting a single database table, a second routing field for redistributing the stock data in the data sublists and a migration log.
Step 203, migrating each data to a first data sublist corresponding to the first sublist identifier.
In the embodiment of the application, when the adjustment system determines the first branch table identifier to be reallocated for each piece of data in the stock data, each piece of data in the stock data is reinserted into the first data branch table corresponding to the first branch table identifier from the current storage position, so that the reallocation process of each piece of data in the stock data with the second routing field as the dimension is completed.
The embodiment of the application provides a sub-table data adjusting method, which comprises the steps of obtaining a plurality of sub-tables of data to be adjusted, adjusting progress states of the plurality of sub-tables of data, and a second routing field for redistributing stock data in the plurality of sub-tables of data; the multiple data sub-tables are sub-tables obtained by performing horizontal splitting processing on a single database table by taking the first routing field as a dimension; under the condition that the adjustment progress state is in a data migration state, determining a first chart identifier redistributed for each data in the stock data based on the first routing field, the second routing field and the obtained migration log; migrating each data to a first data sublist corresponding to the first sublist identifier; that is to say, in the case of not stopping the system service, the same data does not need to be constructed, and the stock data in the multiple data sub-tables are migrated on the original table based on the migration log and the adjustment progress state, that is, the stock data in the multiple data sub-tables, which is obtained by performing horizontal splitting processing on a single database table with the first routing field as a dimension, is redistributed to the stock data in the multiple data sub-tables with the first routing field as the dimension based on the migration log and the adjustment progress state, and is migrated to the redistributed data sub-tables; therefore, dynamic migration in the spreadsheet data is realized, the user is not aware of the dynamic migration, other spreadsheets or systems related to the multiple data spreadsheets are not aware of the dynamic migration, the code invasion is weak, the risk caused by massive revising of the code is reduced, the robustness and the stability of the system are improved, and meanwhile, the system storage resources are saved.
Referring to fig. 3, fig. 3 is an alternative flowchart of a sub-table data adjustment method provided in the embodiment of the present application, which will be described with reference to the steps shown in fig. 3,
step 301, obtaining a plurality of data sublists to be adjusted, adjustment progress states of the plurality of data sublists, and a second routing field for redistributing stock data in the plurality of data sublists.
The plurality of data sub-tables are sub-tables obtained by performing horizontal splitting processing on a single database table by taking the first routing field as a dimension.
Step 302, when the adjustment progress state is in the data migration state, selecting first data of an nth field value corresponding to the first routing field and an mth field value corresponding to the second routing field from the multiple data branch tables.
N is greater than or equal to 1 and less than or equal to N, N is an integer, and N is the number of field values corresponding to the first routing field; m is more than or equal to 1 and less than or equal to M, M is an integer, and M is the number of field values corresponding to the second routing field.
In the embodiment of the application, the first data includes the data of the nth field value corresponding to the first routing field and the mth field value corresponding to the second routing field, the first data may be one, and the first data may also be multiple; each data includes first data.
In the embodiment of the application, after the adjustment system obtains a plurality of data sublists to be adjusted and adjustment progress states of the data sublists and a second routing field for redistributing stock data in the data sublists, whether the adjustment progress states of the data sublists are in a data migration state is judged, and when the adjustment progress states are in the data migration state, first data of an nth field value corresponding to a first routing field and an mth field value corresponding to a second routing field are selected from the data sublists.
In an implementation application scenario, as shown in table 1, the first routing field is used as an identity card number, the second routing field is used as a race for explanation, the number of the identity card number corresponding to the identity card number is 4, the identity card number includes a 1 st identity card number (1231), a 2 nd identity card number (1227), a 3 rd identity card number (1452) and a 4 th identity card number (8945), the number of the race corresponding to the race is 4, and the race includes a 1 st race value (han race), a 2 nd race value (han race), a 3 rd race value (hui race) and a 4 th race value (mongol race). At this time, the adjustment system selects the first data with the identification number value of 1 st and the ethnic value of 1 st from the multiple data sub-tables.
Step 303, under the condition that the migration log does not have migration information of the first data, calculating the mth field value according to the routing rule, and determining the mth field value as the first sub-table identifier redistributed to the first data.
In the embodiment of the application, the sub-table identifier is used for identifying each data sub-table, and the sub-table identifier may be a sub-table serial number of each data sub-table.
In the embodiment of the present application, the routing rule is used to obtain the sub-table identifier of the data sub-table according to the routing field, and the routing rule is also called a routing algorithm or a routing rule calculation module, and the routing algorithm includes, but is not limited to, a Hash (Hash) modulo algorithm and a Range algorithm. The Hash modulo algorithm is performed by computing the Hash value of the routing field and then modulo the determined total number of sub-tables. The Range algorithm assigns a sub-table identity based on the Range of routing field values. The application is particularly limited with respect to the use of routing algorithms.
In the embodiment of the application, under the condition that the adjustment progress state is in the data migration state, first data of an nth field value corresponding to a first routing field and an mth field value corresponding to a second routing field are selected from a plurality of data branch tables; and further, judging whether the migration log has the migration information of the first data or not, if the migration log is determined not to have the migration information of the first data, calculating the mth field value according to the routing rule, and determining the mth field value as the first sub-table identifier redistributed to the first data.
As can be seen from the above, in the embodiment of the present application, under the condition that the adjustment progress state is determined to be in the data migration state, at least one piece of first data of an nth field value corresponding to a first routing field and an mth field value corresponding to a second routing field is located in multiple data sublists, and if a migration record of the first data does not have migration information, it is characterized that the at least one piece of first data has not been migrated or has not been migrated completely, at this time, a calculation is performed on a field value corresponding to the second routing field according to a routing rule, and a new sublist identifier redistributed for the first data is determined, so that under the condition that a system does not stop service, the routing field is dynamically adjusted, and the field value of the first data determined in the multiple data sublists with the old routing field as a dimension is migrated to a data sublist with the new routing field as a dimension, and the new sublist identifier corresponds to the data sublist.
In other embodiments of the present application, when migration information of the first data exists in the migration log, it is described that reallocation has been completed for migration of the first data, and the next data may be processed without processing the piece of data.
Step 304, migrating the first data to a first data sublist corresponding to the first sublist identifier.
In the embodiment of the application, after first data of an nth field value corresponding to a first routing field and an mth field value corresponding to a second routing field are selected from a plurality of data sublists, an initial sublist identifier corresponding to an initial data sublist where the first data is located is obtained, and if the first sublist identifier is the same as the initial sublist identifier, the first data does not need to be migrated; if the first sub-table identifier is different from the initial sub-table identifier, the first data is inserted into the first data sub-table corresponding to the first sub-table identifier, and the first data is deleted from the initial data sub-table corresponding to the initial sub-table identifier, so that the dynamic migration of the first data with the second routing field as a dimensionality is completed, meanwhile, the dynamic migration is not sensitive to a user, other sub-tables or systems related to the plurality of data sub-tables is not sensitive, the code invasion is weak, the risk caused by massive code revising is reduced, the robustness and stability of the system are improved, and meanwhile, system storage resources are saved.
Step 305, registering migration information of the first data in the migration log.
Wherein the migration information includes: an nth field value corresponding to the first routing field, an mth field value corresponding to the second routing field, and a primary key identification for tagging the first data.
Here, the migration information may further include an operation time on the data, and whether a field value may be null, which is not particularly limited in this application.
It is emphasized that, when there is one first data, after the first data is migrated, migration information of the first data is registered in the migration log; when there are a plurality of first data, one piece of migration information for all the first data is registered in the migration log after all the first data migration is completed.
In an implementation scenario, referring to table 3, table 3 is one of a plurality of data sub-tables,
name (I) Identity card number Nationality Type of transaction Transaction time
Zhang San 1231 Han nationality Transferring accounts 22-01-02
Zhang San 1231 Han nationality Deposit of money 22-01-05
Zhao Wu 1452 Hui nationality Deposit of money 21-08-14
Wang Liu 8945 Mongolian Transferring accounts 22-02-09
TABLE 3
When a piece of first data { Zhan, 1231, han nationality, transfer, 22-01-02} with the identity card number value as the 1 st identity card number value and the ethnic group value as the 1 st ethnic group value is selected from the multi-piece data sublist, and after the data is transferred to the first data sublist, whether other residual data with the identity card number value as the 1 st identity card number value and the ethnic group value as the 1 st ethnic group value exist is judged; if no residual data is determined, registering migration information of the first data in the migration log to inform an adjustment system, and performing subsequent adding, deleting, changing and checking operation on the first data with the identity card value of 1 st and the ethnic value of 1 st ethnic value, wherein the score table identification can be calculated according to the ethnic value of 1 st. If it is determined that the remaining data exists, such as { Zhang III, 1231, han nationality, deposit, 22-01-05}, the next data is processed continuously until the identity card value is the 1 st identity card value and all the first data with the ethnic value of 1 st ethnic value are completely migrated, and then a piece of migration information for all the first data is registered in the migration log. Therefore, the migrated data is registered in the migration log, the waste of system resources caused by repeated migration is avoided, meanwhile, the migration log provides reference for the sub-table where the data of the real-time access request of the user is located, the sub-table where the access request of the user is located is rapidly located, and the operation and maintenance efficiency is improved.
And step 306, after all stock data migration is completed, adjusting the adjustment progress state from the data migration state to a migration completed verification data state, and verifying data in all the data sublists.
In the embodiment of the application, after migration of all stock data is completed, redistribution of the stock data is completed, the adjustment system automatically updates the adjustment progress state from the data migration state to the migrated verification data state, and verifies data in all data branch tables to prevent migration errors in the migration process.
In some embodiments, the steps 201 to 203, or the steps 301 to 306 provided in the above embodiments may also be implemented by combining with an operation request of a user for a database, where a timing of the operation request of the user for the database may be before data in a plurality of data sublists is migrated (or redistributed), a timing of the operation request of the user for the database may also be in a process of migrating (or redistributing) data in the plurality of data sublists, and a timing of the operation request of the user for the database may also be after data in the plurality of data sublists has been migrated (or redistributed), so the application is not particularly limited; here, the embodiment of the present application further describes the table data adjustment method in steps 201 to 203 or steps 301 to 306 provided in the above embodiment in combination with an operation request of a user on a database, referring to fig. 4, where fig. 4 is an optional flowchart of the table data adjustment method provided in the embodiment of the present application, and the following description is made in combination with fig. 2 and fig. 4, or in combination with the steps shown in fig. 3 and fig. 4,
step 401, receiving an operation request for the database, and analyzing the operation request to obtain an operation type for operating the database table to be operated.
The operation request carries a first field value corresponding to a first routing field contained in the data to be operated, a second field value corresponding to a second routing field and identification information.
In the embodiment of the application, the operation request is a request for real-time operation on data in a database, and the operation request carries a first field value, a second field value and identification information of the data to be operated.
In the embodiment of the present application, the operation types include add (insert), delete (delete), modify (update), and query (select).
In the embodiment of the application, the identification information is used for representing the unique identification of each data in the plurality of data sublists.
In the embodiment of the application, after receiving the operation request for the database, the adjustment system analyzes the operation request to obtain the operation type for operating the database table to be operated, so that the adjustment system determines to select the first routing field or the second routing field according to different operation types, and locates a plurality of data sub-tables corresponding to the database table.
In other embodiments of the present application, in the process of processing the operation request, after the adjustment system analyzes the operation request to obtain the operation type for operating the database table, the interceptor may also lock the data in the database table in the operation request and intercept the operations of increasing, deleting, modifying and checking the data. Here, the interceptor may intercept related operations on the database based on an Aspect Organized Programming (AOP) principle, and perform related additional logic before or after the database operations are finally performed. The interceptor may be a Mapper interceptor, and the interceptor may also be a plug-in interceptor of MyBatis, which is not limited in this application.
Here, the process of parsing the operation request in step 401 to obtain the operation type of the database table to be operated is further described with reference to fig. 5,
step 501, obtaining an operation code carried by the operation request.
In the embodiment of the application, the operation code is used for uniquely identifying the information of the corresponding instruction content, and the operation code at least has a mapping relation with the operation instruction and the operation type. And determining the database table to be operated by the user and the operation type for operating the database table to be operated based on the instruction content.
Step 502, finding out the operation type having a mapping relation with the operation code from the instruction configuration mapping relation table.
In the embodiment of the application, after the operation code carried by the operation request is obtained, the mapping record having the unique mapping relation with the operation code is searched from the instruction configuration mapping relation table, that is, the instruction content and the operation type having the mapping relation with the operation code are searched. Therefore, the operation request carries a small amount of data for positioning the target number, such as operation codes, and the operation type and the operation instruction which have a mapping relation with the operation codes are quickly searched from the instruction configuration mapping relation table according to the operation codes so as to carry out actual operation on the target data according to the operation type and the operation instruction.
For example, referring to table 4, table 4 shows an instruction configuration mapping relationship table having a mapping relationship between an operation code and an instruction content and an operation type, if an operation code carried in an operation request to a database is received and is update00001, a 3 rd mapping record having a unique mapping relationship with the operation code can be obtained from the instruction configuration mapping relationship table, that is, table 4, and the instruction content and the operation type in the 3 rd mapping record are obtained, a database table to be operated is determined according to the instruction content, and an update operation is performed on data in the database table according to the operation type.
Operation code (unique) Instruction content …… Type of operation
select00001 select xxx Query
delete00001 delete xxx Deleting
update00001 update xxx Updating
update00002 update xxx Updating
…… …… …… ……
insert00001 insert xxx Increase of
insert00002 insert xxx Increase in
TABLE 4
In other embodiments of the present application, the adjustment system may also cache the operation type corresponding to the operation code in the memory, and directly obtain the operation type corresponding to the operation request from the memory when the same operation request for the database is received again, so that the operation request does not need to be repeatedly analyzed, which wastes resources.
And 402, responding to the operation request, selecting different field values based on the adjustment progress state, and determining a second data branch table corresponding to the selected field values from the multiple data branch tables.
Wherein the different field values include a first field value and a second field value.
In the embodiment of the application, the adjustment system analyzes the operation request to obtain the operation type of the operation on the database table to be operated, responds to the operation request to obtain the adjustment progress states of the multiple data sublists corresponding to the database table at present, selects different field values based on the adjustment progress states, and determines a second data sublist corresponding to the selected field value from the multiple data sublists.
In other embodiments of the present application, the process of selecting different field values based on the adjustment progress status in step 402, determining a second data sub-table corresponding to the selected field values from the plurality of data sub-tables is further described,
and under the condition that the adjustment progress state is in an initialization state, selecting a first field value, calculating the first field value according to a routing rule, determining a second table identifier, and determining a second data table corresponding to the second table identifier.
In the embodiment of the application, when the adjustment progress state is in an initialization state, the inventory data in the data sub-tables corresponding to the database table is represented that the migration or redistribution is not started; at the moment, stock data and newly added data in the data sublist are split by taking the first routing field as a dimensionality; therefore, the operation type of the data to be operated corresponding to the operation request, no matter whether the data is added, deleted, modified or inquired, can be positioned according to the field value corresponding to the first routing field; therefore, the adjusting system selects a first field value corresponding to the first routing field, calculates the first field value according to the routing rule to obtain a second sub-table identifier corresponding to the first field value, and determines a second data sub-table corresponding to the second sub-table identifier. At this time, if the operation type is deletion, modification or query, after the underlying data sublist where the data to be operated is located, the data to be operated corresponding to the identification information is quickly located in the second data sublist (or the underlying data sublist); if the operation type is increased, after the underlying data sublist in which the data to be operated is to be stored is located, the data to be operated corresponding to the identification information is added in the second data sublist (or the underlying data sublist). Therefore, under the condition of ensuring the data consistency, the dynamic migration in the data of the tables is realized, the data is not repeated, the user is not aware of the data, other tables or systems related to a plurality of data tables are not aware of the data, the code invasion is weak, the risk caused by the massive revising of the codes is reduced, the robustness and the stability of the system are improved, and the system storage resources are saved.
And under the condition that the adjustment progress state is in a finished state, selecting a second field value, calculating the second field value according to the routing rule, determining a third table identifier, and determining a second data sublist corresponding to the third table identifier.
In the embodiment of the application, when the adjustment progress state is in the completed state, the completion of migration or redistribution of stock data in the plurality of data sub-tables corresponding to the database table is represented; at the moment, the stock data and the newly added data in the data sublist are split by taking the second routing field as a dimensionality; therefore, the operation type of the data to be operated corresponding to the operation request, whether the data is added, deleted, modified or inquired, can be positioned according to the field value corresponding to the second routing field; therefore, the adjustment system selects a second field value corresponding to the second routing field, calculates the second field value according to the routing rule to obtain a third packet identifier corresponding to the second field value, and determines a second data packet corresponding to the third packet identifier. At this time, if the operation type is deletion, modification or query, the underlying data sublist where the data to be operated is located, and the data to be operated corresponding to the identification information is quickly located in the second data sublist (or the underlying data sublist); if the operation type is increased, after the underlying data sublist in which the data to be operated is to be stored is located, the data to be operated corresponding to the identification information is added in the second data sublist (or the underlying data sublist). Therefore, under the condition of ensuring the data consistency, the dynamic migration in the spreadsheet data is realized, the data is not repeated, the user is not aware of the data, other spreadsheets or systems associated with a plurality of data spreadsheets are not aware of the data, the code invasion is weak, the risk caused by the massive revising of the codes is reduced, the robustness and the stability of the system are improved, and the system storage resources are saved.
And under the condition that the adjustment progress state is in a data migration state or a data migration completion verification state, if a first field value and a second field value exist in the migration log, selecting the second field value, calculating the second field value according to a routing rule, determining a third table identifier, and determining a second data sublist corresponding to the third table identifier.
In the embodiment of the application, when the adjustment progress state is in a data migration state or a migration completed verification data state, and a first field value and a second field value exist in a migration log, migration completion or redistribution completion of at least part of data in a plurality of data sub-tables corresponding to a database table is represented; at least part of data and newly added data in the data sublist are split by taking the second routing field as a dimension; therefore, the operation type of the data to be operated corresponding to the operation request, no matter whether the data is added, deleted, modified or inquired, can be positioned according to the field value corresponding to the second routing field; therefore, the adjustment system selects a second field value corresponding to the second routing field, calculates the second field value according to the routing rule to obtain a third packet identifier corresponding to the second field value, and determines a second data packet corresponding to the third packet identifier. At this time, if the operation type is deletion, modification or query, after the underlying data sublist where the data to be operated is located, the data to be operated corresponding to the identification information is quickly located in the second data sublist (or the underlying data sublist); if the operation type is increased, after the underlying data sublist where the data to be operated is located, the data to be operated corresponding to the identification information is added in the second data sublist (or the underlying data sublist). Therefore, under the condition of ensuring the data consistency, the dynamic migration in the spreadsheet data is realized, the data is not repeated, the user is not aware of the data, other spreadsheets or systems associated with a plurality of data spreadsheets are not aware of the data, the code invasion is weak, the risk caused by the massive revising of the codes is reduced, the robustness and the stability of the system are improved, and the system storage resources are saved.
And under the condition that the adjustment progress state is in a data migration state or the migration completion verification data state, if the operation type is one of an inquiry operation type, a modification operation type and a deletion operation type and a first field value and a second field value do not exist in the migration log, selecting the first field value, calculating the first field value according to a routing rule, determining a second table identifier, and determining a second data table corresponding to the second table identifier.
In the embodiment of the application, when the adjustment progress state is in a data migration state or a data migration completion verification state, the operation type is one of query, modification and deletion, and a first field value and a second field value do not exist in a migration log, it is indicated that at least part of data in a plurality of data sublists corresponding to a database table does not start to be migrated or redistributed; at least part of data in the data sublist is still split by taking the first routing field as a dimension; therefore, the operation type of the data to be operated corresponding to the operation request, no matter deletion, modification or query, can be positioned according to the field value corresponding to the first routing field; therefore, the adjusting system selects a first field value corresponding to the first routing field, calculates the first field value according to the routing rule to obtain a second sub-table identifier corresponding to the first field value, and determines a second data sub-table corresponding to the second sub-table identifier; namely, when the operation type is deletion, modification or query, after the underlying data sublist where the data to be operated is located, the data to be operated corresponding to the identification information is quickly located in the second data sublist (or the underlying data sublist). Therefore, under the condition of ensuring the data consistency, the dynamic migration in the spreadsheet data is realized, the data is not repeated, the user is not aware of the data, other spreadsheets or systems associated with a plurality of data spreadsheets are not aware of the data, the code invasion is weak, the risk caused by the massive revising of the codes is reduced, the robustness and the stability of the system are improved, and the system storage resources are saved.
Under the condition that the adjustment progress state is in a data migration state or a data migration completion verification state, if the operation type is increased, the first field value and the second field value do not exist in the migration log, and the data which is not migrated does not have residual data including the first field value, calculating the second field value according to the routing rule, determining a third table identifier, determining a second data sublist corresponding to the third table identifier, adding data to be operated in the second data sublist, and registering migration information of the data to be operated in the migration log.
In the embodiment of the application, when the adjustment progress state is in a data migration state or a migrated verification data state, the operation type is increased, a first field value and a second field value do not exist in a migration log, and residual data including the first field value does not exist in non-migrated data, the increased data to be operated is brand new data, and no stock data needs to be migrated; the adjustment system selects a second field value corresponding to the second routing field, calculates the second field value according to the routing rule to obtain a third packet identifier corresponding to the second field value, and determines a second data packet corresponding to the third packet identifier; further, after the underlying data sublist where the data to be operated is located, the data to be operated corresponding to the identification information is added to the second data sublist (or the underlying data sublist). Finally, the migration information of the data to be operated is directly registered on the migration log, so that the data to be operated does not need to be processed any more, and the data to be operated is redistributed according to the second field value.
And under the condition that the adjustment progress state is in a data migration state or the data migration is completed and verification is performed, if the operation type is increased, the first field value and the second field value do not exist in the migration log, and residual data exist in the data which is not migrated, calculating the first field value according to the routing rule to obtain a second table identifier, and determining a second data table corresponding to the second table identifier.
In the embodiment of the application, when the adjustment progress state is in a data migration state or a migration completed verification data state, the operation type is increased, a first field value and a second field value do not exist in the migration log, and residual data including the first field value exists in non-migrated data, the migration or redistribution of at least part of data in a plurality of data sub-tables corresponding to the database table is not started; at least part of data in the data sublist is still split by taking the first routing field as a dimension; therefore, the added operation type of the data to be operated corresponding to the operation request can be positioned according to the field value corresponding to the first routing field; therefore, the adjusting system selects a first field value corresponding to the first routing field, calculates the first field value according to the routing rule to obtain a second sub-table identifier corresponding to the first field value, and determines a second data sub-table corresponding to the second sub-table identifier so as to add data to be operated corresponding to the identifier information in the second data sub-table (or the bottom data sub-table). Therefore, under the condition of ensuring the data consistency, the dynamic migration in the spreadsheet data is realized, the data is not repeated, the user is not aware of the data, other spreadsheets or systems associated with a plurality of data spreadsheets are not aware of the data, the code invasion is weak, the risk caused by the massive revising of the codes is reduced, the robustness and the stability of the system are improved, and the system storage resources are saved.
And step 403, performing operation processing on the data to be operated corresponding to the identification information according to the operation type in the second data sublist.
In the embodiment of the application, different field values are selected based on the adjustment progress state, after a second data sublist corresponding to the selected field values is determined from the multiple data sublists, operation processing is performed on to-be-operated data corresponding to the identification information in the second data sublist according to the operation type. Of course, after the data to be operated is processed, the operation processing result may also be output, and the operation processing result includes, but is not limited to, processing success and processing failure.
Here, referring to fig. 6, the overall scheme of online adjustment and migration of the sub-table data provided in the present application is further described.
Step 601, after the adjustment system is started, loading the adjustment progress mark parameters into a cache, so that the adjustment system automatically obtains the refreshing perception, and determining an adjustment progress switch (the adjustment progress state) of the multiple data sub-tables.
Step 602, after receiving the real-time online transaction request (the operation request mentioned above), according to the service processing logic, analyzing the operation request to obtain the operation type for operating the database table to be operated.
Here, the online transaction request carries an old field value corresponding to an old routing field contained in the data to be operated, a new field value corresponding to a new routing field, and identification information. There is no sequence between step 601 and step 602.
And step 603, intercepting the operations of adding, deleting, modifying and checking a plurality of data sub-tables by utilizing a Mybatis interceptor.
And here, a Mybatis interceptor is created by utilizing the AOP principle to intercept the increasing, deleting and modifying operations of a plurality of data sub-tables. The multiple data sub-tables are sub-tables obtained by performing horizontal splitting processing on a single database table by taking an old routing field (the first routing field) as a dimension.
Step 604, obtaining the adjustment progress switches of the multiple data sub-tables, the new routing field (the second routing field mentioned above) for redistributing the stock data in the multiple data sub-tables, and the migration log. Further, based on different operation types, adjustment progress switches, new routing fields, old routing fields and migration logs, a bottom-layer data sublist (the second data sublist) corresponding to the online transaction request is determined from the multiple data sublists through the routing rule calculation module. And in the bottom data sublist, operating and processing the data to be operated corresponding to the identification information carried in the online transaction request according to the operation type.
It should be noted that, the transaction timing in this process is not different from that before, and the execution time of the database transaction is not increased. Meanwhile, the batch migration module is responsible for redistributing the stock data according to a new field value corresponding to the new routing field, and the migrated data registers a migration log according to the new field value and the old field value for judging and using the online transaction. And the online and batch perfect matching is carried out, and the online adjustment and data migration of the routing field are finally completed.
It should be noted that, the processing modes of the routing rule calculation modules are different for different operation types of the database table. Here, referring to fig. 7 and 8, fig. 7 is a processing flow of the routing rule calculation module when the operation type is query, modification, or deletion, and fig. 8 is a processing flow of the routing rule calculation module when the operation type is addition.
Here, with reference to fig. 6 and fig. 7, for the operation type of query, modification or deletion, it should be noted that, no new data is added in the query, modification or deletion operation, and the operation is directed at the stock data, so that the data in the bottom layer sublist can be directly located according to the rule of the routing calculation module; here, in the processing logic of the transaction, the new and old field values are all brought into the underlying calculation, and a calculation module calculates the sub-table by using a certain field value according to the routing rule. In particular, the amount of the solvent to be used,
and 701, under the condition that an old field value corresponding to the old routing field and a new field value corresponding to the new routing field are obtained, judging the state of the current progress adjustment switch.
In this embodiment of the application, if the adjustment progress SWITCH is in the SWITCH1 state or the SWITCH2 state, step 702 is executed; if the adjustment progress switch is in the INIT state, then step 703 is executed; if the adjustment progress switch is in FINISH state, go to step 704.
Step 702, judging whether a new field value and an old field value exist in the migration log, if not, executing step 703; if so, go to step 704.
And 703, positioning the bottom-layer data sub-table in which the data to be operated is positioned from the multiple data sub-tables according to the old field value.
Here, the adjustment progress switch is in an INIT state, which indicates that the stock data has not started to migrate or redistribute, and the stock data is a data branch table distributed according to the old field value, so that the data can be located according to the old field value in the query, deletion and update at the moment.
Here, when the adjustment progress SWITCH is in any of SWITCH1 or SWITCH2, it indicates that the program in the background batch migration module has started running or just finished running, and at this time, in which data sublist the data to be operated needs to be operated in the online transaction request, the data sublist cannot be directly calculated with the new field value or the old field value, because the data that may be accessed has already been migrated, or the data has not yet been completely migrated, at this time, an intermediate recording table (migration log) is needed to judge, which data has already been migrated is stored in the intermediate table in cooperation with the online batch. Here, if there is no record in the migration log for the new field value and the old field value sent in the transaction request, it means that the data to be operated has not been migrated, and the data can be located only according to the old field value, so that the data needs to be located according to the old field value for the query, modification and deletion.
And step 704, positioning the bottom-layer data sublist where the data to be operated is located from the multiple data sublists according to the new field value.
Here, the adjustment progress switch is in the FINISH state, the whole routing field switching process is completely implemented, that is, the stock data is migrated or redistributed, and all data including the newly added data are redistributed into the data sub-table according to the new field value, so that the data can be directly positioned according to the new field value.
Here, when the adjustment progress SWITCH is in either of SWITCH1 or SWITCH2 states, if there is a record in the migration log of the new field value and the old field value sent in the transaction request, it indicates that all the stock data of the new and old field values have been reallocated according to the new field value, and the query, modification and deletion can be directed to the data with the new field value.
Here, with reference to fig. 6 and 8, for the operation type being increase, it should be noted that, unlike the handling of stock data such as query, modification, deletion, and the like, if newly added data is uniformly routed according to the old routing field, the amount of migrated data may be increased when migration reallocation is subsequently performed; of course, the routing cannot be completely performed according to the new routing field, otherwise, data of the same old field value may not be in the same branch table, and other transactions are affected, for example, a query transaction may miss a part of data. Thus, special processing is required for the new operation. In particular, the amount of the solvent to be used,
step 801, under the condition that an old field value corresponding to an old routing field and a new field value corresponding to a new routing field are obtained, judging the state of the current adjustment progress switch.
In the embodiment of the present application, if the adjustment progress switch is in the INIT state or the FINISH state, step 805 is executed; if the adjustment progress SWITCH is in the SWITCH1 state or the SWITCH2 state, step 802 is executed.
Step 802, judging whether a new field value and an old field value in the newly added data exist in the migration log, if not, executing step 803; if so, go to step 805.
Step 803, judging whether the plurality of data sub-tables have residual non-migrated data containing new field values and old field values, if not, executing step 804; if so, go to step 805.
Step 804, registering the migration information of the new data in the migration log.
Here, when the adjustment progress SWITCH is in any one of the SWITCH1 and SWITCH2 states, the batch migration module is migrating the stock data, and at this time, if new data comes in, it is first necessary to determine whether other remaining data of the old field value of the new data has been migrated according to the migration log. And if no residual data of the non-migrated data is migrated, directly delivering the newly added data to a route calculation module to calculate the data sublist according to the new field value. That is, the newly added data has no remaining data based on the old field value, which indicates that the newly added data is a new data and no remaining data needs to be migrated, so that the migration log is directly registered. If the data is left, the data is sent to the route rule calculation module for calculation, and the data sub-table is calculated according to the old field value.
And step 805, positioning a bottom layer data sub-table in which the newly added data is to be stored from the plurality of data sub-tables according to the route calculation module.
The adjustment progress switch is in an INIT state, stock data do not start to migrate or redistribute, the stock data are distributed according to the old field value, therefore, the newly added operation locates the data sub-table according to the old field value, the newly added data fall into the calculated sub-table, and after the subsequent data migration or data redistribution is started, the batch migration program is responsible for the migration and redistribution.
Here, the adjustment progress switch is in the FINISH state, and the whole routing field switching process is completely implemented, that is, the stock data is migrated and redistributed, so that the newly added operation in this state directly locates the data sub-table according to the new field value, the newly added data falls into the sub-table obtained by calculation, and the data is calculated according to the new field value in the subsequent inquiry, deletion and update of the newly added data.
Here, how to process the online transaction request before data migration, during data migration, and after data migration is completed, as described in fig. 7 and fig. 8, here, with reference to fig. 6 and fig. 9, in a case where the adjustment progress SWITCH is in the SWITCH1 state, in order to accelerate the migration speed, a general batch migration module performs parallel processing according to a certain dimension segment, here, taking an old field value corresponding to an old routing field as a and a new field value corresponding to a new routing field as B as an example, a migration processing flow in which stock data in a plurality of data segment tables is redistributed according to a new field value corresponding to a new routing field is exemplarily described,
and step 901, under the condition of obtaining an old field value corresponding to the old routing field and a new field value corresponding to the new routing field, fishing out all first data of which the old field value is equal to A in the stock data, and performing circular processing.
Here, all records with old field value equal to a require the same transaction to complete migration, otherwise query transaction may be affected, and therefore, all third data with old field value equal to a are first fished and then processed one by one.
Here, assume that the new field value of the retrieved record is B.
Step 902, if the processing is finished, execute step 910; if not, go to step 903.
Step 903, any third data is retrieved from all the third data and locked, and the new field value of the third data is B.
Step 904, judging whether migration information with a new field value of B and an old field value of A exists in the migration log, if so, executing step 910; if not, go to step 905.
Step 905, calculating the new field value B according to the routing rule, and determining the first table identifier of the third data to be fished.
Here, the old field value a is calculated according to the routing rule, and the initial sublist identity of the third data to be fished is determined.
Step 906, judging whether the initial sub-table identification is consistent with the first sub-table identification, if so, returning to the step 902; if not, go to step 907.
Step 907, inserting the third data obtained by the fishing into the first data sublist corresponding to the first sublist identifier, and deleting the third data obtained by the fishing from the initial data sublist corresponding to the initial sublist identifier.
Step 908, judging whether other data with the new route value equal to B still exist in all the third data, if yes, returning to step 902; if not, step 909 is executed.
In step 909, migration information of all the third data is registered in the migration log.
At step 910, the process is complete.
It should be emphasized that when the batch migration task completes the redistribution of all the stock data, the adjustment progress SWITCH is automatically updated to SWITCH2 for the online transaction and the judgment and implementation stage of the operation and maintenance personnel.
Continuing with the exemplary structure of the sub-table data adjusting apparatus 154 provided in the embodiments of the present application implemented as a software module, in some embodiments, as shown in fig. 1, the software module stored in the sub-table data adjusting apparatus 154 of the memory 150 may be a sub-table data adjusting apparatus in the terminal 100, including:
an obtaining module 1541, configured to obtain a plurality of data sublists to be adjusted, adjustment progress states of the plurality of data sublists, and a second routing field for reallocating stock data in the plurality of data sublists; the multiple data sub-tables are sub-tables obtained by performing horizontal splitting processing on a single database table by taking the first routing field as a dimension;
a determining module 1542, configured to determine, when the adjustment progress state is in the data migration state, a first chart identifier reallocated to each piece of data in the inventory data based on the first routing field, the second routing field, and the obtained migration log;
the processing module 1543 is configured to migrate each piece of data into the first data sublist corresponding to the first sublist identifier.
In some embodiments, the processing module 1543 is further configured to select, from the multiple data sub-tables, first data of an nth field value corresponding to the first routing field and an mth field value corresponding to the second routing field; n is more than or equal to 1 and less than or equal to N, N is an integer, and N is the number of field values corresponding to the first routing field; m is more than or equal to 1 and less than or equal to M, M is an integer, and M is the number of field values corresponding to the second routing field; the determining module 1542 is further configured to, when the migration log does not have migration information of the first data, calculate an m-th field value according to a routing rule, and determine a first table identifier reallocated for the first data.
In some embodiments, the processing module 1543 is further configured to register migration information of the first data in the migration log; wherein the migration information includes: an nth field value corresponding to the first routing field, an mth field value corresponding to the second routing field, and a primary key identification for tagging the first data.
In some embodiments, the processing module 1543 is further configured to adjust the adjustment progress status from the data migration status to a migration completed verification data status, and verify data in all data sublists.
In some embodiments, the receiving module 1544 is configured to receive an operation request for the database, and parse the operation request to obtain an operation type for operating the database table to be operated; the operation request carries a first field value corresponding to a first routing field contained in the data to be operated, a second field value corresponding to a second routing field and identification information; the processing module 1543 is further configured to select, in response to the operation request, a different field value based on the adjustment progress status, and determine, from the plurality of data partial tables, a second data partial table corresponding to the selected field value; wherein the different field values include a first field value and a second field value; and in the second data sublist, performing operation processing on the data to be operated corresponding to the identification information according to the operation type.
In some embodiments, the obtaining module 1541 is further configured to obtain an operation code carried in the operation request; the processing module 1543 is further configured to search, from the instruction configuration mapping relationship table, an operation type having a mapping relationship with the operation code.
In some embodiments, the determining module 1542 is further configured to, in a case that the adjustment progress state is in an initialization state, select a first field value, calculate the first field value according to a routing rule, determine a second sub-table identifier, and determine a second data sub-table corresponding to the second sub-table identifier; under the condition that the adjustment progress state is in a finished state, selecting a second field value, calculating the second field value according to a routing rule, determining a third table identifier, and determining a second data sublist corresponding to the third table identifier; and under the condition that the adjustment progress state is in a data migration state or a data migration completion verification state, if a first field value and a second field value exist in the migration log, selecting the second field value, calculating the second field value according to a routing rule, determining a third table identifier, and determining a second data sublist corresponding to the third table identifier.
In some embodiments, if the operation type is one of a query operation type, a modify operation type, and a delete operation type, the determining module 1542 is further configured to select a first field value if the first field value and a second field value do not exist in the migration log, calculate the first field value according to the routing rule, determine a second sublist identifier, and determine a second data sublist corresponding to the second sublist identifier.
In some embodiments, the operation type is an adding operation type, the determining module 1542 is further configured to, if the first field value and the second field value do not exist in the migration log and the remaining data including the first field value does not exist in the non-migrated data, calculate the second field value according to the routing rule, determine a third table identifier, determine a second data sublist corresponding to the third table identifier, add the data to be operated in the second data sublist, and register migration information of the data to be operated in the migration log; if the first field value and the second field value do not exist in the migration log and the residual data exists in the data which is not migrated, calculating the first field value according to the routing rule to obtain a second sub-table identifier, and determining a second data sub-table corresponding to the second sub-table identifier.
It should be noted that the description of the apparatus in the embodiment of the present application is similar to the description of the method embodiment, and has similar beneficial effects to the method embodiment, and therefore, the description is not repeated. For technical details not disclosed in the embodiments of the apparatus, reference is made to the description of the embodiments of the method of the present application for understanding.
Embodiments of the present application provide a storage medium having stored therein executable instructions that, when executed by a processor, cause the processor to perform a method provided by embodiments of the present application, for example, the method as illustrated in fig. 2-5.
The storage medium obtains a plurality of data sublists to be adjusted, adjustment progress states of the data sublists and a second routing field for redistributing stock data in the data sublists; the multiple data sub-tables are sub-tables obtained by performing horizontal splitting processing on a single database table by taking the first routing field as a dimension; under the condition that the adjustment progress state is in a data migration state, determining a first chart identifier redistributed for each data in the stock data based on the first routing field, the second routing field and the obtained migration log; migrating each data to a first data sublist corresponding to the first sublist identifier; that is to say, under the condition of not stopping the system service, the migration of the stock data in the multiple data sub-tables is all performed on the original table by the method that the same data does not need to be constructed based on the migration log and the adjustment progress state, that is, the stock data in the multiple data sub-tables is obtained by performing horizontal splitting processing on a single database table by taking the first route field as a dimension, and the stock data in the multiple data sub-tables is redistributed by taking the first route field as the dimension based on the migration log and the adjustment progress state and is migrated to the redistributed data sub-tables; therefore, dynamic migration in the spreadsheet data is realized, the user is not aware of the dynamic migration, other spreadsheets or systems related to the multiple data spreadsheets are not aware of the dynamic migration, the code invasion is weak, the risk caused by massive revising of the code is reduced, the robustness and the stability of the system are improved, and meanwhile, the system storage resources are saved.
In some embodiments, the storage medium may be a computer-readable storage medium, such as a Memory, for example, a Ferroelectric Random Access Memory (FRAM), a Read Only Memory (ROM), a Programmable Read Only Memory (PROM), an Erasable Programmable Read Only Memory (EPROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a flash Memory, a magnetic surface Memory, an optical disc, or a Compact disc Read Only Memory (CD-ROM), etc.; or may be various devices including one or any combination of the above memories.
In some embodiments, the executable instructions may be in the form of a program, software module, script, or code written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
By way of example, executable instructions may correspond, but do not necessarily have to correspond, to files in a file system, and may be stored in a portion of a file that holds other programs or data, such as in one or more scripts in a hypertext Markup Language (hypertext Markup Language) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code). By way of example, executable instructions may be deployed to be executed on one computing device or on multiple computing devices at one site or distributed across multiple sites and interconnected by a communication network.
The above description is only an example of the present application, and is not intended to limit the scope of the present application. Any modification, equivalent replacement, and improvement made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims (11)

1. A method for adjusting tabular data, comprising:
acquiring a plurality of data sublists to be adjusted, adjustment progress states of the data sublists and a second routing field for redistributing stock data in the data sublists; the multiple data sub-tables are sub-tables obtained by performing horizontal splitting processing on a single database table by taking a first routing field as a dimension;
under the condition that the adjustment progress state is in a data migration state, determining a first chart identifier redistributed for each data in the stock data based on the first routing field, the second routing field and the obtained migration log;
and migrating each data to a first data sublist corresponding to the first sublist identifier.
2. The method of claim 1, wherein said determining a first table identification to reallocate for each of said inventory data based on said first routing field, said second routing field, and an obtained migration log comprises:
selecting first data of an nth field value corresponding to the first routing field and an mth field value corresponding to the second routing field from the multiple data sublists; n is greater than or equal to 1 and less than or equal to N, N is an integer, and N is the number of field values corresponding to the first routing field; m is more than or equal to 1 and less than or equal to M, M is an integer, and M is the number of field values corresponding to the second routing field;
and under the condition that the migration log does not have migration information of the first data, calculating the mth field value according to a routing rule, and determining the first sub-table identifier redistributed for the first data.
3. The method of claim 2, wherein after migrating said each data into the first data sub-table corresponding to said first sub-table identifier, said method comprises:
registering the migration information of the first data in the migration log;
wherein the migration information comprises: an nth field value corresponding to the first routing field, an mth field value corresponding to the second routing field, and a primary key identifier for marking the first data.
4. The method of claim 1, wherein after migrating each data into the first data sub-table corresponding to the first sub-table identifier, the method comprises:
and adjusting the adjustment progress state from the data migration state to a migrated verification data state, and verifying data in all the data sublists.
5. The method according to any one of claims 1 to 4, further comprising:
receiving an operation request for the database, and analyzing the operation request to obtain an operation type for operating the database table to be operated; the operation request carries a first field value corresponding to the first routing field, a second field value corresponding to the second routing field and identification information, wherein the first field value, the second field value and the identification information are contained in data to be operated;
responding to the operation request, selecting different field values based on the adjustment progress state, and determining a second data sublist corresponding to the selected field values from the multiple data sublists; wherein the different field values include the first field value and the second field value;
and in the second data sublist, performing operation processing on the data to be operated corresponding to the identification information according to the operation type.
6. The method of claim 5, wherein parsing the operation request to obtain an operation type of the database table to be operated on comprises:
obtaining an operation code carried by the operation request;
and searching the operation type having a mapping relation with the operation code from an instruction configuration mapping relation table.
7. The method of claim 5, wherein selecting a different routing field based on the adjustment progress status, and determining a second data sub-table corresponding to the selected routing field from the plurality of data sub-tables comprises:
under the condition that the adjustment progress state is in an initialization state, selecting the first field value, calculating the first field value according to a routing rule, determining a second sub-table identifier, and determining a second data sub-table corresponding to the second sub-table identifier;
under the condition that the adjustment progress state is in a finished state, selecting the second field value, calculating the second field value according to the routing rule, determining a third table identifier, and determining the second data sublist corresponding to the third table identifier;
and under the condition that the adjustment progress state is in the data migration state or the migration completion verification data state, if the first field value and the second field value exist in the migration log, selecting the second field value, calculating the second field value according to the routing rule, determining the third table identifier, and determining the second data sublist corresponding to the third table identifier.
8. The method of claim 7, wherein if the operation type is one of a query operation type, a modify operation type, and a delete operation type, the method further comprises:
if the first field value and the second field value do not exist in the migration log, the first field value is selected, the first field value is calculated according to the routing rule, the second table division identification is determined, and the second data sub-table corresponding to the second table division identification is determined.
9. The method of claim 7, wherein the operation type is an add operation type, the method further comprising:
if the first field value and the second field value do not exist in the migration log and the non-migrated data does not include the remaining data including the first field value, calculating the second field value according to the routing rule, determining the third table identifier, determining the second data sublist corresponding to the third table identifier, adding the data to be operated in the second data sublist, and registering the migration information of the data to be operated in the migration log;
if the first field value and the second field value do not exist in the migration log and the residual data exists in the data which is not migrated, calculating the first field value according to the routing rule to obtain a second sub-table identifier, and determining the second data sub-table corresponding to the second sub-table identifier.
10. An adjustment system, comprising:
a memory for storing executable instructions; a processor for implementing the method of any one of claims 1 to 9 when executing executable instructions stored in the memory.
11. A storage medium having stored thereon executable instructions for causing a processor to perform the method of any one of claims 1 to 9 when executed.
CN202211062652.9A 2022-08-31 2022-08-31 Method and system for adjusting sub-table data and storage medium Pending CN115982119A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211062652.9A CN115982119A (en) 2022-08-31 2022-08-31 Method and system for adjusting sub-table data and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211062652.9A CN115982119A (en) 2022-08-31 2022-08-31 Method and system for adjusting sub-table data and storage medium

Publications (1)

Publication Number Publication Date
CN115982119A true CN115982119A (en) 2023-04-18

Family

ID=85961767

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211062652.9A Pending CN115982119A (en) 2022-08-31 2022-08-31 Method and system for adjusting sub-table data and storage medium

Country Status (1)

Country Link
CN (1) CN115982119A (en)

Similar Documents

Publication Publication Date Title
US11593337B2 (en) Data processing method, device, and a storage medium
CN110209650B (en) Data normalization and migration method and device, computer equipment and storage medium
CN109710190B (en) Data storage method, device, equipment and storage medium
US20120158795A1 (en) Entity triggers for materialized view maintenance
US20040267782A1 (en) Database system
CN108319656A (en) Realize the method, apparatus and calculate node and system that gray scale is issued
KR20200146006A (en) Method, apparatus, device and storage medium for upgrading application
US9977804B2 (en) Index updates using parallel and hybrid execution
CN108536745B (en) Shell-based data table extraction method, terminal, equipment and storage medium
US10747777B2 (en) Computer system and transaction processing management method
US10338910B2 (en) Multi-tenant upgrading
US9355106B2 (en) Sensor data locating
US11100047B2 (en) Method, device and computer program product for deleting snapshots
CN110019111B (en) Data processing method, data processing device, storage medium and processor
CN110084486B (en) Resource management method and device
CN112306993A (en) Data reading method, device and equipment based on Redis and readable storage medium
CN110795447A (en) Data processing method, data processing system, electronic device, and medium
CN112039975A (en) Method, device, equipment and storage medium for processing message field
US9588884B2 (en) Systems and methods for in-place reorganization of device storage
CN104317660A (en) Bank parameter managing system
CN112199391A (en) Data locking detection method and device and computer readable storage medium
CN111522827A (en) Data updating method and device and electronic equipment
CN115982119A (en) Method and system for adjusting sub-table data and storage medium
US11954328B2 (en) Storage management device, storage management method, and program
CN112817931B (en) Incremental version file generation method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination