CN110941602B - Database configuration method and device, electronic equipment and storage medium - Google Patents
Database configuration method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN110941602B CN110941602B CN201911140830.3A CN201911140830A CN110941602B CN 110941602 B CN110941602 B CN 110941602B CN 201911140830 A CN201911140830 A CN 201911140830A CN 110941602 B CN110941602 B CN 110941602B
- Authority
- CN
- China
- Prior art keywords
- day
- data amount
- data
- database
- daily
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a configuration method, a device, electronic equipment and a storage medium of a database, wherein the method comprises the following steps: acquiring total data volume on the nth day and total data volume on the nth-1 day in a database to determine the data volume on the nth day and daily average growth data volume, wherein N is a positive integer greater than or equal to 1; determining the data volume of the (N+1) th day according to the data volume of the (N) th day and the daily gain data volume; determining the number of groups, libraries, and tables required based on the data volume on day n+1 and a predetermined rule; the database is configured by performing a group-by-group-list operation based on the determined number of required groups, libraries, and lists, each list including a predetermined number of sequence numbers, to store the amount of data on day n+1. The invention can avoid the large amount of data migration work after expanding the volume database and the table number, and can reduce the resource consumption of parallel query data.
Description
Technical Field
The present invention relates to the field of data processing, and in particular, to a method and apparatus for configuring a database, an electronic device, and a storage medium.
Background
With the rapid increase of the payment and receipt traffic of the bank agent, the data volume of the payment and receipt system of the bank agent is greatly increased. Along with the shorter and shorter processing time requirements of customers on the pay-and-receive service system, the pay-and-receive system of the bank gradually develops from the single application architecture to the distributed system, and the implementation of a database and table splitting mechanism is faced in the modification process of the distributed system.
At present, IT industry has various database and table dividing mechanisms, such as performing database and table dividing according to hash values (hash) of partition keys or modulo arithmetic, but these methods are all fixed in configuration database number and table dividing number, and do not have the characteristic of flexibility. Although the database and table dividing mechanism is used more mature in the distributed system, after the capacity of the database and the table number is expanded along with the increase of the traffic, the migration work after the data redistribution is increased, and even the parallel query of the service system is brought with performance pressure.
Therefore, the banking and pay-per-view system mainly has the following technical problems in the implementation process: (1) With the increase of the traffic, after the implementation of the sub-database sub-tables, when the database and the table number are transversely increased, the redistribution of the data brings great migration challenges to the implementation of the project, and the continuity of the service is hindered; (2) When the database and the table number are transversely adjusted, hysteresis is generated by comparing the manual judgment with the adjustment, so that a larger pressure is brought to the stable operation of the system.
Disclosure of Invention
In view of the above, the present invention provides a method, an apparatus, an electronic device and a storage medium for configuring a database, so as to solve at least one of the above-mentioned problems.
According to a first aspect of the present invention, there is provided a method of configuring a database, the method comprising: acquiring total data volume on the nth day and total data volume on the nth-1 day in a database to determine the data volume on the nth day and daily average growth data volume, wherein N is a positive integer greater than or equal to 1; determining the data volume of the (n+1) th day according to the data volume of the (N) th day and the daily gain data volume; determining the number of groups, libraries and tables required according to the data amount on the n+1th day and a predetermined rule; and performing a grouping library grouping operation according to the determined number of the required groups, libraries and tables to configure the database to store the data amount of the (n+1) th day, wherein each table comprises a predetermined number of serial numbers.
According to a second aspect of the present invention, there is provided a configuration apparatus of a database, the apparatus comprising: the data quantity determining unit is used for obtaining the total data quantity on the nth day and the total data quantity on the nth-1 day in the database so as to determine the data quantity on the nth day and the daily average growth data quantity, wherein N is a positive integer greater than or equal to 1; a data amount prediction unit configured to determine a data amount on the n+1th day according to the data amount on the nth day and the average daily growth data amount; a library table prediction unit for determining the number of required groups, libraries and tables according to the data amount of the n+1th day and a predetermined rule; and a database configuration unit for performing a grouping-base grouping operation according to the determined number of the required groups, base and tables to configure the database to store the data amount of the n+1th day, wherein each table includes a predetermined number of serial numbers.
According to a third aspect of the present invention there is provided an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the method of configuring a database as described above when the program is executed.
According to a fourth aspect of the present invention there is provided a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of the above-described method of configuring a database.
According to the technical scheme, the data amount of the next day is determined according to the daily data amount and the daily average growth data amount determined by the acquired daily total data amount of the database, so that the number of groups, libraries and tables required for storing the data amount of the next day can be determined, the database is configured by carrying out grouping library grouping operation according to the determined number of the required groups, libraries and tables, so as to store the data amount of the next day, and the database configuration is dynamically adjusted by analyzing the growth trend of the data amount.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a database configuration method according to an embodiment of the invention;
FIG. 2 is a schematic diagram of grouping information, repository information and table information according to an embodiment of the present invention;
FIG. 3 is a block diagram of a database configuration apparatus according to an embodiment of the present invention;
FIG. 4 is a block diagram of the daily gain data amount determination module 311 according to an embodiment of the present invention;
FIG. 5 is a detailed block diagram of a database configuration apparatus according to an embodiment of the present invention;
FIG. 6 is an exemplary block diagram of a database configuration apparatus according to an embodiment of the present invention;
fig. 7 is a schematic diagram of an electronic device according to an embodiment of the invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Because the prior art database and table dividing mechanism is to pre-fix and distribute the number of database and table dividing, when the number of databases and tables is increased, a large amount of data migration work is brought to data redistribution. At present, a technical scheme for flexibly expanding the capacity of a database and a table according to the increasing trend of the traffic is lacking, which brings greater performance consumption to the application scene of parallel query data.
Based on this, the embodiment of the invention provides a database configuration scheme, which adjusts the database and table division configuration by analyzing the business (or called operation) data growth trend, and performs the database and table division operation according to the business data quantity pre-judgment, so that a large amount of data migration work after the capacity of the database and the table number are expanded can be avoided, and the resource consumption of parallel query data can be reduced. Embodiments of the present invention are described in detail below with reference to the accompanying drawings.
FIG. 1 is a flowchart of a method for configuring a database according to an embodiment of the present invention, as shown in FIG. 1, the method including:
Specifically, the data amount on the nth day is the difference between the total data amount on the nth day and the total data amount on the N-1 th day.
The amount of daily gain data is determined by: firstly, determining daily data amount in a preset period (for example, 3 days) according to the total data amount on the N day and the total data amount on the N-1 day in a database; the daily increase data amount is then determined from the absolute increase data amount between the daily data amounts.
For example, the data amount on day N-1 is a, the data amount on day N is b, the data amount on day N+1 is c, and the daily increase data amount on these days is [ |a-b|+|c-b| ]/2, where||represents the absolute value.
And 102, determining the data volume of the (n+1) th day according to the data volume of the (N) th day and the average daily growth data volume.
And adding the data volume on the N th day and the daily gain data volume to obtain a numerical value which is the data volume on the (N+1) th day.
And 104, performing a grouping library grouping operation according to the determined number of the required groups, libraries and tables to configure the database so as to store the data volume of the (n+1) th day, wherein each table comprises a preset number of serial numbers.
The data amount of the next day is determined (or called prediction) according to the daily data amount and the daily growth data amount determined by the acquired daily total data amount of the database, so that the number of groups, libraries and tables required for storing the data amount of the next day can be determined, the database is configured by carrying out grouping library grouping operation according to the determined number of the required groups, libraries and tables, so as to store the data amount of the next day, and the database configuration is dynamically adjusted by analyzing the increasing trend of the data amount.
In an embodiment of the present invention, an increase in the amount of data in the database corresponds to an increase in business (or job) data, each of which may be stored in a corresponding table of the database. In actual operation, each table may store the same predetermined number of jobs as the serial number, each job data including a job identification (e.g., job serial number) and job content, the job identification corresponding to one serial number.
In actual operation, the serial number refers to a serial number of a payment-collecting operation, and the serial number value is generated in an incremental manner and is used for uniquely corresponding to a serial number of the payment-collecting operation.
When the job data is stored in the position corresponding to the serial number, the corresponding relation between the job identification (for example, the serial number) and the corresponding serial number can be established and stored, so that the corresponding job data can be conveniently searched from the database table according to the job identification in the later period, and the resource consumption of the parallel query data is reduced.
Specifically, the job serial number is randomly generated by the service system, and the generation rule is not uniform and globally unique. Therefore, the mapping relation between the serial number and the serial number of the operation serial number is established, and the operation serial number is cached by a distributed caching technology, so that the subsequent quick inquiry can be facilitated. When the user inquires the operation flow information, the corresponding serial number is inquired from the distributed cache according to the operation flow number, and then the operation flow information is inquired from the library number and the table number corresponding to the serial number and returned to the user.
In the embodiment of the invention, the operation flow information is stored in the database, so that the change of the operation quantity can be obtained through the increasing trend of the data quantity in the database. Here, the data amount in the database refers to the number of database table records of the database table of the service system for realizing the database and table division. For example, when the collection service flow water meter TB performs the operation of sorting the database and the table, the data size of the flow water meter TB is the sum of the numbers of records of all TB tables of all databases after sorting the database and the table.
Specifically, grouping may be performed by date, and then the data amount of the previous day=the sum of the record numbers of all table names under all library numbers under the group number (group number of the previous day grouping).
For example, the group number on the day is 20190102, the library number of the group number 20190101 is 1, 2, 3, and each library includes three tables, and the table names are Tb001, tb002, and Tb003, then the data amount on the previous day (20190101) =the number of table names Tb001 record count+the number of table names Tb002 record count+the number of table names Tb003 record count.
In the implementation process, when the n+1st day is a special day (for example, a special holiday), the data amount may change obviously, and at this time, the data amount of the n+1st day may be determined according to the data amount of the N day, the daily average growth data amount, and the annual average growth data amount of the special day.
The amount of data for the above-described annual average increase for a particular day may be determined by: firstly, determining the data quantity of a special day every year according to the total data quantity of a database every day; the annual average growth data amount for the particular day is then determined based on the data amount for that particular day each year.
That is, for the statistical daily data amount, the data amount of the next day is analyzed by linear regression as y=rx1 (x 1 represents the number of days the system operates), and the data amount of the next day is calculated as y. In addition, from the special day delta trend per year (e.g., delta t=mx2, x2 is the number of years the system is running), it is analyzed that the data amount for a special day should be t=y+t.
For example, if the special day is 1 month and 5 days each year, the data amount of the special day in 4 years (may be 3 years, 5 years, 6 years, etc.), the present invention is not limited thereto, is acquired:
the data amount n1=63 of 20150105;
the data amount n2=68 of 20160105;
the data amount n3=62 of 20170105;
the data amount n4=69 of 20180105.
The annual average of a particular day increases by an amount of data t= [ |n2-n1|+|n3-n2|+|n4-n3| ]/3, where||represents an absolute value.
And 20190101 data amount t1=95;
20190102 data amount t2=100;
20190103 data amount t3=90;
20190104 data amount t4=97;
then, the data amount of 20190105 =t4+|t2-t1+|t2-t3+|t4-t3| >/3+|n2-n1+|n2-n3+|n4-n3| >/3.
The data amount of 20190105 before 4 days is acquired here, and in actual operation, 5 days, 6 days, 7 days, etc. may be acquired, and may be selected according to actual conditions, and the present invention is not limited thereto.
For data quantity prediction of special days, the service response to the market can be met more flexibly by comprehensively considering the data growth trend and the growth quantity of the special days.
In actual operation, the distributed database may include groups, libraries, and tables, and the information configuring the database includes: grouping information, database information and table information.
For a better understanding of the embodiments of the present invention, the following details are described in connection with the grouping information, the database information and the table information shown in fig. 2, as shown in fig. 2:
(1) The grouping information includes: group number, group name, service system number, processing unit number, start sequence number, stop sequence number, and service (or job) data packet range information representing the service system.
Fig. 2 shows 2 groups, and for a group with a group number of 1, the grouping information thereof specifically includes:
group number: 1, a step of;
group name: g01;
service system number: b001 (pay-as-you-pay service);
processing unit number: SPU01 (a distributed service system, the actual system physical deployment unit, i.e., the node that can handle the service);
starting sequence number: 2019010100000000;
cut-off sequence number: 2019010199999999.
the above cut-off sequence numbers are merely exemplary and may be based on actual conditions.
(2) The library information comprises: the database information represents the business (or operation) data grouping and then distributes the business (or operation) data grouping into corresponding databases according to the hash values, wherein the hash values can be the values of serial number modulo operation.
With continued reference to fig. 2, assuming that the number of sub-banks of the service system number B001 is 3 and the number of sub-tables of each bank is 3, taking group number=1 as an example, the sub-bank information includes:
the library numbers are 1, 2 and 3;
the reservoir names are respectively Db01, db02 and Db03;
the group number is 1;
the hash value is the value modulo the sequence number: if the serial number mapped by the service serial number (102001 e456bfs 789) is 201901010000007, the serial number modulo operation is 201901010000007% 9=7, and the hash value of the service serial number (102001 e456bfs 789) is 7.
(3) The sub-table information includes: table number, library number, start sequence number, stop sequence number. The sub-table information indicates that packet service data is distributed to corresponding table numbers in accordance with the number of sub-tables, and a sequence number range of a certain table number distribution is specified.
For example, taking the group name G01 as an example, the starting sequence number of this group is 2019010100000000, and the ending sequence number is 2019010199999999; the pool numbers belonging to the group names are 1, 2 and 3, the table numbers of each pool are 1, 2 and 3, and the pool names Db01 correspond to the table names Tb001, tb002 and Tb003.
In practice, the number of banks in each group, the number of tables in each bank, and the data amount threshold of each table may be set, and after predicting the data amount on day n+1 in step 102, the databases to be added, and the number of tables in each bank, may be calculated according to a predetermined rule (i.e., the thresholds of the banks, tables, and data in the tables), and then the group of component bank group table information may be established so as to store the data on day n+1.
For example, when the table number threshold of each library is set to 3 and the single table data amount threshold is set to 100 ten thousand, the predicted data amount t=30000000 on the n+1th day and the current database number is 3, then the number of databases to be added is required to be T/3000000-3=7. In actual operation, when the number of databases to be added is smaller than 0, the value is 0.
Then, based on the above information, for example, the following grouping database and table information can be established:
(1) The grouping information is as follows:
group number 5;
group name G05;
service system number: b001 (pay-as-you-pay service);
processing unit number: SPU01 (a distributed service system, the actual system physical deployment unit, i.e., the node that can handle the service);
a start sequence number 20190105000000000;
cut-off sequence number 20190105999999999.
(2) The library information is as follows, and the library numbers, the library names, the belonging group numbers and the hash values are sequentially as follows:
(3) The sub-table information is as follows, and is sequentially each table number, library number, starting sequence number and ending sequence number:
after the packet library sub-table information is established, the data on the (n+1) th day can be stored, when the job data is stored to the position corresponding to the serial number, the corresponding relation between the job identification (for example, the serial number) and the corresponding serial number is established and stored, so that the corresponding job data can be conveniently searched from the database table according to the job identification in the later period, a large amount of data migration work after the capacity expansion database and the table number can be avoided, and the resource consumption of parallel query data can be reduced.
Based on similar inventive concepts, the embodiments of the present invention also provide a configuration device of a database, preferably, the device is used to implement the foregoing method embodiments. Fig. 3 is a block diagram of the structure of the apparatus, as shown in fig. 3, including: a data amount determination unit 31, a data amount prediction unit 32, a library table prediction unit 33, and a database configuration unit 34, wherein:
a data amount determining unit 31, configured to obtain an nth day total data amount and an nth-1 day total data amount in the database, so as to determine an nth day data amount and a daily average growth data amount, where N is a positive integer greater than or equal to 1;
a data amount prediction unit 32 for determining the data amount on the n+1th day from the data amount on the nth day and the average daily gain data amount;
a library table prediction unit 33 for determining the number of required groups, libraries, and tables according to the data amount on the n+1th day and a predetermined rule;
a database configuration unit 34, configured to perform a grouping-base grouping operation according to the determined number of the required groups, bases and tables to configure the database to store the data amount of the n+1th day, wherein each table includes a predetermined number of serial numbers.
Each table is used for storing the predetermined number of jobs, and each job comprises a job identification and job content, wherein the job identification corresponds to a serial number.
The data amount of the next day is determined (or called predicted) by the data amount prediction unit 32 according to the daily data amount and daily increase data amount determined by the total daily data amount of the database obtained by the data amount determination unit 31, so that the database table prediction unit 33 can determine the number of groups, libraries and tables required for storing the data amount of the next day, then the database configuration unit 34 performs the grouping database sub-table operation according to the determined number of the required groups, libraries and tables to configure the database so as to store the data amount of the next day, and the database configuration is dynamically adjusted by analyzing the increasing trend of the data amount.
Specifically, the above-described data amount determination unit 31 includes a daily gain data amount determination module 311, and as shown in fig. 4, the daily gain data amount determination module 311 includes: a daily data amount determination sub-module 3111 and a daily growth data amount determination sub-module 3112, wherein:
a daily data amount determining sub-module 3111, configured to determine a daily data amount in a predetermined period according to an nth day total data amount and an nth-1 day total data amount in the database;
the daily growth data amount determination submodule 3112 is configured to determine the daily growth data amount from absolute growth data amounts between daily data amounts.
In an implementation process, as shown in fig. 5, the apparatus may further include: a special day data amount determination unit 35 and a special year growth data amount determination unit 36, wherein:
a special day data amount determining unit 35 configured to determine, when the n+1th day is a special day, a data amount of the special day per year based on a total data amount of each day of the database;
an average growth data amount determination unit 36 for determining an average growth data amount for a particular day from the data amount for the particular day each year.
At this time, the data amount prediction unit 32 is specifically configured to: and determining the data volume of the (n+1) th day according to the data volume of the (N) th day, the daily increase data volume and the annual increase data volume.
The data amount prediction unit 32 predicts the data amount for a particular day, and can more flexibly satisfy the response of the business to the market by comprehensively considering the data increasing trend and the increasing amount for the particular day.
With continued reference to fig. 5, the apparatus further includes: the correspondence establishing unit 37 is configured to establish and store a correspondence between the job identifier (for example, job serial number) and the corresponding serial number. When a user inquires the operation flow information, the corresponding serial number is required to be inquired from the distributed cache according to the operation flow number, and then the operation flow information is inquired from the library number and the table number corresponding to the serial number and returned to the user, so that the resource consumption of parallel inquiry data can be reduced.
The specific execution process of each unit, each module, and each sub-module may be referred to the description in the above method embodiment, and will not be repeated herein.
In actual operation, the units, the modules and the sub-modules may be combined or may be arranged singly, and the invention is not limited thereto.
Fig. 6 is an exemplary structural diagram of a database configuration apparatus according to an embodiment of the present invention, as shown in fig. 6, the apparatus including:
the data prediction center 61, the database-division-table configuration center 62, wherein the data prediction center 61 is used for predicting the data amount to be generated in the future according to the data amount in the database, so as to predict the required database capacity, the database-division-table configuration center 62 is mainly used for configuring grouping information, database-division information and table-division information, and the database and table are increased based on the database capacity predicted to be required by the data prediction center 61, and the grouping information, the database-division information and the table-division information are configured for storing the data to be generated.
After storing the data, a mapping relationship of the corresponding job identification (e.g., job serial number) and the corresponding serial number is established.
In the pay-and-pay distributed system, the job serial number is randomly generated by the service system, so that the generation rule is not uniform and globally unique. Therefore, by establishing the mapping relation between the operation serial number and the serial number and carrying out distributed cache on the mapping relation, when a user uses the operation serial number to inquire the service in an actual service scene, the user can quickly find the corresponding operation data from the database table, so that the system inquiry efficiency can be improved, and the resource consumption of the parallel inquiry data can be reduced.
By grouping the operation data according to a certain time sequence and a data generation sequence, the distributed system of the grouping database sub-table does not need to redistribute the data when the capacity is expanded according to service growth, so that the problem of data migration does not exist, and therefore, the embodiment of the invention can support the distributed system of the dynamic grouping database sub-table without data migration.
When a user performs service inquiry or maintenance operation on a service system, the sub-library and sub-table configuration center searches the sub-library and sub-table information according to the serial number of the service system serial number map, and the service system does not need to develop sub-library and sub-table logic when being developed, only needs to search in the sub-library and sub-table configuration center, so that decoupling of the sub-library and sub-table configuration center and the service system can be realized.
The embodiment of the invention predicts and dynamically adjusts the distribution of the sub-database sub-table according to the data volume growing trend, supports the distributed system by constructing a sub-database sub-table configuration center, and the mapping relation between the irregular operation serial number and the grouping database and table serial number is cached by using a distributed caching technology, so that a dynamic grouping database and table distributed system without data migration can be supported.
Fig. 7 is a schematic diagram of an electronic device according to an embodiment of the invention. The electronic device shown in fig. 7 is a general-purpose data processing apparatus comprising a general-purpose computer hardware structure including at least a processor 701 and a memory 702. The processor 701 and the memory 702 are connected by a bus 703. The memory 702 is adapted to store one or more instructions or programs executable by the processor 701. The one or more instructions or programs are executed by the processor 701 to implement the steps in the database configuration method described above.
The processor 701 may be a separate microprocessor or may be a set of one or more microprocessors. Thus, the processor 701 performs the process of the data and the control of other devices by executing the commands stored in the memory 702, thereby executing the method flow of the embodiment of the present invention as described above. The bus 703 connects the above-described components together, while connecting the above-described components to a display controller 704 and a display device, and an input/output (I/O) device 705. Input/output (I/O) device 705 may be a mouse, keyboard, modem, network interface, touch input device, somatosensory input device, printer, and other devices known in the art. Typically, input/output (I/O) devices 705 are connected to the system through input/output (I/O) controllers 706.
The memory 702 may store software components such as an operating system, communication modules, interaction modules, and application programs, among others. Each of the modules and applications described above corresponds to a set of executable program instructions that perform one or more functions and methods described in the embodiments of the invention.
The embodiment of the invention also provides a computer readable storage medium, on which a computer program is stored, which when being executed by a processor, implements the steps of the above-mentioned database configuration method.
In summary, the embodiment of the invention provides a configuration scheme of a database, which dynamically adjusts the configuration of sub-databases and sub-tables by analyzing the growth trend of service data, realizes the operation of sub-databases and sub-tables with service prejudgement, flexibly configures the service data to be stored in the positions of the database and the table numbers with definite rules by the sub-database sub-table configuration center, can avoid a large amount of data migration work after expanding the database and the table numbers, and is convenient for a later user to quickly find corresponding operation data from the database table according to the operation identification by establishing and caching the corresponding relation between the operation identification and the corresponding serial number, thereby reducing the resource consumption of parallel query data.
Preferred embodiments of the present invention are described above with reference to the accompanying drawings. The many features and advantages of the embodiments are apparent from the detailed specification, and thus, it is intended by the appended claims to cover all such features and advantages of the embodiments which fall within the true spirit and scope thereof. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the embodiments of the invention to the exact construction and operation illustrated and described, and accordingly, all suitable modifications and equivalents may be resorted to, falling within the scope thereof.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The principles and embodiments of the present invention have been described in detail with reference to specific examples, which are provided to facilitate understanding of the method and core ideas of the present invention; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present invention, the present description should not be construed as limiting the present invention in view of the above.
Claims (10)
1. A method for configuring a database, the method comprising:
acquiring total data volume on the nth day and total data volume on the nth-1 day in a database to determine the data volume on the nth day and daily average growth data volume, wherein N is a positive integer greater than or equal to 1;
determining the data volume of the (n+1) th day according to the data volume of the (N) th day and the daily gain data volume;
determining the number of groups, libraries and tables required according to the data amount on the n+1th day and a predetermined rule;
performing a grouping library grouping operation according to the determined number of the required groups, libraries, and tables to configure the database to store the data amount on the n+1th day, wherein each table includes a predetermined number of sequence numbers;
when the n+1th day is a special day, the method further comprises:
determining the data amount of the special day every year according to the total data amount of the database every day;
determining the annual average growth data volume of the special day according to the data volume of the special day every year;
determining the data amount on day n+1 from the data amount on day N and the daily gain data amount includes:
determining the data volume of the (n+1) th day according to the data volume of the (N) th day, the average daily growth data volume and the average annual growth data volume;
determining the data amount on the n+1th day according to the data amount on the nth day, the average daily growth data amount and the average annual growth data amount, including:
the sum of the data amount on the nth day, the average daily increase data amount, and the average annual increase data amount is determined as the data amount on the n+1th day.
2. The method of configuring a database according to claim 1, wherein the daily gain data amount is determined by:
determining daily data volume in a preset period according to the total data volume on the N th day and the total data volume on the N-1 th day in the database;
the daily increase data amount is determined from the absolute increase data amount between daily data amounts.
3. The method of configuring a database according to claim 1, wherein each table stores the predetermined number of jobs, each job including a job identification and job contents, the job identification corresponding to one serial number.
4. A method of configuring a database according to claim 3, wherein the method further comprises:
and establishing and storing the corresponding relation between the job identification and the corresponding serial number.
5. A database configuration apparatus, the apparatus comprising:
the data quantity determining unit is used for obtaining the total data quantity on the nth day and the total data quantity on the nth-1 day in the database so as to determine the data quantity on the nth day and the daily average growth data quantity, wherein N is a positive integer greater than or equal to 1;
a data amount prediction unit configured to determine a data amount on the n+1th day according to the data amount on the nth day and the average daily growth data amount;
a library table prediction unit for determining the number of required groups, libraries and tables according to the data amount of the n+1th day and a predetermined rule;
a database configuration unit configured to perform a grouping-library grouping operation according to the determined number of the required groups, libraries, and tables to configure the database to store the data amount of the n+1th day, wherein each table includes a predetermined number of serial numbers;
the apparatus further comprises:
a special day data amount determining unit configured to determine, when the n+1th day is a special day, a data amount of the special day every year according to a total data amount of each day of the database;
a special day annual average growth data amount determining unit configured to determine an annual average growth data amount of the special day based on the data amount of the special day every year;
the data amount prediction unit is specifically configured to:
determining the data volume of the (n+1) th day according to the data volume of the (N) th day, the average daily growth data volume and the average annual growth data volume;
the data amount prediction unit is specifically configured to:
the sum of the data amount on the nth day, the average daily increase data amount, and the average annual increase data amount is determined as the data amount on the n+1th day.
6. The apparatus according to claim 5, wherein the data amount determination unit includes: the daily gain data amount determining module,
the average daily gain data amount determining module comprises:
the daily data volume determining submodule is used for determining daily data volume in a preset period according to the total data volume of the N th day and the total data volume of the N-1 th day in the database;
a daily increase data amount determination submodule for determining the daily increase data amount according to the absolute increase data amount between the daily data amounts.
7. The database configuration apparatus according to claim 5, wherein each table stores the predetermined number of jobs, each job including a job identification and job content, the job identification corresponding to one serial number.
8. The database configuration device according to claim 7, wherein the device further comprises:
and the corresponding relation establishing unit is used for establishing and storing the corresponding relation between the job identification and the corresponding serial number.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the method for configuring a database according to any one of claims 1 to 4 when the program is executed.
10. A computer-readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, carries out the steps of the method for configuring a database according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911140830.3A CN110941602B (en) | 2019-11-20 | 2019-11-20 | Database configuration method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911140830.3A CN110941602B (en) | 2019-11-20 | 2019-11-20 | Database configuration method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110941602A CN110941602A (en) | 2020-03-31 |
| CN110941602B true CN110941602B (en) | 2023-05-12 |
Family
ID=69907146
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911140830.3A Active CN110941602B (en) | 2019-11-20 | 2019-11-20 | Database configuration method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110941602B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111680313B (en) * | 2020-06-08 | 2022-06-28 | 南京领行科技股份有限公司 | Data processing method, device, equipment and storage medium |
| CN114253938A (en) * | 2020-09-22 | 2022-03-29 | 中兴通讯股份有限公司 | Data management method, data management device, and storage medium |
| CN112965956A (en) * | 2021-03-18 | 2021-06-15 | 上海东普信息科技有限公司 | Database horizontal capacity expansion method, device, equipment and storage medium |
| CN113111070A (en) * | 2021-05-08 | 2021-07-13 | 福建天晴数码有限公司 | Method and system for performing database and table division based on grouping |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104636450A (en) * | 2015-01-26 | 2015-05-20 | 上海新炬网络信息技术有限公司 | Database table space monitoring method |
| CN106547481A (en) * | 2016-09-29 | 2017-03-29 | 浙江宇视科技有限公司 | A kind of data method for pre-distributing and equipment |
| CN109213761A (en) * | 2018-08-06 | 2019-01-15 | 北京马上慧科技术有限公司 | Table method is divided in point library that a kind of spread exempts from migration |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2019087027A (en) * | 2017-11-07 | 2019-06-06 | 株式会社東芝 | Demand prediction device, demand prediction method and demand prediction program |
| CN110321334B (en) * | 2019-05-24 | 2024-03-19 | 平安银行股份有限公司 | Method and device for automatic partition capacity expansion of database |
-
2019
- 2019-11-20 CN CN201911140830.3A patent/CN110941602B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104636450A (en) * | 2015-01-26 | 2015-05-20 | 上海新炬网络信息技术有限公司 | Database table space monitoring method |
| CN106547481A (en) * | 2016-09-29 | 2017-03-29 | 浙江宇视科技有限公司 | A kind of data method for pre-distributing and equipment |
| CN109213761A (en) * | 2018-08-06 | 2019-01-15 | 北京马上慧科技术有限公司 | Table method is divided in point library that a kind of spread exempts from migration |
Non-Patent Citations (2)
| Title |
|---|
| Forecasting with big data: A review;Hassani H, Silva E S.;Annals of Data Science;全文 * |
| 容量评估管理方式探索――数据分析在信息技术运维领域的应用;万方达;;金融电子化(第11期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110941602A (en) | 2020-03-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110941602B (en) | Database configuration method and device, electronic equipment and storage medium | |
| CN109155763B (en) | Digital signal processing on data stream | |
| CN102831120B (en) | A kind of data processing method and system | |
| CN110399535B (en) | Data query method, device and equipment | |
| CN106407207B (en) | Real-time newly-added data updating method and device | |
| CN107783985B (en) | Distributed database query method, device and management system | |
| JP3510042B2 (en) | Database management method and system | |
| CN105468720A (en) | Method for integrating distributed data processing systems, corresponding systems and data processing method | |
| CN110245134B (en) | Increment synchronization method applied to search service | |
| CN112163048A (en) | Method and device for realizing OLAP analysis based on ClickHouse | |
| WO2014067449A1 (en) | System and method for flexible distributed massively parallel processing (mpp) database | |
| Ho et al. | Optimal algorithms for cross-rack communication optimization in mapreduce framework | |
| CN105786918B (en) | Data query method and device based on data loading storage space | |
| CN106339475A (en) | Distributed storage system for mass data | |
| CN104111936A (en) | Method and system for querying data | |
| CN111722918A (en) | Service identification code generation method and device, storage medium and electronic equipment | |
| CN105183880A (en) | Hash join method and device | |
| CN111488323B (en) | Data processing method and device and electronic equipment | |
| CN115587118A (en) | Task data dimension table association processing method and device and electronic equipment | |
| CN103092886A (en) | Achieving method, device and system for data query operation | |
| US11599540B2 (en) | Query execution apparatus, method, and system for processing data, query containing a composite primitive | |
| JP2009037369A (en) | Resource assignment method to database server | |
| JP7440007B2 (en) | Systems, methods and apparatus for querying databases | |
| CN110837499B (en) | Data access processing method, device, electronic equipment and storage medium | |
| CN107451142B (en) | Method and apparatus for writing and querying data in database, management system and computer-readable storage medium thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20221009 Address after: 12 / F, 15 / F, 99 Yincheng Road, Pudong New Area pilot Free Trade Zone, Shanghai, 200120 Applicant after: Jianxin Financial Science and Technology Co.,Ltd. Address before: 25 Financial Street, Xicheng District, Beijing 100033 Applicant before: CHINA CONSTRUCTION BANK Corp. Applicant before: Jianxin Financial Science and Technology Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |