CN110647577A - Data cube partitioning method and device, computer equipment and storage medium - Google Patents
Data cube partitioning method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN110647577A CN110647577A CN201910711717.XA CN201910711717A CN110647577A CN 110647577 A CN110647577 A CN 110647577A CN 201910711717 A CN201910711717 A CN 201910711717A CN 110647577 A CN110647577 A CN 110647577A
- Authority
- CN
- China
- Prior art keywords
- data
- cube
- channel information
- partitioning
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/278—Data partitioning, e.g. horizontal or vertical partitioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application belongs to the technical field of information security, and relates to a partitioning method of a data cube, which comprises the following steps: receiving a partition request carrying database identification information; acquiring original service data corresponding to the database identification information in a service database, wherein the original service data comprises time information and channel information; preprocessing the time information and the channel information; constructing a data cube corresponding to the original service data; partitioning the data cube according to the channel information to obtain a plurality of subdata cubes; and respectively storing the plurality of sub data cubes into different storage media. The application also provides a partitioning device of the data cube, computer equipment and a storage medium. According to the data cube partitioning storage method and device, the data cube is stored in a partitioning mode, a plurality of partitioned subdata cubes needing to be inquired can work in parallel, and therefore the data inquiring efficiency is improved.
Description
Technical Field
The present application relates to the field of information security technologies, and in particular, to a data cube partitioning method and apparatus, a computer device, and a storage medium.
Background
With the development and application of database technology, the amount of data stored in the database is shifted from megabytes (M) and gigabytes (G) of the 80 th 20 th century to terabytes (T) and gigabytes (P) at present, and at the same time, the query requirements of users are more and more complicated, and the query or manipulation not only refers to querying or manipulating one or more records in a relational table, but also refers to data analysis and information synthesis of tens of millions of records in multiple tables.
In the existing data management method, a fact table and a dimension table are constructed, and the dimension table and the fact table are linked together through a primary key word and a foreign key word to form a relational database in a star mode, so that data management is realized.
However, the conventional data management method is generally not intelligent, and as the amount of data stored in the database is increased, the number of data tables generated by the data management method is increased, so that the storage space occupied by the data management method is increased, and the processing efficiency of the database is greatly reduced.
Disclosure of Invention
An embodiment of the application aims to provide a data cube partitioning method, a data cube partitioning device, computer equipment and a storage medium, and aims to solve the problems that an existing data management method occupies a large amount of storage space and is low in processing efficiency.
In order to solve the above technical problem, an embodiment of the present invention provides a partitioning method for a data cube, which adopts the following technical solutions:
receiving a partition request carrying database identification information;
acquiring original service data corresponding to the database identification information in a service database, wherein the original service data comprises time information and channel information;
preprocessing the time information and the channel information;
constructing a data cube corresponding to the original service data;
partitioning the data cube according to the channel information to obtain a plurality of subdata cubes;
and respectively storing the plurality of sub data cubes into different storage media.
In order to solve the above technical problem, an embodiment of the present invention further provides a partitioning device for a data cube, which adopts the following technical solutions:
the request receiving module is used for receiving a partition request carrying database identification information;
the data acquisition module is used for acquiring original service data corresponding to the database identification information in a service database, wherein the original service data comprises time information and channel information;
the preprocessing module is used for preprocessing the time information and the channel information;
the cube construction module is used for constructing a data cube corresponding to the original service data;
the partition processing module is used for partitioning the data cube according to the channel information to obtain a plurality of sub data cubes; and
and the partition storage module is used for respectively storing the plurality of sub data cubes into different storage media.
In order to solve the above technical problem, an embodiment of the present application further provides a computer device, which adopts the following technical solutions:
comprising a memory and a processor;
the memory has stored therein a computer program which when executed by the processor implements the steps of the method for partitioning a data cube as described above.
In order to solve the above technical problem, an embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solutions:
the computer-readable storage medium has stored thereon a computer program which, when executed by a processor, implements the steps of the partitioning method for a data cube as described above.
Compared with the prior art, the embodiment of the application mainly has the following beneficial effects:
the invention provides a partitioning method of a data cube, which comprises the steps of receiving a partitioning request carrying database identification information; acquiring original service data corresponding to the database identification information in a service database, wherein the original service data comprises time information and channel information; preprocessing the time information and the channel information; constructing a data cube corresponding to the original service data; partitioning the data cube according to the channel information to obtain a plurality of subdata cubes; and respectively storing the plurality of sub data cubes into different storage media. By partitioning and cutting the data in the original database according to the channel information and storing the data in different storage media, a user can enable a plurality of sub data cubes which need to be inquired and are cut to work in parallel when inquiring the data information, and therefore the efficiency of inquiring the data is greatly improved.
Drawings
In order to more clearly illustrate the solution of the present application, the drawings needed for describing the embodiments of the present application will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present application, and that other drawings can be obtained by those skilled in the art without inventive effort.
FIG. 1 is a flowchart illustrating an implementation of a partitioning method for a data cube according to an embodiment of the present invention;
FIG. 2 is a diagram illustrating an example of a data cube provided by an embodiment of the invention;
FIG. 3 is an exemplary diagram of a partitioned data cube according to an embodiment of the present invention;
FIG. 4 is a flowchart of the implementation of step S104 in FIG. 1;
FIG. 5 is a flowchart illustrating an implementation of a partitioning apparatus for a data cube according to a second embodiment of the present invention;
FIG. 6 is a schematic block diagram of the cube construction module 104 of FIG. 5;
FIG. 7 is a schematic block diagram of one embodiment of a computer device according to the present application.
Detailed Description
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "including" and "having," and any variations thereof, in the description and claims of this application and the description of the above figures are intended to cover non-exclusive inclusions. The terms "first," "second," and the like in the description and claims of this application or in the above-described drawings are used for distinguishing between different objects and not for describing a particular order.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.
According to the partitioning method for the data cube provided by the embodiment of the invention, the data in the original database is partitioned and diced according to time and a service body and stored in different storage media, so that a plurality of diced sub data cubes to be queried can work in parallel when a user queries data information, and the efficiency of querying data is greatly improved.
Example one
Fig. 1 is a flowchart illustrating an implementation of a partitioning method for a data cube according to an embodiment of the present invention, and for convenience of description, only the portions related to the present invention are shown.
In step S101, a partition request carrying database identification information is received.
In the embodiment of the present invention, the database identification information is used to uniquely identify a database storing business data, and may be an organization name, such as a database of a commercial bank (gsyhsjk), a database of an agricultural bank (nyyhsjk), and the like; number information is also possible, such as 0001, 0002, etc., as examples; organization name and number information may also be incorporated, such as an industrial and commercial bank database 0001, an agricultural bank database 0002, etc., it being understood that the examples of database identification information herein are for ease of understanding only and are not intended to limit the present invention.
In the embodiment of the present invention, the database for storing the business data may be a relational database, or may be a data warehouse based on a Hadoop spectrum (Hadoop).
In step S102, original service data corresponding to the database identification information is obtained in a service database, where the original service data includes time information and channel information.
In the embodiment of the present invention, the original service data refers to service data of a large data table stored in the database.
In the embodiment of the present invention, the time information is used to record the execution time of the service data.
In the embodiment of the present invention, the channel information is used for recording the main body providing the business service, and the channel information may be organization name information, for example, such as a general issuing bank, a contracting bank, a Fujian agriculture letter, and the like.
In the embodiment of the present invention, the original service data may further include merchant information, region information, industry information, transaction amount information, and the like.
In step S103, the time information and the channel information are preprocessed.
In the embodiment of the present invention, the preprocessing operation is used for performing a preliminary screening operation on the time information and the channel information, and the preprocessing operation includes decryption, decoding and/or data deduplication processing on the time information and the channel information, for example, texts with similar address contents actually point to the same address, and texts with similar business names point to the same business.
In step S104, a data cube corresponding to the original service data is constructed.
In the embodiment of the present invention, the data cube is used for performing dynamic multidimensional analysis on data stored in the Relational database, and the data cube may be a Relational analysis processing method (Relational OLAP) based on the Relational database, as shown in fig. 2, the data cube is constructed by using time information and channel information as main axes.
In step S105, the data cube is partitioned according to the channel information, and a plurality of sub data cubes are obtained.
In the embodiment of the present invention, the partition processing is used to segment the file system file corresponding to the original service data according to the channel information dimension of the data cube, so as to obtain a cube file with a smaller data capacity, as shown in fig. 3, the cube is a sub-data cube obtained by segmenting the original service data according to the channel information.
In step S106, the plurality of child data cubes are stored in different storage media, respectively.
In the embodiment of the invention, a partitioning method of a data cube is provided, which receives a partitioning request carrying database identification information; acquiring original service data corresponding to the database identification information in a service database, wherein the original service data comprises time information and channel information; preprocessing the time information and the channel information; constructing a data cube corresponding to the original service data; partitioning the data cube according to the channel information to obtain a plurality of subdata cubes; and respectively storing the plurality of sub data cubes into different storage media. By partitioning and cutting the data in the original database according to the channel information and storing the data in different storage media, a user can enable a plurality of sub data cubes which need to be inquired and are cut to work in parallel when inquiring the data information, and therefore the efficiency of inquiring the data is greatly improved.
Fig. 4 shows a flowchart for implementing step S104 in fig. 1, and for convenience of explanation, only the part relevant to the present invention is shown.
In some optional implementation manners of the first embodiment of the present invention, the step S104 specifically includes: step S201, step S202, and step S203.
In step S201, the time information and the channel information are respectively used as the main axes of the initial cube.
In step S202, extended data corresponding to the time information and the channel information is acquired in the original service data.
In the embodiment of the present invention, since the original business data is stored in the relational database or a data warehouse based on a Hadoop spectrum (Hadoop), other fact tables corresponding to the time information and the channel information may be obtained according to a dimension table in the relational database or the data warehouse.
In the embodiment of the present invention, the extension data refers to data of other fact tables corresponding to the time information and the channel information in the service database.
In the embodiment of the invention, as the data of the business database is continuously updated, the data cube can also dynamically update the filled data.
In step S203, the extended data is stored to the initial cube based on the correspondence between the time information or the channel information and the extended data, so as to obtain the data cube.
In the embodiment of the invention, the data content of the data cube is perfected by storing the extended data of the time information or the channel information into the initial cube, so that the conversion from the dispersed association of various ER relationships or MAP data objects to the unified multidimensional analysis data cube is completed, and the association corresponding to the time information and the channel information is established.
In some optional implementation manners as the first embodiment of the present invention, the original service data further includes fact data corresponding to the channel information, and the step S105 specifically includes: and segmenting the data cube into a plurality of sub data cubes based on the corresponding relation between the channel information and the fact data.
In the embodiment of the present invention, the fact data refers to fact table data corresponding to the channel information dimension, and the fact data includes digital data (facts), for example, data generated by cash registration transaction, it should be understood that the example of the fact data is only for convenience of understanding and is not used to limit the present invention.
In some optional implementation manners as the first embodiment of the present invention, the step 202 specifically includes: and carrying out data conversion operation on the extension data.
In the embodiment of the present invention, the data conversion operation is used to further specify the format of the data information, and the data conversion operation may be blank type character processing (blank data deletion), encoding processing (scrambling code deletion), character processing (blank type character processing, special character processing, and character string representation mode conversion), representation conversion, and the like.
In the embodiment of the invention, the original service data is processed in the data format in advance, so that the problem that a subsequently generated data cube contains wrong data or indiscernible code data is avoided.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and can include the processes of the embodiments of the methods described above when the computer program is executed. The storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disk, a Read-Only Memory (ROM), or a Random Access Memory (RAM).
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless explicitly stated herein. Moreover, at least a portion of the steps in the flow chart of the figure may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
Example two
With further reference to fig. 5, as an implementation of the method shown in fig. 1, the present application provides a partitioning apparatus for a data cube, where an embodiment of the apparatus corresponds to the embodiment of the method shown in fig. 1, and the apparatus may be applied to various electronic devices.
As shown in fig. 5, a partitioning apparatus 100 for a data cube according to a second embodiment of the present invention includes: a request accepting module 101, a data acquiring module 102, a preprocessing module 103, a cube constructing module 104, a partition processing module 105, and a partition storing module 106. Wherein:
a request receiving module 101, configured to receive a partition request carrying database identification information;
a data obtaining module 102, configured to obtain, in a service database, original service data corresponding to the database identification information, where the original service data includes time information and channel information;
the preprocessing module 103 is used for preprocessing the time information and the channel information;
a cube construction module 104, configured to construct a data cube corresponding to the original business data;
the partition processing module 105 is configured to perform partition processing on the data cube according to the channel information to obtain a plurality of sub data cubes; and
and the partition storage module 106 is configured to store the plurality of child data cubes into different storage media respectively.
In the embodiment of the present invention, the database identification information is used to uniquely identify a database storing business data, and may be an organization name, such as a database of a commercial bank (gsyhsjk), a database of an agricultural bank (nyyhsjk), and the like; number information is also possible, such as 0001, 0002, etc., as examples; organization name and number information may also be incorporated, such as an industrial and commercial bank database 0001, an agricultural bank database 0002, etc., it being understood that the examples of database identification information herein are for ease of understanding only and are not intended to limit the present invention.
In the embodiment of the present invention, the database for storing the business data may be a relational database, or may be a data warehouse based on a Hadoop spectrum (Hadoop).
In the embodiment of the present invention, the original service data refers to service data of a large data table stored in the database.
In the embodiment of the present invention, the time information is used to record the execution time of the service data.
In the embodiment of the present invention, the channel information is used for recording the main body providing the business service, and the channel information may be organization name information, for example, such as a general issuing bank, a contracting bank, a Fujian agriculture letter, and the like.
In the embodiment of the present invention, the original service data may further include merchant information, region information, industry information, transaction amount information, and the like.
In the embodiment of the present invention, the preprocessing operation is used for performing a preliminary screening operation on the time information and the channel information, and the preprocessing operation includes decryption, decoding and/or data deduplication processing on the time information and the channel information, for example, texts with similar address contents actually point to the same address, and texts with similar business names point to the same business.
In the embodiment of the present invention, the data cube is used for performing dynamic multidimensional analysis on data stored in the Relational database, and the data cube may be a Relational analysis processing method (Relational OLAP) based on the Relational database, as shown in fig. 2, the data cube is constructed by using time information and channel information as main axes.
In the embodiment of the present invention, the partition processing is used to segment the file system file corresponding to the original service data according to the channel information dimension of the data cube, so as to obtain a cube file with a smaller data capacity, as shown in fig. 3, the cube is a sub-data cube obtained by segmenting the original service data according to the channel information.
In the embodiment of the invention, a partitioning device of a data cube is provided, wherein a request receiving module is used for receiving a partitioning request carrying database identification information; the data acquisition module is used for acquiring original service data corresponding to the database identification information in a service database, wherein the original service data comprises time information and channel information; the preprocessing module is used for preprocessing the time information and the channel information; the cube construction module is used for constructing a data cube corresponding to the original service data; the partition processing module is used for partitioning the data cube according to the channel information to obtain a plurality of sub data cubes; and the partition storage module is used for respectively storing the plurality of sub data cubes into different storage media. By partitioning and cutting the data in the original database according to the channel information and storing the data in different storage media, a user can enable a plurality of sub data cubes which need to be inquired and are cut to work in parallel when inquiring the data information, and therefore the efficiency of inquiring the data is greatly improved.
With continued reference to fig. 6, a schematic structural diagram of a cube construction module provided in the second embodiment of the present invention is shown, and for convenience of explanation, only the parts related to the present invention are shown.
In some optional implementations of the second embodiment of the present invention, the cube construction module 104 includes: a main axis determination sub-module 1041, an extended data acquisition sub-module 1042, and a cube perfection sub-module 1043. Wherein:
the main shaft determining submodule is used for respectively taking the time information and the channel information as main shafts of an initial cube;
an extended data obtaining sub-module, configured to obtain, in the original service data, extended data corresponding to the time information and the channel information;
and the cube perfecting submodule is used for storing the extended data into the initial cube based on the corresponding relation between the time information or the channel information and the extended data to obtain the data cube.
In the embodiment of the present invention, since the original business data is stored in the relational database or a data warehouse based on a Hadoop spectrum (Hadoop), other fact tables corresponding to the time information and the channel information may be obtained according to a dimension table in the relational database or the data warehouse.
In the embodiment of the present invention, the extension data refers to data of other fact tables corresponding to the time information and the channel information in the service database.
In the embodiment of the invention, as the data of the business database is continuously updated, the data cube can also dynamically update the filled data.
In the embodiment of the invention, the data content of the data cube is perfected by storing the extended data of the time information or the channel information into the initial cube, so that the conversion from the dispersed association of various ER relationships or MAP data objects to the unified multidimensional analysis data cube is completed, and the association corresponding to the time information and the channel information is established.
In some optional implementations as the first embodiment of the present invention, the partition processing module includes: and a partition processing submodule. Wherein:
and the partition processing submodule is used for segmenting the data cube into a plurality of sub data cubes based on the corresponding relation between the channel information and the fact data.
In the embodiment of the present invention, the fact data refers to fact table data corresponding to the channel information dimension, and the fact data includes digital data (facts), for example, data generated by cash registration transaction, it should be understood that the example of the fact data is only for convenience of understanding and is not used to limit the present invention.
In some optional implementations as the first embodiment of the present invention, the extended data obtaining sub-module 1042 specifically includes: and a data conversion submodule. Wherein:
and the data conversion submodule is used for carrying out data conversion operation on the extended data.
In the embodiment of the present invention, the data conversion operation is used to further specify the format of the data information, and the data conversion operation may be blank type character processing (blank data deletion), encoding processing (scrambling code deletion), character processing (blank type character processing, special character processing, and character string representation mode conversion), representation conversion, and the like.
In the embodiment of the invention, the original service data is processed in the data format in advance, so that the problem that a subsequently generated data cube contains wrong data or indiscernible code data is avoided.
In summary, the embodiment of the present invention provides a partitioning method for a data cube, which receives a partitioning request sent by a user, where the partitioning request at least carries database identification information; responding to the partition request, and acquiring original service data in a service database corresponding to the database identification information, wherein the original service data at least carries time information and channel information; constructing a data cube corresponding to the original service data by taking the time information and the channel information as main axes; performing partition storage operation on the data cube based on a preset partition rule; outputting a partition complete signal to the user. By partitioning and blocking the data in the original database according to time and a service body and storing the data in different storage media, a user can enable a plurality of blocked subdata cubes to be queried to work in parallel when querying data information, and the efficiency of querying data is greatly improved. Meanwhile, the extended data of the time information or the channel information is stored in the initial cube, so that the data content of the data cube is perfected, the conversion from the dispersed association of various ER relationships or MAP data objects to the unified multidimensional analysis data cube is further completed, and the association corresponding to the time information and the channel information is established; by processing the original service data in a data format in advance, the problem that a subsequently generated data cube contains wrong data or indiscriminate code data is avoided.
In order to solve the technical problem, an embodiment of the present application further provides a computer device. Referring to fig. 7, fig. 7 is a block diagram of a basic structure of a computer device according to the present embodiment.
The computer device 7 comprises a memory 71, a processor 72, a network interface 73, which are communicatively connected to each other via a system bus. It is noted that only a computer device 7 having components 71-73 is shown, but it is to be understood that not all of the shown components are required to be implemented, and that more or fewer components may be implemented instead. As will be understood by those skilled in the art, the computer device is a device capable of automatically performing numerical calculation and/or information processing according to a preset or stored instruction, and the hardware includes, but is not limited to, a microprocessor, an Application Specific Integrated Circuit (ASIC), a Programmable gate array (FPGA), a Digital Signal Processor (DSP), an embedded device, and the like.
The computer device can be a desktop computer, a notebook, a palm computer, a cloud server and other computing devices. The computer equipment can carry out man-machine interaction with a user through a keyboard, a mouse, a remote controller, a touch panel or voice control equipment and the like.
The memory 71 includes at least one type of readable storage medium including a flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the storage 71 may be an internal storage unit of the computer device 7, such as a hard disk or a memory of the computer device 7. In other embodiments, the memory 71 may also be an external storage device of the computer device 7, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a flash Card (FlashCard), and the like, which are provided on the computer device 7. Of course, the memory 71 may also comprise both an internal storage unit of the computer device 7 and an external storage device thereof. In this embodiment, the memory 71 is generally used for storing an operating system installed in the computer device 7 and various types of application software, such as program codes of a partition method of a data cube. Further, the memory 71 may also be used to temporarily store various types of data that have been output or are to be output.
The processor 72 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 72 is typically used to control the overall operation of the computer device 7. In this embodiment, the processor 72 is configured to execute the program code stored in the memory 71 or process data, for example, execute the program code of the partition method of the data cube.
The network interface 73 may comprise a wireless network interface or a wired network interface, and the network interface 73 is generally used for establishing a communication connection between the computer device 7 and other electronic devices.
The present application further provides another embodiment, which provides a computer-readable storage medium storing a partitioning program for a data cube, the partitioning program being executable by at least one processor to cause the at least one processor to perform the steps of the partitioning method for a data cube as described above.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present application.
It is to be understood that the above-described embodiments are merely illustrative of some, but not restrictive, of the broad invention, and that the appended drawings illustrate preferred embodiments of the invention and do not limit the scope of the invention. This application is capable of embodiments in many different forms and is provided for the purpose of enabling a thorough understanding of the disclosure of the application. Although the present application has been described in detail with reference to the foregoing embodiments, it will be apparent to one skilled in the art that the present application may be practiced without modification or with equivalents of some of the features described in the foregoing embodiments. All equivalent structures made by using the contents of the specification and the drawings of the present application are directly or indirectly applied to other related technical fields and are within the protection scope of the present application.
Claims (10)
1. A method of partitioning a data cube, comprising:
receiving a partition request carrying database identification information;
acquiring original service data corresponding to the database identification information in a service database, wherein the original service data comprises time information and channel information;
preprocessing the time information and the channel information;
constructing a data cube corresponding to the original service data;
partitioning the data cube according to the channel information to obtain a plurality of subdata cubes;
and respectively storing the plurality of sub data cubes into different storage media.
2. The method for partitioning a data cube according to claim 1, wherein said step of constructing a data cube corresponding to said original business data comprises:
respectively taking the time information and the channel information as main shafts of an initial cube;
acquiring extended data corresponding to the time information and the channel information from the original service data;
and storing the extended data to the initial cube based on the corresponding relation between the time information or the channel information and the extended data to obtain the data cube.
3. The method for partitioning a data cube according to claim 1, wherein the raw service data further includes fact data corresponding to the channel information, and the step of partitioning the data cube according to the channel information includes:
and segmenting the data cube into a plurality of sub data cubes based on the corresponding relation between the channel information and the fact data.
4. The method of partitioning a data cube of claim 1, wherein pre-processing the time information and the channel information comprises:
and carrying out decryption, decoding and data deduplication processing on the time information and the channel information.
5. An apparatus for partitioning a data cube, the apparatus comprising:
the request receiving module is used for receiving a partition request carrying database identification information;
the data acquisition module is used for acquiring original service data corresponding to the database identification information in a service database, wherein the original service data comprises time information and channel information;
the preprocessing module is used for preprocessing the time information and the channel information;
the cube construction module is used for constructing a data cube corresponding to the original service data;
the partition processing module is used for partitioning the data cube according to the channel information to obtain a plurality of sub data cubes; and
and the partition storage module is used for respectively storing the plurality of sub data cubes into different storage media.
6. The apparatus for partitioning a data cube of claim 5, wherein said cube construction module comprises:
the main shaft determining submodule is used for respectively taking the time information and the channel information as main shafts of an initial cube;
an extended data obtaining sub-module, configured to obtain, in the original service data, extended data corresponding to the time information and the channel information;
and the cube perfecting submodule is used for storing the extended data into the initial cube based on the corresponding relation between the time information or the channel information and the extended data to obtain the data cube.
7. The apparatus for partitioning a data cube according to claim 5, wherein said raw business data further contains fact data corresponding to said channel information, said partitioning processing module comprising:
and the partition processing submodule is used for segmenting the data cube into a plurality of sub data cubes based on the corresponding relation between the channel information and the fact data.
8. The apparatus for partitioning a data cube of claim 5, wherein said preprocessing module comprises:
and the preprocessing submodule is used for carrying out decryption, decoding and data deduplication processing on the time information and the channel information.
9. A computer device comprising a memory having stored therein a computer program and a processor implementing the steps of the partitioning method of the data cube according to any one of claims 1 to 4 when executing the computer program.
10. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, carries out the steps of the partitioning method of the data cube according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910711717.XA CN110647577A (en) | 2019-08-02 | 2019-08-02 | Data cube partitioning method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910711717.XA CN110647577A (en) | 2019-08-02 | 2019-08-02 | Data cube partitioning method and device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110647577A true CN110647577A (en) | 2020-01-03 |
Family
ID=68989979
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910711717.XA Pending CN110647577A (en) | 2019-08-02 | 2019-08-02 | Data cube partitioning method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110647577A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112650819A (en) * | 2020-12-29 | 2021-04-13 | 清华大学 | Method, device and equipment for constructing metadata cube and storage medium |
| CN113297333A (en) * | 2021-03-17 | 2021-08-24 | 无锡极数宝大数据科技有限公司 | Data processing method, device, server and storage medium |
| CN117931811A (en) * | 2024-03-22 | 2024-04-26 | 腾讯科技（深圳）有限公司 | Database processing method, device, computer equipment and storage medium |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1347529A (en) * | 1999-01-15 | 2002-05-01 | 米泰吉公司 | Method for visualizing information in data warehousing environment |
| CN101452485A (en) * | 2008-12-31 | 2009-06-10 | 中国建设银行股份有限公司 | Method and device for generating multidimensional cubic based on relational database |
| CN102663616A (en) * | 2012-03-19 | 2012-09-12 | 北京国双科技有限公司 | Method and system for measuring web advertising effectiveness based on multiple-contact attribution model |
| CN103679309A (en) * | 2012-09-13 | 2014-03-26 | 余明阳 | Intelligent statistical analysis method for power data warehouse services |
| US20150026116A1 (en) * | 2013-07-22 | 2015-01-22 | International Business Machines Corporation | Managing sparsity in an multidimensional data structure |
| CN105894217A (en) * | 2016-05-25 | 2016-08-24 | 国网浙江省电力公司 | Power marketing multi-business system data center construction method |
| US9495437B1 (en) * | 2013-11-06 | 2016-11-15 | Dell Software Inc. | System and method for accessing dimensional databases |
| CN109299197A (en) * | 2018-08-07 | 2019-02-01 | 威富通科技有限公司 | A kind of partition method of data cube, system and terminal device |
| CN109472034A (en) * | 2018-11-02 | 2019-03-15 | 深圳壹账通智能科技有限公司 | Languages conversion method, device, equipment and storage medium based on data subregion |
| CN109669934A (en) * | 2018-12-11 | 2019-04-23 | 江苏瑞中数据股份有限公司 | A kind of data warehouse and its construction method suiting electric power customer service |
| CN109815231A (en) * | 2018-12-26 | 2019-05-28 | 亚信科技（中国）有限公司 | Data fragmentation method, device, computer equipment and storage medium |
-
2019
- 2019-08-02 CN CN201910711717.XA patent/CN110647577A/en active Pending
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1347529A (en) * | 1999-01-15 | 2002-05-01 | 米泰吉公司 | Method for visualizing information in data warehousing environment |
| CN101452485A (en) * | 2008-12-31 | 2009-06-10 | 中国建设银行股份有限公司 | Method and device for generating multidimensional cubic based on relational database |
| CN102663616A (en) * | 2012-03-19 | 2012-09-12 | 北京国双科技有限公司 | Method and system for measuring web advertising effectiveness based on multiple-contact attribution model |
| CN103679309A (en) * | 2012-09-13 | 2014-03-26 | 余明阳 | Intelligent statistical analysis method for power data warehouse services |
| US20150026116A1 (en) * | 2013-07-22 | 2015-01-22 | International Business Machines Corporation | Managing sparsity in an multidimensional data structure |
| US9495437B1 (en) * | 2013-11-06 | 2016-11-15 | Dell Software Inc. | System and method for accessing dimensional databases |
| CN105894217A (en) * | 2016-05-25 | 2016-08-24 | 国网浙江省电力公司 | Power marketing multi-business system data center construction method |
| CN109299197A (en) * | 2018-08-07 | 2019-02-01 | 威富通科技有限公司 | A kind of partition method of data cube, system and terminal device |
| CN109472034A (en) * | 2018-11-02 | 2019-03-15 | 深圳壹账通智能科技有限公司 | Languages conversion method, device, equipment and storage medium based on data subregion |
| CN109669934A (en) * | 2018-12-11 | 2019-04-23 | 江苏瑞中数据股份有限公司 | A kind of data warehouse and its construction method suiting electric power customer service |
| CN109815231A (en) * | 2018-12-26 | 2019-05-28 | 亚信科技（中国）有限公司 | Data fragmentation method, device, computer equipment and storage medium |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112650819A (en) * | 2020-12-29 | 2021-04-13 | 清华大学 | Method, device and equipment for constructing metadata cube and storage medium |
| CN113297333A (en) * | 2021-03-17 | 2021-08-24 | 无锡极数宝大数据科技有限公司 | Data processing method, device, server and storage medium |
| CN117931811A (en) * | 2024-03-22 | 2024-04-26 | 腾讯科技（深圳）有限公司 | Database processing method, device, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109977110B (en) | Data cleaning method, device and equipment | |
| WO2019140828A1 (en) | Electronic apparatus, method for querying logs in distributed system, and storage medium | |
| CN112765271A (en) | Block chain transaction index storage method and device, computer equipment and medium | |
| CN110647577A (en) | Data cube partitioning method and device, computer equipment and storage medium | |
| CN109241159B (en) | Partition query method and system for data cube and terminal equipment | |
| CN110659282B (en) | Data route construction method, device, computer equipment and storage medium | |
| CN108875077B (en) | Column storage method and device of database, server and storage medium | |
| CN111782763A (en) | Information retrieval method based on voice semantics and related equipment thereof | |
| CN113407785B (en) | Data processing method and system based on distributed storage system | |
| CN113129150A (en) | Transaction data processing method and device, terminal device and readable storage medium | |
| CN111553556A (en) | Business data analysis method and device, computer equipment and storage medium | |
| CN113010542B (en) | Service data processing method, device, computer equipment and storage medium | |
| CN108108392B (en) | Commodity data management method, commodity data management device, computer equipment and storage medium | |
| WO2019071907A1 (en) | Method for identifying help information based on operation page, and application server | |
| CN109542912B (en) | Interval data storage method, device, server and storage medium | |
| US8005844B2 (en) | On-line organization of data sets | |
| CN113254457B (en) | Account checking method, account checking system and computer readable storage medium | |
| CN114912003A (en) | Document searching method and device, computer equipment and storage medium | |
| CN110674224B (en) | Entity data processing method, device and equipment and computer readable storage medium | |
| CN114625372A (en) | Automatic component compiling method and device, computer equipment and storage medium | |
| CN114372183A (en) | User portrait analysis method based on high-dimensional data projection | |
| CN114116699A (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| US20240054586A1 (en) | Systems and methods for automated real estate property matching across disparate data sources | |
| CN108984719B (en) | Data deleting method and device based on column storage, server and storage medium | |
| CN103019665A (en) | Receipt opening method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200103 |