WO2018119976A1 - Procédé d'optimisation d'implantation de données efficace destiné à un système d'entrepôt de données - Google Patents
Procédé d'optimisation d'implantation de données efficace destiné à un système d'entrepôt de données Download PDFInfo
- Publication number
- WO2018119976A1 WO2018119976A1 PCT/CN2016/113364 CN2016113364W WO2018119976A1 WO 2018119976 A1 WO2018119976 A1 WO 2018119976A1 CN 2016113364 W CN2016113364 W CN 2016113364W WO 2018119976 A1 WO2018119976 A1 WO 2018119976A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- data
- column
- file
- columns
- block
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Definitions
- the present invention relates to the field of data processing, and in particular, to an efficient and optimized data layout method applied to a data warehouse system.
- Structured data is the most common type of data storage in database management systems.
- partitioning method of table structure in structured data has a great impact on query and space efficiency. This is done by a single node. Data processing efficiency and network data transmission differences between different nodes.
- Row storage is a commonly used data layout structure that divides the data in a table into rows, and then stores the partitioned data blocks on different data nodes, where each node stores the rows in turn on disk. Its shortcoming is that in the query process, even if the column is not used, the entire row of data needs to be loaded into the memory and perform unnecessary query operations, thus extending the query time.
- Another common data layout structure is column storage, which divides the data in the table into columns, and then stores the different columns on different data nodes, each of which stores the columns in turn on disk. Its disadvantage is that the results obtained by different columns in the query process need to be transmitted between nodes to produce the final result. This way increases the data transmission loss and reduces the query efficiency.
- RCFile A Fast and Space-efficient Data Placement Structure in MapReduce-based Warehouse Systems
- the proposed RCFile is a common data layout scheme applied to distributed storage systems. It mainly combines the storage methods of row storage and column storage to build files. The data within the block. When the table data needs to be stored, it first divides the table file according to the row format, wherein each divided row group has the same size, and then stores the row group in different areas of the file block, and simultaneously in each row group. Column order is stored contiguously, and this way of storage avoids the drawbacks of row storage and column storage modes.
- RCFile's data compression method is relatively simple, and each column data in each row group is separately compressed and stored. This way of compressing all the data is not suitable for the reading and use of common data.
- the primary key in a table is used in almost every query. In each query for it, it is necessary to perform column data once. Decompression, this way results in higher time and computational overhead.
- its way of fault tolerance for data is the multi-copy mode of the underlying storage system, which takes up more storage space than the error correction code.
- Zebra is a column-oriented data layout structure. In order to avoid the defect of the multi-node recombination query result inherent in the column layout, it divides the columns of the data table into multiple column groups, and stores each column group separately, in the storage. In each column group, the data is stored in a row-stored format. Each of the column groups consists of multiple columns, and one column can belong to different column groups. This storage method largely avoids the storage of query results on multiple nodes.
- the Zebra storage layout needs to group the columns in the table before storing the data table. For a query, there is no guarantee that all the columns to be used are in the same column group. In this case, The result of the query still requires reorganization of data rows between multiple nodes. Because the column group is on the same node, one column can be in multiple column groups at the same time, which actually adds duplicate data to the original data, which increases storage overhead.
- the present invention provides an efficient and optimized data layout method applied to a data warehouse system, which solves the problem of increasing query overhead and occupying more storage space in the prior art.
- the invention is realized by the following technical solutions: designing and manufacturing an efficient and optimized data layout method applied to a data warehouse system, comprising the following steps: (A) performing block file basic data layout; (B) performing column classification processing; (C) Perform table file storage.
- the table file is horizontally divided into equal-sized row groups, and then the row groups are sequentially stored in the block file by column storage; each row group is composed of three
- the partial components are respectively a synchronization part, a metadata part and an actual data part, and the synchronization part is used for the system to distinguish two adjacent row groups when reading data, and the metadata part includes the system can be in the row group Differentiating the size information of different columns and different fields in each column and column classification information for systematically distinguishing different kinds of columns, the actual data portion is used for storing actual data.
- a column classification strategy based on the frequency of use is used to reduce the decompression cost of the common column, and the column is divided into a query column and a code column.
- step (C) data is stored in a manner of using both a copy and an RDP code check block.
- the matrix generated by the RDP code is a file group, and the data block of each file group stores two copies and two RDP code check blocks generated by the storage blocks when stored; two copy storage On different nodes, the other two check blocks are stored on nodes that do not contain any data blocks of the file group.
- an RDP code generation group is a matrix of (p-1) ⁇ (p + 1), wherein the parameter p is an arbitrary prime number greater than 2, and the last two columns of each matrix are generated. Data is verified, and other columns store information data; the RDP code is divided into a row check block and a diagonal check block, and the row check block is obtained by laterally adding information data, and the diagonal check block is composed of information data. The diagonal addition is obtained; the RDP code organization information block file generates a verification file.
- the code column is used to divide the data column, the column whose frequency is greater than or equal to the coded threshold is divided into the query column, and the frequency of use is less than the coded threshold is the code column.
- the invention has the beneficial effects that the upper layer data warehouse system can obtain faster query rate and occupy less storage space than the conventional solution when processing large-scale structured data; in terms of query rate, by setting different codes
- the threshold is to meet the needs of data management. Generally speaking, the smaller the encoding threshold, the higher the query rate will be, and the larger the storage space occupied by the data. Data warehouse managers can set a reasonable coding threshold through actual business requirements, which can make the data warehouse system get a good compromise between query rate and space occupation. In terms of storage space, use the construction parameters to determine the row group and The size of the file group, by constructing the row group, makes the data avoid additional data reading and result reorganization during the query.
- the data table takes less than three copies of the fault-tolerant method in the fault-tolerant space, and is fault-tolerant. With no less than three copies, this storage method allows the system to take up less storage space in terms of data fault tolerance, saving physical storage costs.
- FIG. 1 is a schematic view of space occupation of the present invention
- FIG. 2 is a schematic diagram of a query rate according to the present invention.
- An efficient and optimized data layout method applied to a data warehouse system includes the following steps: (A) performing block file basic data layout; (B) performing column classification processing; and (C) performing table file storage.
- the table file is horizontally divided into equal-sized row groups, and then the row groups are sequentially stored in the block file by column storage; each row group is composed of three parts, which are respectively a synchronization part.
- a metadata portion for distinguishing between two adjacent row groups when the data is read, and an actual data portion the metadata portion including the system distinguishing different columns and each column in the row group
- the size information of the different domains and the column classification information for the system to distinguish the different kinds of columns, the actual data portion is used to store the actual data.
- a column classification strategy based on the frequency of use is used to reduce the decompression cost of the common column, and the column is divided into a query column and a code column.
- the data is stored in a manner of using both the replica and the RDP code check block.
- the matrix generated by the RDP code is a file group, and the data block of each file group stores two copies and two RDP code check blocks generated by the storage blocks when stored; two copies are stored on different nodes, and The other two check blocks are stored on nodes that do not contain any data blocks of the file set.
- An RDP code generation group is a (p-1) ⁇ (p+1) matrix, where the parameter p is an arbitrary prime number greater than 2, the last two columns of each matrix are generated check data, and the other columns store information.
- Data; the RDP code is divided into a row check block and a diagonal check block, and the row check block is obtained by laterally adding information data, and the diagonal check block is obtained by adding diagonal lines of information data;
- the RDP code organization information block file generates a verification file.
- the code column is used to divide the data column, the column whose frequency is greater than or equal to the coded threshold is divided into the query column, and the frequency of use is less than the coded threshold is the code column.
- the EStore of the present invention is used for the underlying storage file block data layout of the distributed storage system, and the optimized layout of the data enables the system query execution rate to be improved while reducing the storage space occupied by the data for error correction.
- the EStore first divides the table file into equal-sized row groups in the block file data layout, and then stores the row groups in the block file in a column storage manner.
- the size of the row group is determined by the system's build parameters, and the system's build parameters also affect the size of the filegroup, as described in the next section.
- the EStore system stores the row groups on such blocks.
- the first part is the synchronization part, which is used by the system to distinguish two adjacent line groups when reading data.
- the second part is the metadata section, which contains the size information that the system can distinguish between different columns and different fields in each column in the row group. In addition to this, column classification information is included for the system to distinguish between different types of columns.
- the third part is the actual data part, which is used to store the actual data, which are organized in column groups in column storage.
- a column classification strategy based on frequency of use is used to reduce the decompression cost of common columns.
- the system divides the columns in a data table into two types, one for the query column and the other for the code column. Each column is divided into one of these types.
- the system uses the code threshold value to divide the data column.
- Some queries are usually executed periodically for information decision making or data mining. In such a case, it is considered that such a query has a frequency of use, and each query will often use the same column for a data table.
- the data table is preprocessed, and the query used by each column in the table is counted, and the value obtained by adding the frequency of the queries is the frequency of use of the column. In this way, the frequency of use of each column can be obtained.
- the column whose frequency is greater than the coding threshold is divided into the query column. If the frequency is less than the coding threshold, the code threshold is the code column.
- each column in a row group is actually stored, the columns are stored in different forms depending on the kind of these columns.
- the query column requires that the data is read fast, then the data is stored according to the original format of the data, and for the code column, based on the storage space requirement, the column is compressed and stored using a common data compression algorithm.
- the classification information for these columns is saved in the second part of the row group. This way reduces the possibility of decompressing data when querying, and improves the query efficiency of the system.
- the data system manager can set a different coding threshold, so that the system can get a good balance between query rate and storage space. .
- RDP codes to fault-tolerant data.
- An RDP code generation group is a (p-1) ⁇ (p+1) matrix, where the parameter p is an arbitrary prime number greater than 2, the last two columns of each matrix are generated check data, and the other columns store information. data.
- the first column is obtained by lateral addition of information data, called a row check block.
- the second column is obtained by adding the diagonals of the information data, called the diagonal check block.
- the main problem with applying RDP codes to block verification of distributed storage systems is how to organize the information block files to generate checksum files.
- EStore uses the build parameters to determine the size of this check matrix.
- the EStore defines the construction parameter as an arbitrary prime number greater than 2. If the size of the constructed prime number is k, then its RDP generation matrix is a matrix of (k ⁇ 1) ⁇ (k+1) size, that is, a total of k+1 Files, each file is internally divided into k-1 blocks. It is known from the previous section that the file blocks are composed of the same size group, so the row group is regarded as the basic symbol in the RDP generation matrix, and each file block It will contain k ⁇ 1 row groups, so the size of the row group in the file block is determined by the size of the block and the build parameters.
- the matrix generated by each RDP code in the EStore is referred to as a file group.
- a modular table file which often contains a large number of storage blocks.
- the EStore divides the storage blocks according to the size of the construction parameters, and divides the storage blocks into different file groups, where each file group contains k-1 such storage. Block, then use these blocks to regenerate 2 check blocks in each file group, so that each file group will eventually contain k+1 file blocks.
- each row group in block 4 contains all the row checkers.
- r 0,4 is the exclusive OR of the row groups ⁇ r 0,0 , r 0,1 , r 0,2 ,r 0.3 ⁇ .
- Block 5 contains all of the diagonal check symbols, for example, r 0,5 is the exclusive OR of the row set ⁇ r 0,0 , r 3,2 , r 2,3 , r 1,4 ⁇ .
- the data is stored in the EStore using both the copy and the check block.
- the data block of each of its file sets contains two copies in the storage system, and also stores two RDP code check blocks generated by these storage blocks.
- the reason that the system still uses the copy mode is that the RDP code requires a large transmission bandwidth when the file is restored. Therefore, for each data block, the system will still store one more copy on other nodes, so that when a single node fails, the data block can still be obtained by copy transmission. Only when two nodes storing the same data block fail at the same time, it is necessary to restore the data block by means of RDP code data recovery. Since such a situation does not occur frequently in a distributed storage system, it is still acceptable for the transmission bandwidth of the RDP code repair in the case where the construction parameters are not very large.
- EStore's file group storage two copies of each data block are stored on different nodes, and the other two check blocks are stored on nodes that do not contain any data blocks of the file group, so that When two copies of an arbitrary data block are corrupted at the same time, the original data block is restored using the RDP repair method.
- the present invention uses three optimization strategies to improve the data processing performance of the data warehouse system, which is manifested in the improvement of the query rate and the reduction of the fault-tolerant space occupation.
- the data is reasonably organized in the file block of hdfs.
- the table file in the relational data is horizontally divided into equal-sized row groups, each hdfs file block stores one or more row groups, and data is stored in a column storage manner within each row group.
- the data is pre-processed before the data table is stored, the frequency of use of different columns is counted, the code threshold is set to classify the column, and the code column below the coding threshold is stored in the data compression mode. Above the encoding threshold is the query column, which holds the data in its native format.
- the RDP code is used to construct the error correction strategy, and the data table file is divided into different file groups.
- the size of the file group is determined by the system construction parameters.
- the system generates two additional check file blocks for data repair within each file group.
- the system uses double copy plus check block to perform data fault tolerance. Each data block in the file group is saved on two different nodes, and the check block is saved on other nodes.
- the system uses the construction parameters to encode two important parameters of the threshold to store the table file. By setting different parameters, the system can meet the specific data management requirements in terms of query efficiency and space occupation.
- the EStore system is built in the actual distributed system, and the performance comparison between the EStore and the existing common data layout structure RCFile is made in terms of storage space occupation and query rate.
- the parameter t is used to indicate the encoding threshold.
- Figure 2 shows the difference in query rates for different data layouts. It can be seen that the EStore query rate is higher than RCFile under all coding thresholds, because EStore reduces the time consumption of data decompression during the query process.
- Performance comparisons with RCFiles with EStores with different encoding thresholds include space occupancy and query execution time.
- the above experimental results reflect the performance advantages of the EStore column classification and fault tolerance strategy.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
L'invention concerne un procédé d'optimisation d'implantation de données efficace destiné à un système d'entrepôt de données, le procédé comprenant les étapes qui consistent : (A) à agencer une implantation de données sur la base de fichiers de blocs; (B) à effectuer un tri de colonnes; et (C) à mémoriser des fichiers de table. Lors du traitement de données structurées à grande échelle dans le système d'entrepôt de données à un niveau supérieur, l'invention présente une très grande efficacité d'interrogation et occupe moins d'espace de mémoire que les solutions classiques.
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201680090379.7A CN110268397B (zh) | 2016-12-30 | 2016-12-30 | 应用于数据仓库***的高效优化数据布局方法 |
PCT/CN2016/113364 WO2018119976A1 (fr) | 2016-12-30 | 2016-12-30 | Procédé d'optimisation d'implantation de données efficace destiné à un système d'entrepôt de données |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/CN2016/113364 WO2018119976A1 (fr) | 2016-12-30 | 2016-12-30 | Procédé d'optimisation d'implantation de données efficace destiné à un système d'entrepôt de données |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2018119976A1 true WO2018119976A1 (fr) | 2018-07-05 |
Family
ID=62706678
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2016/113364 WO2018119976A1 (fr) | 2016-12-30 | 2016-12-30 | Procédé d'optimisation d'implantation de données efficace destiné à un système d'entrepôt de données |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN110268397B (fr) |
WO (1) | WO2018119976A1 (fr) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112579597A (zh) * | 2020-12-15 | 2021-03-30 | 西安邮电大学 | 一种压缩敏感的数据库文件存储方法及*** |
CN116931845A (zh) * | 2023-09-18 | 2023-10-24 | 新华三信息技术有限公司 | 一种数据布局方法、装置及电子设备 |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN116719822B (zh) * | 2023-08-10 | 2023-12-22 | 深圳市连用科技有限公司 | 一种海量结构化数据的存储方法及*** |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101339524A (zh) * | 2008-05-22 | 2009-01-07 | 清华大学 | 大规模磁盘阵列存储***的磁盘容错方法 |
US20100235677A1 (en) * | 2007-09-21 | 2010-09-16 | Wylie Jay J | Generating A Parallel Recovery Plan For A Data Storage System |
CN103186566A (zh) * | 2011-12-28 | 2013-07-03 | ***通信集团河北有限公司 | 一种数据分级存储方法、装置及*** |
CN103699676A (zh) * | 2013-12-30 | 2014-04-02 | 厦门市美亚柏科信息股份有限公司 | 基于mssql server表分区及自动维护方法及*** |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US3694813A (en) * | 1970-10-30 | 1972-09-26 | Ibm | Method of achieving data compaction utilizing variable-length dependent coding techniques |
CN102521363A (zh) * | 2011-12-15 | 2012-06-27 | 武汉达梦数据库有限公司 | 基于列分解的列存储数据库数值数据压缩方法 |
CN102737132A (zh) * | 2012-06-25 | 2012-10-17 | 天津神舟通用数据技术有限公司 | 基于数据库行列混合存储的多规则复合压缩方法 |
CN103118133B (zh) * | 2013-02-28 | 2015-09-02 | 浙江大学 | 基于文件访问频次的混合云存储方法 |
US9722637B2 (en) * | 2013-03-26 | 2017-08-01 | Peking University Shenzhen Graduate School | Construction of MBR (minimum bandwidth regenerating) codes and a method to repair the storage nodes |
US9244935B2 (en) * | 2013-06-14 | 2016-01-26 | International Business Machines Corporation | Data encoding and processing columnar data |
CN103440244A (zh) * | 2013-07-12 | 2013-12-11 | 广东电子工业研究院有限公司 | 一种大数据存储优化方法 |
-
2016
- 2016-12-30 CN CN201680090379.7A patent/CN110268397B/zh active Active
- 2016-12-30 WO PCT/CN2016/113364 patent/WO2018119976A1/fr active Application Filing
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20100235677A1 (en) * | 2007-09-21 | 2010-09-16 | Wylie Jay J | Generating A Parallel Recovery Plan For A Data Storage System |
CN101339524A (zh) * | 2008-05-22 | 2009-01-07 | 清华大学 | 大规模磁盘阵列存储***的磁盘容错方法 |
CN103186566A (zh) * | 2011-12-28 | 2013-07-03 | ***通信集团河北有限公司 | 一种数据分级存储方法、装置及*** |
CN103699676A (zh) * | 2013-12-30 | 2014-04-02 | 厦门市美亚柏科信息股份有限公司 | 基于mssql server表分区及自动维护方法及*** |
Non-Patent Citations (1)
Title |
---|
WANG, Z.: "Performance Optimization of a Massive Data Query and Analysis System on Hadoop", ELECTRONIC TECHNOLOGY & INFORMATION SCIENCE, CHINA MASTER'S THESES, 15 August 2015 (2015-08-15) * |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112579597A (zh) * | 2020-12-15 | 2021-03-30 | 西安邮电大学 | 一种压缩敏感的数据库文件存储方法及*** |
CN112579597B (zh) * | 2020-12-15 | 2023-03-21 | 西安邮电大学 | 一种压缩敏感的数据库文件存储方法及*** |
CN116931845A (zh) * | 2023-09-18 | 2023-10-24 | 新华三信息技术有限公司 | 一种数据布局方法、装置及电子设备 |
CN116931845B (zh) * | 2023-09-18 | 2023-12-12 | 新华三信息技术有限公司 | 一种数据布局方法、装置及电子设备 |
Also Published As
Publication number | Publication date |
---|---|
CN110268397A (zh) | 2019-09-20 |
CN110268397B (zh) | 2023-06-13 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20210021371A1 (en) | Distributed Storage System Data Management And Security | |
US10719250B2 (en) | System and method for combining erasure-coded protection sets | |
CN103944981B (zh) | 一种基于纠删码技术改进的云存储***及实现方法 | |
US20080313241A1 (en) | Distributed data storage using erasure resilient coding | |
CN101840366B (zh) | 环链式n+1位奇偶校验码的存储方法 | |
US20120089799A1 (en) | Data backup processing method, data storage node apparatus and data storage device | |
US11656942B2 (en) | Methods for data writing and for data recovery, electronic devices, and program products | |
CN114090345B (zh) | 一种磁盘阵列数据恢复方法、***、存储介质及设备 | |
WO2018119976A1 (fr) | Procédé d'optimisation d'implantation de données efficace destiné à un système d'entrepôt de données | |
CN105703782B (zh) | 一种基于递增移位矩阵的网络编码方法及*** | |
WO2023103213A1 (fr) | Procédé et dispositif de stockage de données pour base de données distribuée | |
WO2024021594A1 (fr) | Procédé et dispositif de codage pour réseau de disques raid6, procédé et dispositif de décodage pour réseau de disques raid6, et support | |
Esmaili et al. | CORE: Cross-object redundancy for efficient data repair in storage systems | |
CN102843212B (zh) | 编解码处理方法及装置 | |
WO2024001974A1 (fr) | Procédé et dispositif de récupération locale de données, et support de stockage | |
CN114115729B (zh) | 一种raid下的高效数据迁移方法 | |
CN107153661A (zh) | 一种基于hdfs***的数据的存储、读取方法及其装置 | |
US7831859B2 (en) | Method for providing fault tolerance to multiple servers | |
US10599520B2 (en) | Meta-copysets for fault-tolerant data storage | |
CN116248129A (zh) | 一种容错的数据分段压缩方法、恢复方法、设备及*** | |
CN106911793B (zh) | I/o优化的分布式存储数据修复方法 | |
WO2018209541A1 (fr) | Structure de codage sur la base de codes de répétition fractionnelle à conception en t, et procédé de codage | |
CN114756175A (zh) | 一种用于磁盘阵列的解码方法、***、设备及介质 | |
Tebbi et al. | Linear programming bounds for distributed storage codes | |
CN110781163A (zh) | 基于完全图的异构部分重复码构造及故障节点修复方法 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 16925712 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 16925712 Country of ref document: EP Kind code of ref document: A1 |