CN104536903B - A kind of mixing storage method and system stored classifiedly by data attribute - Google Patents

A kind of mixing storage method and system stored classifiedly by data attribute Download PDF

Info

Publication number
CN104536903B
CN104536903B CN201410825963.5A CN201410825963A CN104536903B CN 104536903 B CN104536903 B CN 104536903B CN 201410825963 A CN201410825963 A CN 201410825963A CN 104536903 B CN104536903 B CN 104536903B
Authority
CN
China
Prior art keywords
data
metadata
file system
block
lbn
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201410825963.5A
Other languages
Chinese (zh)
Other versions
CN104536903A (en
Inventor
陈俭喜
冯丹
刘景宁
吴婵明
方雪娇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huazhong University of Science and Technology
Original Assignee
Huazhong University of Science and Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huazhong University of Science and Technology filed Critical Huazhong University of Science and Technology
Priority to CN201410825963.5A priority Critical patent/CN104536903B/en
Publication of CN104536903A publication Critical patent/CN104536903A/en
Application granted granted Critical
Publication of CN104536903B publication Critical patent/CN104536903B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Landscapes

  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a kind of mixing storage method and system stored classifiedly by data attribute.Taken up space in view of metadata in file system the features such as small, access is frequent, construct a kind of mixed architecture being made up of hard disk drive and solid state hard disc, data in data attribute in file system is divided into data and metadata, place the data in the RAID5 being made up of HDD, metadata is stored in the RAID1 being made up of SSD, the data in file system and meta data category are stored in different medium, the purpose of the other equipment of different array classes to reach, lift file system I/O Performance And Reliabilities.On the one hand, file system data and metadata are respectively stored into multiple equipment, the I/O bandwidth of multiple equipment can be polymerize, improve the performance for accessing data;On the other hand, the characteristics of considering data and metadata, data and metadata is deposited using different equipment, systematic function and reliability is improved, has saved cost.

Description

A kind of mixing storage method and system stored classifiedly by data attribute
Technical field
The invention belongs to computer memory technical field, is mixed more particularly, to a kind of by what data attribute stored classifiedly Storage method and system are closed, can effectively lift the I/O Performance And Reliabilities of storage system.
Background technology
With the development of information technology, people need to handle and access increasing data, and file system is user The main path of data in storage device is accessed, therefore how efficiently file system use by organization and management mass data, raising The problem of efficiency that family accesses data is one important.I/O intensive applications make I/O subsystems turn into whole computer system Bottleneck, manager of the file system as bottom storage resource, it should provide for upper layer application and flexibly and effectively service, more to have Effect ground utilizes bottom storage resource.In addition, the management of metadata also becomes many data-intensive programs in file system Bottleneck.
File system metadata is exactly the data of data.In the corresponding metadata knot of each file of field of filesystems Structure body.This structure includes the attribute of file, such as document creation, modification and the time of access, file owner, authority, Map information etc..User must obtain the metadata information of data, further, the reading to file first before data are accessed The metadata for being required for first accessing file is write, checks whether there is the authority of access, and calculates the position for obtaining the file to be accessed And the position of specific data block is obtained, then data could be operated accordingly, so more than 50% in file system User access request be directed to the operation of metadata, intensive access of metadata I/O makes I/O subsystems in local file system Become the bottleneck of whole computer system.But metadata operation with the operation of data is to mix in local file system Together, because the load of metadata and the load difference of data are very big, when data-intensive operation and metadata are intensive When operation mixes the resource for concomitantly accessing local file system, the load of metadata will necessarily phase with the load of data Mutually influence, cause the decline of whole storage system I/O performances, make bottleneck effect of the I/O subsystems for whole computer system It is more obvious.
Once metadata is lost or made a mistake, the data in whole file system are by inaccessible, it is contemplated that file system In system the characteristics of metadata and metadata is to the importance of whole file system, a kind of effective solution method is needed badly to ensure The access efficiency of metadata and access reliability in file system.
The content of the invention
For the disadvantages described above or Improvement requirement of prior art, stored classifiedly the invention provides a kind of by data attribute Storage method and system are mixed, is sorted data into according to the attribute of data in file system and is stored in different medium, different arrays The equipment of rank, the access efficiency of metadata in file system can be effectively improved and access reliability, so as to improve storage system Performance And Reliability.
To achieve the above object, according to one aspect of the present invention, there is provided a kind of to be mixed by what data attribute stored classifiedly Close storage method, it is characterised in that comprise the following steps:(1) data of file system and metadata are distributed in file respectively The different sections of each block group of system;(2) data in different sections and metadata in each block group of file system are deposited respectively It is put into different equipment, data is deposited using hard disk drive, metadata is deposited using solid state hard disc.
Preferably, the step (1) further comprises following sub-step:Metadata is divided into the member of fixed allocation by (1-1) The metadata of data and dynamically distributes;Wherein, by the superblock of each block group of file system to the number between index node table According to the metadata as fixed allocation, the metadata using the directory block in file system as dynamically distributes;(1-2) calculation document The metadata of dynamically distributes needs the data block total number SumMetadata distributed in system;The each block of (1-3) in file system SumMetadata data block is reserved after the index node table of group;Wherein, last data block reserved in each block group Logical block number (LBN) is designated as M;(1-4) by the data allocations of dynamically distributes after the index node table of each block group of file system it is pre- In the SumMetadata data block stayed;Data distribution is more than to M data in the logical block number (LBN) of each block group of file system In block.
Preferably, the step (2) further comprises following sub-step:(2-1) creates RAID1 with solid state hard disc, with hard Disk drive creates RAID5;(2-2) creates RAID1 and RAID5 into a mixed logic equipment;(2-3) receives user to mixed The read-write requests that logical equipment is sent, are checked the logical block number (LBN) of request, according to the logical block number (LBN) of request, to ask into Row redirects.
Preferably, in the step (2-3), when the logical block number (LBN) of request is less than or equal to M, redirect requests to In RAID1;When the logical block number (LBN) of request is more than M, redirect requests in RAID5.
It is another aspect of this invention to provide that providing a kind of mixing storage system stored classifiedly by data attribute, it is special Sign is, including:Grouped accumulation module, for the data of file system and metadata to be distributed in each of file system respectively The different sections of block group;Mix storage address mapping block, for by the data in different sections in each block group of file system and Metadata is stored in different equipment respectively, and data are deposited using hard disk drive, and metadata is deposited using solid state hard disc.
Preferably, the grouped accumulation module includes:First module, for metadata to be divided into the metadata of fixed allocation With the metadata of dynamically distributes;Wherein, the superblock of each block group of file system is made to the data between index node table For the metadata of fixed allocation, the metadata using the directory block in file system as dynamically distributes;Second module, for calculating The metadata of dynamically distributes needs the data block total number SumMetadata distributed in file system;3rd module, in file SumMetadata data block is reserved after the index node table of each block group of system;Wherein, that is reserved in each block group is last The logical block number (LBN) of one data block is designated as M;4th module, for by the data allocations of dynamically distributes in each of file system In the SumMetadata data block reserved after the index node table of block group;By data distribution file system each block group Logical block number (LBN) more than M data block in.
Preferably, the mixing storage address mapping block includes:5th module, for creating RAID1 with solid state hard disc, RAID5 is created with hard disk drive;6th module, for RAID1 and RAID5 to be created into a mixed logic equipment;7th Module, the read-write requests sent for receiving user to mixed logic equipment, is checked the logical block number (LBN) of request, according to please The logical block number (LBN) asked, request is redirected.
Preferably, the 7th module redirects requests to RAID1 when the logical block number (LBN) of request is less than or equal to M In;When the logical block number (LBN) of request is more than M, redirect requests in RAID5.
The present invention is based on a kind of phenomenon of generally existing, i.e., the behaviour of the operation of metadata and data in local file system Work mixes, because the load of metadata and the load difference of data are very big, when data-intensive operation and first number When mixing the resource for concomitantly accessing local file system according to intensive, metadata load and the load of data must It can so influence each other, cause whole system I/O hydraulic performance declines.Therefore, the present invention considers that metadata is shared in file system Space is small, accesses the features such as frequent, constructs one kind by hard disk drive (Hard Disk Drive, HDD) and solid state hard disc The mixed architecture of (Solid State Drive, SSD) composition, is divided into data by the data in data attribute in file system (data) and metadata (metadata), place the data in HDD, metadata is stored in SSD, to reach file system Data and meta data category in system are stored in different medium, the purpose of the other equipment of different array classes so that file system I/O Performance And Reliability all gets a promotion.On the one hand, file system data and metadata are respectively stored into depositing for multiple Large Copacities Store up in equipment, the I/O bandwidth of multiple storage devices can be polymerize, improve the performance for accessing data;On the other hand, data are considered And the characteristics of metadata, data and metadata are deposited using different storage devices, so not only increase systematic function Cost is saved.
Brief description of the drawings
Fig. 1 is the mixing storage method flow chart stored classifiedly by data attribute of the embodiment of the present invention;
Fig. 2 is the principle schematic of the mixing storage method stored classifiedly by data attribute of the embodiment of the present invention;
Fig. 3 is mixing storage address mapping principle schematic diagram;
Fig. 4 is the process chart of user's request.
Embodiment
In order to make the purpose , technical scheme and advantage of the present invention be clearer, it is right below in conjunction with drawings and Examples The present invention is further elaborated.It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, and It is not used in the restriction present invention.As long as in addition, technical characteristic involved in each embodiment of invention described below Conflict can is not formed each other to be mutually combined.
As shown in figure 1, the mixing storage method stored classifiedly by data attribute of the embodiment of the present invention comprises the following steps:
(1) data of file system and metadata are distributed in the different sections of each block group of file system respectively.
Further comprise following sub-step:
Metadata is divided into the metadata of fixed allocation and the metadata of dynamically distributes by (1-1);Wherein, by file system Metadata of the superblock of each block group to the data between index node table as fixed allocation, by the catalogue in file system Metadata of the block as dynamically distributes.
The metadata of dynamically distributes needs the data block total number SumMetadata distributed in (1-2) calculation document system.
(1-3) reserves SumMetadata data block after the index node table of each block group of file system;Wherein, The logical block number (LBN) for last data block reserved in each block group is designated as M.
(1-4) reserves the data allocations of dynamically distributes after the index node table of each block group of file system In SumMetadata data block;Data distribution is more than to M data block in the logical block number (LBN) of each block group of file system In.
By the data block allocation strategy for changing file system so that all metadata in file system concentrate on one Rise.
(2) data in different sections and metadata in each block group of file system are stored in different equipment respectively, Data are deposited using hard disk drive, metadata is deposited using solid state hard disc.
Further comprise following sub-step:
(2-1) creates RAID1 with solid state hard disc, and RAID5 is created with hard disk drive.
Under the Device mapper frameworks of (2-2) in linux kernel, RAID1 and RAID5 is created into a mixing Logical device.
(2-3) receives the read-write requests that user sends to mixed logic equipment, the logical block number (LBN) of request is checked, root According to the logical block number (LBN) of request, request is redirected in Device mapper mapping function.Specifically, in request When logical block number (LBN) is less than or equal to M, redirect requests in RAID1;When the logical block number (LBN) of request is more than M, by request weight It is directed in RAID5.
Wherein, when metadata and data are stored in different equipment respectively, and the migration of data need not be carried out, Only different equipment need to be redirected that to when request issues, reduce the cost of Data Migration and deposited in internal memory Put the expense of the mapping table of address of cache.
It is described in detail with reference to Fig. 2~4 pair mixing storage method stored classifiedly by data attribute of the invention.
As shown in Fig. 2 the mixing storage method stored classifiedly by data attribute is mainly made up of two parts:In file system Data by attributive classification aggregation 100 and mixing storage address mapping 200.
File system includes Virtual File System 110 and specific file system 120.Data are mainly realized in file system Assemble by attributive classification, need not be modified for Virtual File System, for certain specific file system, such as Ext2, by control data block allocation strategy, using last logical block number (LBN) M of data block reserved in block group as boundary, make Obtain metadata and data are left concentratedly in former and later two parts of each piece of group respectively, realize that data are by attribute point in file system Class is assembled.Enable in this way when mixing storage system progress address of cache by simply checking logical block number (LBN) (LBN) whether it is less than or equal to or more than M with regard to request of data and metadata request can be distinguished, so as to which the address for realizing different is reflected Penetrate.
Mix in storage address mapping 200, the logic block device of the establishment part 210 of logic block device mainly exists The mixed logic equipment created under Device mapper, it is used for checking the logic of upper layer request including a mapping algorithm The magnitude relationship of block number and M, to judge that upper layer request is metadata request or request of data, then selected according to judged result Select and redirect request into the RAID5 220 of RAID1 230 or the HDD composition of SSD compositions.In this way, storage system passes through letter Single judgement can just navigate to metadata and data different storage mediums, on the array of different stage, so not only carry The access efficiency of high metadata deposits the reliability that metadata also improves metadata using RAID1 simultaneously.
Fig. 3 is the principle schematic of mixing storage address mapping.For a user, can only be to by the Hes of RAID1 250 The logic block device 260 that RAID5 240 is formed is operated, and can carry out subregion and establishment file system to the logic block device System.The physical address of the logic block device corresponds with RAID5, therefore RAID5 houses wherein all data (data) Request, and RAID1 is only intended to the metadata request (metadata) after storage is redirected by mapping algorithm, so just SSD can be met relative to the small feature of HDD capacity.
Fig. 4 is the process chart of user's request.All read-writes of logic block device 260 are issued to firstly, for user Request, after the system receives, data are divided by attribute by the file system 120 that have modified data block allocation strategy Class is assembled, and is then issued request and is realized address of cache to mixed logic equipment.Because by the data block point for changing file system Metadata and data have been concentrated on to former and later two portions of each piece of group respectively with strategy in each piece of group using M as boundary Point, so all requests for reaching mixed logic equipment, it is only necessary to the LBN of request is checked just by mapping algorithm Can interpolate that the request is metadata request or request of data, so as to redirect requests to different storage device RAID1 In 230 or RAID5 220.
All processes of summary so that mixing storage system realizes attribute using data by metadata and data Separately reliability of the storage to improve the performance of whole storage system and ensure metadata storage.
As it will be easily appreciated by one skilled in the art that the foregoing is merely illustrative of the preferred embodiments of the present invention, not to The limitation present invention, all any modification, equivalent and improvement made within the spirit and principles of the invention etc., all should be included Within protection scope of the present invention.

Claims (4)

1. a kind of mixing storage method stored classifiedly by data attribute, it is characterised in that comprise the following steps:
(1) data of file system and metadata are distributed in the different sections of each block group of file system respectively, specific bag Include following sub-step:
Metadata is divided into the metadata of fixed allocation and the metadata of dynamically distributes by (1-1);Wherein, by each of file system Metadata of the superblock of block group to the data between index node table as fixed allocation, the directory block in file system is made For the metadata of dynamically distributes;
The metadata of dynamically distributes needs the data block total number SumMetadata distributed in (1-2) calculation document system;
(1-3) reserves SumMetadata data block after the index node table of each block group of file system;Wherein, each The logical block number (LBN) for last data block reserved in block group is designated as M;
(1-4) reserves the data allocations of dynamically distributes after the index node table of each block group of file system In SumMetadata data block;Data distribution is more than to M data block in the logical block number (LBN) of each block group of file system In;
(2) data in different sections and metadata in each block group of file system are stored in different equipment respectively, used Hard disk drive deposits data, and metadata is deposited using solid state hard disc;
Wherein, step (2) further comprises following sub-step:
(2-1) creates RAID1 with solid state hard disc, and RAID5 is created with hard disk drive;
(2-2) creates RAID1 and RAID5 into a mixed logic equipment;
(2-3) receives the read-write requests that are sent to mixed logic equipment of user, and the logical block number (LBN) of request is checked, according to asking The logical block number (LBN) asked, request is redirected.
2. the mixing storage method stored classifiedly as claimed in claim 1 by data attribute, it is characterised in that the step In (2-3), when the logical block number (LBN) of request is less than or equal to M, redirect requests in RAID1;In the logical block number (LBN) of request During more than M, redirect requests in RAID5.
A kind of 3. mixing storage system stored classifiedly by data attribute, it is characterised in that including:
Grouped accumulation module, for by the data of file system and metadata distribute respectively file system each block group not Same section, wherein, the grouped accumulation module includes:
First module, for metadata to be divided into the metadata of fixed allocation and the metadata of dynamically distributes;Wherein, by file system Metadata of the superblock of each block group of system to the data between index node table as fixed allocation, by file system Metadata of the directory block as dynamically distributes;
Second module, the metadata for dynamically distributes in calculation document system need the data block total number distributed SumMetadata;
3rd module, for reserving SumMetadata data block after the index node table of each block group of file system;Its In, the logical block number (LBN) for last data block reserved in each block group is designated as M;
4th module, for the data allocations of dynamically distributes to be reserved after the index node table of each block group of file system SumMetadata data block in;Data distribution is more than to M data block in the logical block number (LBN) of each block group of file system In;
Storage address mapping block is mixed, for the data in different sections and metadata in each block group of file system to be deposited respectively It is put into different equipment, data is deposited using hard disk drive, metadata is deposited using solid state hard disc;
Wherein, the mixing storage address mapping block includes:
5th module, for creating RAID1 with solid state hard disc, RAID5 is created with hard disk drive;
6th module, for RAID1 and RAID5 to be created into a mixed logic equipment;
7th module, the read-write requests sent for receiving user to mixed logic equipment, is examined to the logical block number (LBN) of request Look into, according to the logical block number (LBN) of request, request is redirected.
4. the mixing storage system stored classifiedly as claimed in claim 3 by data attribute, it is characterised in that the 7th mould Block is redirected requests in RAID1 when the logical block number (LBN) of request is less than or equal to M;It is more than M in the logical block number (LBN) of request When, redirect requests in RAID5.
CN201410825963.5A 2014-12-25 2014-12-25 A kind of mixing storage method and system stored classifiedly by data attribute Active CN104536903B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201410825963.5A CN104536903B (en) 2014-12-25 2014-12-25 A kind of mixing storage method and system stored classifiedly by data attribute

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201410825963.5A CN104536903B (en) 2014-12-25 2014-12-25 A kind of mixing storage method and system stored classifiedly by data attribute

Publications (2)

Publication Number Publication Date
CN104536903A CN104536903A (en) 2015-04-22
CN104536903B true CN104536903B (en) 2018-02-23

Family

ID=52852433

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201410825963.5A Active CN104536903B (en) 2014-12-25 2014-12-25 A kind of mixing storage method and system stored classifiedly by data attribute

Country Status (1)

Country Link
CN (1) CN104536903B (en)

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105138286A (en) * 2015-08-11 2015-12-09 智云创新(北京)科技有限公司 Method for mixed utilization of SSD and SMR hard disks in disk file system
US9880772B2 (en) * 2015-09-21 2018-01-30 Micron Technology, Inc. Systems and methods for providing file information in a memory system protocol
CN107766374B (en) * 2016-08-19 2021-05-25 上海凯翔信息科技有限公司 Optimization method and system for storage and reading of massive small files
CN106599236A (en) * 2016-12-20 2017-04-26 北海市云盛科技有限公司 Metadata storage method and apparatus for file system
CN107479825A (en) * 2017-06-30 2017-12-15 华为技术有限公司 A kind of storage system, solid state hard disc and date storage method
CN107479827A (en) * 2017-07-24 2017-12-15 上海德拓信息技术股份有限公司 A kind of mixing storage system implementation method based on IO and separated from meta-data
CN108154463B (en) * 2017-12-06 2021-12-24 中国航空工业集团公司西安航空计算技术研究所 Method for managing modeled GPU (graphics processing Unit) video memory system
CN109002266A (en) * 2018-08-29 2018-12-14 郑州云海信息技术有限公司 A method of promoting metadata reliability in traditional RAID group
CN111190550B (en) * 2019-12-31 2024-03-29 深圳市安云信息科技有限公司 Metadata acceleration method and device and storage equipment
CN112000296B (en) * 2020-08-28 2024-04-09 北京计算机技术及应用研究所 Performance optimization system in full flash memory array
CN115599704B (en) * 2022-11-30 2023-03-17 湖南国科亿存信息科技有限公司 File system metadata separate storage method and device and storage medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102646079A (en) * 2012-04-23 2012-08-22 中国人民解放军国防科学技术大学 Disk data protection method oriented to Linux operating system
CN102821138A (en) * 2012-07-09 2012-12-12 广州鼎鼎信息科技有限公司 Metadata distributed storage method applicable to cloud storage system
CN103838853A (en) * 2014-03-17 2014-06-04 华中科技大学 Mixed file system based on different storage media
CN103902669A (en) * 2014-03-17 2014-07-02 华中科技大学 Separating type file system based on different storage media

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5524144B2 (en) * 2011-08-08 2014-06-18 株式会社東芝 Memory system having a key-value store system

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102646079A (en) * 2012-04-23 2012-08-22 中国人民解放军国防科学技术大学 Disk data protection method oriented to Linux operating system
CN102821138A (en) * 2012-07-09 2012-12-12 广州鼎鼎信息科技有限公司 Metadata distributed storage method applicable to cloud storage system
CN103838853A (en) * 2014-03-17 2014-06-04 华中科技大学 Mixed file system based on different storage media
CN103902669A (en) * 2014-03-17 2014-07-02 华中科技大学 Separating type file system based on different storage media

Also Published As

Publication number Publication date
CN104536903A (en) 2015-04-22

Similar Documents

Publication Publication Date Title
CN104536903B (en) A kind of mixing storage method and system stored classifiedly by data attribute
KR101930117B1 (en) Volatile memory representation of nonvolatile storage device set
US9710187B1 (en) Managing data relocation in storage systems
US9395937B1 (en) Managing storage space in storage systems
CN107046563B (en) A kind of implementation method, system and the cloud platform of distribution type high efficient cloud disk
US8443163B1 (en) Methods, systems, and computer readable medium for tier-based data storage resource allocation and data relocation in a data storage array
US9529545B1 (en) Managing data deduplication in storage systems based on storage space characteristics
US9817766B1 (en) Managing relocation of slices in storage systems
US8775731B2 (en) Write spike performance enhancement in hybrid storage systems
US10102210B2 (en) Systems and methods for multi-threaded shadow migration
US9400792B1 (en) File system inline fine grained tiering
US20160103764A1 (en) Methods and systems for cache management in storage systems
US10871911B2 (en) Reducing data amplification when replicating objects across different sites
JP2016528617A (en) Resource load balancing
US9355121B1 (en) Segregating data and metadata in a file system
US20200349121A1 (en) File system for anonymous write
US11803329B2 (en) Methods and systems for processing write requests in a storage system
CN104731517A (en) Method and device for allocating capacity of storage pool
CN106960011A (en) Metadata of distributed type file system management system and method
US10057348B2 (en) Storage fabric address based data block retrieval
US9069471B2 (en) Passing hint of page allocation of thin provisioning with multiple virtual volumes fit to parallel data access
US8504764B2 (en) Method and apparatus to manage object-based tiers
US8725979B1 (en) Efficient methods and systems for allocating storage volumes
CN105354294A (en) Nested file management system and method
CN105404471A (en) Distributed cloud storage cache layer realization method

Legal Events

Date Code Title Description
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant