CN104536903B - A kind of mixing storage method and system stored classifiedly by data attribute - Google Patents
A kind of mixing storage method and system stored classifiedly by data attribute Download PDFInfo
- Publication number
- CN104536903B CN104536903B CN201410825963.5A CN201410825963A CN104536903B CN 104536903 B CN104536903 B CN 104536903B CN 201410825963 A CN201410825963 A CN 201410825963A CN 104536903 B CN104536903 B CN 104536903B
- Authority
- CN
- China
- Prior art keywords
- data
- metadata
- file system
- block
- lbn
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of mixing storage method and system stored classifiedly by data attribute.Taken up space in view of metadata in file system the features such as small, access is frequent, construct a kind of mixed architecture being made up of hard disk drive and solid state hard disc, data in data attribute in file system is divided into data and metadata, place the data in the RAID5 being made up of HDD, metadata is stored in the RAID1 being made up of SSD, the data in file system and meta data category are stored in different medium, the purpose of the other equipment of different array classes to reach, lift file system I/O Performance And Reliabilities.On the one hand, file system data and metadata are respectively stored into multiple equipment, the I/O bandwidth of multiple equipment can be polymerize, improve the performance for accessing data;On the other hand, the characteristics of considering data and metadata, data and metadata is deposited using different equipment, systematic function and reliability is improved, has saved cost.
Description
Technical field
The invention belongs to computer memory technical field, is mixed more particularly, to a kind of by what data attribute stored classifiedly
Storage method and system are closed, can effectively lift the I/O Performance And Reliabilities of storage system.
Background technology
With the development of information technology, people need to handle and access increasing data, and file system is user
The main path of data in storage device is accessed, therefore how efficiently file system use by organization and management mass data, raising
The problem of efficiency that family accesses data is one important.I/O intensive applications make I/O subsystems turn into whole computer system
Bottleneck, manager of the file system as bottom storage resource, it should provide for upper layer application and flexibly and effectively service, more to have
Effect ground utilizes bottom storage resource.In addition, the management of metadata also becomes many data-intensive programs in file system
Bottleneck.
File system metadata is exactly the data of data.In the corresponding metadata knot of each file of field of filesystems
Structure body.This structure includes the attribute of file, such as document creation, modification and the time of access, file owner, authority,
Map information etc..User must obtain the metadata information of data, further, the reading to file first before data are accessed
The metadata for being required for first accessing file is write, checks whether there is the authority of access, and calculates the position for obtaining the file to be accessed
And the position of specific data block is obtained, then data could be operated accordingly, so more than 50% in file system
User access request be directed to the operation of metadata, intensive access of metadata I/O makes I/O subsystems in local file system
Become the bottleneck of whole computer system.But metadata operation with the operation of data is to mix in local file system
Together, because the load of metadata and the load difference of data are very big, when data-intensive operation and metadata are intensive
When operation mixes the resource for concomitantly accessing local file system, the load of metadata will necessarily phase with the load of data
Mutually influence, cause the decline of whole storage system I/O performances, make bottleneck effect of the I/O subsystems for whole computer system
It is more obvious.
Once metadata is lost or made a mistake, the data in whole file system are by inaccessible, it is contemplated that file system
In system the characteristics of metadata and metadata is to the importance of whole file system, a kind of effective solution method is needed badly to ensure
The access efficiency of metadata and access reliability in file system.
The content of the invention
For the disadvantages described above or Improvement requirement of prior art, stored classifiedly the invention provides a kind of by data attribute
Storage method and system are mixed, is sorted data into according to the attribute of data in file system and is stored in different medium, different arrays
The equipment of rank, the access efficiency of metadata in file system can be effectively improved and access reliability, so as to improve storage system
Performance And Reliability.
To achieve the above object, according to one aspect of the present invention, there is provided a kind of to be mixed by what data attribute stored classifiedly
Close storage method, it is characterised in that comprise the following steps:(1) data of file system and metadata are distributed in file respectively
The different sections of each block group of system;(2) data in different sections and metadata in each block group of file system are deposited respectively
It is put into different equipment, data is deposited using hard disk drive, metadata is deposited using solid state hard disc.
Preferably, the step (1) further comprises following sub-step:Metadata is divided into the member of fixed allocation by (1-1)
The metadata of data and dynamically distributes;Wherein, by the superblock of each block group of file system to the number between index node table
According to the metadata as fixed allocation, the metadata using the directory block in file system as dynamically distributes;(1-2) calculation document
The metadata of dynamically distributes needs the data block total number SumMetadata distributed in system;The each block of (1-3) in file system
SumMetadata data block is reserved after the index node table of group;Wherein, last data block reserved in each block group
Logical block number (LBN) is designated as M;(1-4) by the data allocations of dynamically distributes after the index node table of each block group of file system it is pre-
In the SumMetadata data block stayed;Data distribution is more than to M data in the logical block number (LBN) of each block group of file system
In block.
Preferably, the step (2) further comprises following sub-step:(2-1) creates RAID1 with solid state hard disc, with hard
Disk drive creates RAID5;(2-2) creates RAID1 and RAID5 into a mixed logic equipment;(2-3) receives user to mixed
The read-write requests that logical equipment is sent, are checked the logical block number (LBN) of request, according to the logical block number (LBN) of request, to ask into
Row redirects.
Preferably, in the step (2-3), when the logical block number (LBN) of request is less than or equal to M, redirect requests to
In RAID1;When the logical block number (LBN) of request is more than M, redirect requests in RAID5.
It is another aspect of this invention to provide that providing a kind of mixing storage system stored classifiedly by data attribute, it is special
Sign is, including:Grouped accumulation module, for the data of file system and metadata to be distributed in each of file system respectively
The different sections of block group;Mix storage address mapping block, for by the data in different sections in each block group of file system and
Metadata is stored in different equipment respectively, and data are deposited using hard disk drive, and metadata is deposited using solid state hard disc.
Preferably, the grouped accumulation module includes:First module, for metadata to be divided into the metadata of fixed allocation
With the metadata of dynamically distributes;Wherein, the superblock of each block group of file system is made to the data between index node table
For the metadata of fixed allocation, the metadata using the directory block in file system as dynamically distributes;Second module, for calculating
The metadata of dynamically distributes needs the data block total number SumMetadata distributed in file system;3rd module, in file
SumMetadata data block is reserved after the index node table of each block group of system;Wherein, that is reserved in each block group is last
The logical block number (LBN) of one data block is designated as M;4th module, for by the data allocations of dynamically distributes in each of file system
In the SumMetadata data block reserved after the index node table of block group;By data distribution file system each block group
Logical block number (LBN) more than M data block in.
Preferably, the mixing storage address mapping block includes:5th module, for creating RAID1 with solid state hard disc,
RAID5 is created with hard disk drive;6th module, for RAID1 and RAID5 to be created into a mixed logic equipment;7th
Module, the read-write requests sent for receiving user to mixed logic equipment, is checked the logical block number (LBN) of request, according to please
The logical block number (LBN) asked, request is redirected.
Preferably, the 7th module redirects requests to RAID1 when the logical block number (LBN) of request is less than or equal to M
In;When the logical block number (LBN) of request is more than M, redirect requests in RAID5.
The present invention is based on a kind of phenomenon of generally existing, i.e., the behaviour of the operation of metadata and data in local file system
Work mixes, because the load of metadata and the load difference of data are very big, when data-intensive operation and first number
When mixing the resource for concomitantly accessing local file system according to intensive, metadata load and the load of data must
It can so influence each other, cause whole system I/O hydraulic performance declines.Therefore, the present invention considers that metadata is shared in file system
Space is small, accesses the features such as frequent, constructs one kind by hard disk drive (Hard Disk Drive, HDD) and solid state hard disc
The mixed architecture of (Solid State Drive, SSD) composition, is divided into data by the data in data attribute in file system
(data) and metadata (metadata), place the data in HDD, metadata is stored in SSD, to reach file system
Data and meta data category in system are stored in different medium, the purpose of the other equipment of different array classes so that file system I/O
Performance And Reliability all gets a promotion.On the one hand, file system data and metadata are respectively stored into depositing for multiple Large Copacities
Store up in equipment, the I/O bandwidth of multiple storage devices can be polymerize, improve the performance for accessing data;On the other hand, data are considered
And the characteristics of metadata, data and metadata are deposited using different storage devices, so not only increase systematic function
Cost is saved.
Brief description of the drawings
Fig. 1 is the mixing storage method flow chart stored classifiedly by data attribute of the embodiment of the present invention;
Fig. 2 is the principle schematic of the mixing storage method stored classifiedly by data attribute of the embodiment of the present invention;
Fig. 3 is mixing storage address mapping principle schematic diagram;
Fig. 4 is the process chart of user's request.
Embodiment
In order to make the purpose , technical scheme and advantage of the present invention be clearer, it is right below in conjunction with drawings and Examples
The present invention is further elaborated.It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, and
It is not used in the restriction present invention.As long as in addition, technical characteristic involved in each embodiment of invention described below
Conflict can is not formed each other to be mutually combined.
As shown in figure 1, the mixing storage method stored classifiedly by data attribute of the embodiment of the present invention comprises the following steps:
(1) data of file system and metadata are distributed in the different sections of each block group of file system respectively.
Further comprise following sub-step:
Metadata is divided into the metadata of fixed allocation and the metadata of dynamically distributes by (1-1);Wherein, by file system
Metadata of the superblock of each block group to the data between index node table as fixed allocation, by the catalogue in file system
Metadata of the block as dynamically distributes.
The metadata of dynamically distributes needs the data block total number SumMetadata distributed in (1-2) calculation document system.
(1-3) reserves SumMetadata data block after the index node table of each block group of file system;Wherein,
The logical block number (LBN) for last data block reserved in each block group is designated as M.
(1-4) reserves the data allocations of dynamically distributes after the index node table of each block group of file system
In SumMetadata data block;Data distribution is more than to M data block in the logical block number (LBN) of each block group of file system
In.
By the data block allocation strategy for changing file system so that all metadata in file system concentrate on one
Rise.
(2) data in different sections and metadata in each block group of file system are stored in different equipment respectively,
Data are deposited using hard disk drive, metadata is deposited using solid state hard disc.
Further comprise following sub-step:
(2-1) creates RAID1 with solid state hard disc, and RAID5 is created with hard disk drive.
Under the Device mapper frameworks of (2-2) in linux kernel, RAID1 and RAID5 is created into a mixing
Logical device.
(2-3) receives the read-write requests that user sends to mixed logic equipment, the logical block number (LBN) of request is checked, root
According to the logical block number (LBN) of request, request is redirected in Device mapper mapping function.Specifically, in request
When logical block number (LBN) is less than or equal to M, redirect requests in RAID1;When the logical block number (LBN) of request is more than M, by request weight
It is directed in RAID5.
Wherein, when metadata and data are stored in different equipment respectively, and the migration of data need not be carried out,
Only different equipment need to be redirected that to when request issues, reduce the cost of Data Migration and deposited in internal memory
Put the expense of the mapping table of address of cache.
It is described in detail with reference to Fig. 2~4 pair mixing storage method stored classifiedly by data attribute of the invention.
As shown in Fig. 2 the mixing storage method stored classifiedly by data attribute is mainly made up of two parts:In file system
Data by attributive classification aggregation 100 and mixing storage address mapping 200.
File system includes Virtual File System 110 and specific file system 120.Data are mainly realized in file system
Assemble by attributive classification, need not be modified for Virtual File System, for certain specific file system, such as
Ext2, by control data block allocation strategy, using last logical block number (LBN) M of data block reserved in block group as boundary, make
Obtain metadata and data are left concentratedly in former and later two parts of each piece of group respectively, realize that data are by attribute point in file system
Class is assembled.Enable in this way when mixing storage system progress address of cache by simply checking logical block number (LBN)
(LBN) whether it is less than or equal to or more than M with regard to request of data and metadata request can be distinguished, so as to which the address for realizing different is reflected
Penetrate.
Mix in storage address mapping 200, the logic block device of the establishment part 210 of logic block device mainly exists
The mixed logic equipment created under Device mapper, it is used for checking the logic of upper layer request including a mapping algorithm
The magnitude relationship of block number and M, to judge that upper layer request is metadata request or request of data, then selected according to judged result
Select and redirect request into the RAID5 220 of RAID1 230 or the HDD composition of SSD compositions.In this way, storage system passes through letter
Single judgement can just navigate to metadata and data different storage mediums, on the array of different stage, so not only carry
The access efficiency of high metadata deposits the reliability that metadata also improves metadata using RAID1 simultaneously.
Fig. 3 is the principle schematic of mixing storage address mapping.For a user, can only be to by the Hes of RAID1 250
The logic block device 260 that RAID5 240 is formed is operated, and can carry out subregion and establishment file system to the logic block device
System.The physical address of the logic block device corresponds with RAID5, therefore RAID5 houses wherein all data (data)
Request, and RAID1 is only intended to the metadata request (metadata) after storage is redirected by mapping algorithm, so just
SSD can be met relative to the small feature of HDD capacity.
Fig. 4 is the process chart of user's request.All read-writes of logic block device 260 are issued to firstly, for user
Request, after the system receives, data are divided by attribute by the file system 120 that have modified data block allocation strategy
Class is assembled, and is then issued request and is realized address of cache to mixed logic equipment.Because by the data block point for changing file system
Metadata and data have been concentrated on to former and later two portions of each piece of group respectively with strategy in each piece of group using M as boundary
Point, so all requests for reaching mixed logic equipment, it is only necessary to the LBN of request is checked just by mapping algorithm
Can interpolate that the request is metadata request or request of data, so as to redirect requests to different storage device RAID1
In 230 or RAID5 220.
All processes of summary so that mixing storage system realizes attribute using data by metadata and data
Separately reliability of the storage to improve the performance of whole storage system and ensure metadata storage.
As it will be easily appreciated by one skilled in the art that the foregoing is merely illustrative of the preferred embodiments of the present invention, not to
The limitation present invention, all any modification, equivalent and improvement made within the spirit and principles of the invention etc., all should be included
Within protection scope of the present invention.
Claims (4)
1. a kind of mixing storage method stored classifiedly by data attribute, it is characterised in that comprise the following steps:
(1) data of file system and metadata are distributed in the different sections of each block group of file system respectively, specific bag
Include following sub-step:
Metadata is divided into the metadata of fixed allocation and the metadata of dynamically distributes by (1-1);Wherein, by each of file system
Metadata of the superblock of block group to the data between index node table as fixed allocation, the directory block in file system is made
For the metadata of dynamically distributes;
The metadata of dynamically distributes needs the data block total number SumMetadata distributed in (1-2) calculation document system;
(1-3) reserves SumMetadata data block after the index node table of each block group of file system;Wherein, each
The logical block number (LBN) for last data block reserved in block group is designated as M;
(1-4) reserves the data allocations of dynamically distributes after the index node table of each block group of file system
In SumMetadata data block;Data distribution is more than to M data block in the logical block number (LBN) of each block group of file system
In;
(2) data in different sections and metadata in each block group of file system are stored in different equipment respectively, used
Hard disk drive deposits data, and metadata is deposited using solid state hard disc;
Wherein, step (2) further comprises following sub-step:
(2-1) creates RAID1 with solid state hard disc, and RAID5 is created with hard disk drive;
(2-2) creates RAID1 and RAID5 into a mixed logic equipment;
(2-3) receives the read-write requests that are sent to mixed logic equipment of user, and the logical block number (LBN) of request is checked, according to asking
The logical block number (LBN) asked, request is redirected.
2. the mixing storage method stored classifiedly as claimed in claim 1 by data attribute, it is characterised in that the step
In (2-3), when the logical block number (LBN) of request is less than or equal to M, redirect requests in RAID1;In the logical block number (LBN) of request
During more than M, redirect requests in RAID5.
A kind of 3. mixing storage system stored classifiedly by data attribute, it is characterised in that including:
Grouped accumulation module, for by the data of file system and metadata distribute respectively file system each block group not
Same section, wherein, the grouped accumulation module includes:
First module, for metadata to be divided into the metadata of fixed allocation and the metadata of dynamically distributes;Wherein, by file system
Metadata of the superblock of each block group of system to the data between index node table as fixed allocation, by file system
Metadata of the directory block as dynamically distributes;
Second module, the metadata for dynamically distributes in calculation document system need the data block total number distributed
SumMetadata;
3rd module, for reserving SumMetadata data block after the index node table of each block group of file system;Its
In, the logical block number (LBN) for last data block reserved in each block group is designated as M;
4th module, for the data allocations of dynamically distributes to be reserved after the index node table of each block group of file system
SumMetadata data block in;Data distribution is more than to M data block in the logical block number (LBN) of each block group of file system
In;
Storage address mapping block is mixed, for the data in different sections and metadata in each block group of file system to be deposited respectively
It is put into different equipment, data is deposited using hard disk drive, metadata is deposited using solid state hard disc;
Wherein, the mixing storage address mapping block includes:
5th module, for creating RAID1 with solid state hard disc, RAID5 is created with hard disk drive;
6th module, for RAID1 and RAID5 to be created into a mixed logic equipment;
7th module, the read-write requests sent for receiving user to mixed logic equipment, is examined to the logical block number (LBN) of request
Look into, according to the logical block number (LBN) of request, request is redirected.
4. the mixing storage system stored classifiedly as claimed in claim 3 by data attribute, it is characterised in that the 7th mould
Block is redirected requests in RAID1 when the logical block number (LBN) of request is less than or equal to M;It is more than M in the logical block number (LBN) of request
When, redirect requests in RAID5.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201410825963.5A CN104536903B (en) | 2014-12-25 | 2014-12-25 | A kind of mixing storage method and system stored classifiedly by data attribute |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201410825963.5A CN104536903B (en) | 2014-12-25 | 2014-12-25 | A kind of mixing storage method and system stored classifiedly by data attribute |
Publications (2)
Publication Number | Publication Date |
---|---|
CN104536903A CN104536903A (en) | 2015-04-22 |
CN104536903B true CN104536903B (en) | 2018-02-23 |
Family
ID=52852433
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201410825963.5A Active CN104536903B (en) | 2014-12-25 | 2014-12-25 | A kind of mixing storage method and system stored classifiedly by data attribute |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN104536903B (en) |
Families Citing this family (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN105138286A (en) * | 2015-08-11 | 2015-12-09 | 智云创新(北京)科技有限公司 | Method for mixed utilization of SSD and SMR hard disks in disk file system |
US9880772B2 (en) * | 2015-09-21 | 2018-01-30 | Micron Technology, Inc. | Systems and methods for providing file information in a memory system protocol |
CN107766374B (en) * | 2016-08-19 | 2021-05-25 | 上海凯翔信息科技有限公司 | Optimization method and system for storage and reading of massive small files |
CN106599236A (en) * | 2016-12-20 | 2017-04-26 | 北海市云盛科技有限公司 | Metadata storage method and apparatus for file system |
CN107479825A (en) * | 2017-06-30 | 2017-12-15 | 华为技术有限公司 | A kind of storage system, solid state hard disc and date storage method |
CN107479827A (en) * | 2017-07-24 | 2017-12-15 | 上海德拓信息技术股份有限公司 | A kind of mixing storage system implementation method based on IO and separated from meta-data |
CN108154463B (en) * | 2017-12-06 | 2021-12-24 | 中国航空工业集团公司西安航空计算技术研究所 | Method for managing modeled GPU (graphics processing Unit) video memory system |
CN109002266A (en) * | 2018-08-29 | 2018-12-14 | 郑州云海信息技术有限公司 | A method of promoting metadata reliability in traditional RAID group |
CN111190550B (en) * | 2019-12-31 | 2024-03-29 | 深圳市安云信息科技有限公司 | Metadata acceleration method and device and storage equipment |
CN112000296B (en) * | 2020-08-28 | 2024-04-09 | 北京计算机技术及应用研究所 | Performance optimization system in full flash memory array |
CN115599704B (en) * | 2022-11-30 | 2023-03-17 | 湖南国科亿存信息科技有限公司 | File system metadata separate storage method and device and storage medium |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102646079A (en) * | 2012-04-23 | 2012-08-22 | 中国人民解放军国防科学技术大学 | Disk data protection method oriented to Linux operating system |
CN102821138A (en) * | 2012-07-09 | 2012-12-12 | 广州鼎鼎信息科技有限公司 | Metadata distributed storage method applicable to cloud storage system |
CN103838853A (en) * | 2014-03-17 | 2014-06-04 | 华中科技大学 | Mixed file system based on different storage media |
CN103902669A (en) * | 2014-03-17 | 2014-07-02 | 华中科技大学 | Separating type file system based on different storage media |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP5524144B2 (en) * | 2011-08-08 | 2014-06-18 | 株式会社東芝 | Memory system having a key-value store system |
-
2014
- 2014-12-25 CN CN201410825963.5A patent/CN104536903B/en active Active
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102646079A (en) * | 2012-04-23 | 2012-08-22 | 中国人民解放军国防科学技术大学 | Disk data protection method oriented to Linux operating system |
CN102821138A (en) * | 2012-07-09 | 2012-12-12 | 广州鼎鼎信息科技有限公司 | Metadata distributed storage method applicable to cloud storage system |
CN103838853A (en) * | 2014-03-17 | 2014-06-04 | 华中科技大学 | Mixed file system based on different storage media |
CN103902669A (en) * | 2014-03-17 | 2014-07-02 | 华中科技大学 | Separating type file system based on different storage media |
Also Published As
Publication number | Publication date |
---|---|
CN104536903A (en) | 2015-04-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN104536903B (en) | A kind of mixing storage method and system stored classifiedly by data attribute | |
KR101930117B1 (en) | Volatile memory representation of nonvolatile storage device set | |
US9710187B1 (en) | Managing data relocation in storage systems | |
US9395937B1 (en) | Managing storage space in storage systems | |
CN107046563B (en) | A kind of implementation method, system and the cloud platform of distribution type high efficient cloud disk | |
US8443163B1 (en) | Methods, systems, and computer readable medium for tier-based data storage resource allocation and data relocation in a data storage array | |
US9529545B1 (en) | Managing data deduplication in storage systems based on storage space characteristics | |
US9817766B1 (en) | Managing relocation of slices in storage systems | |
US8775731B2 (en) | Write spike performance enhancement in hybrid storage systems | |
US10102210B2 (en) | Systems and methods for multi-threaded shadow migration | |
US9400792B1 (en) | File system inline fine grained tiering | |
US20160103764A1 (en) | Methods and systems for cache management in storage systems | |
US10871911B2 (en) | Reducing data amplification when replicating objects across different sites | |
JP2016528617A (en) | Resource load balancing | |
US9355121B1 (en) | Segregating data and metadata in a file system | |
US20200349121A1 (en) | File system for anonymous write | |
US11803329B2 (en) | Methods and systems for processing write requests in a storage system | |
CN104731517A (en) | Method and device for allocating capacity of storage pool | |
CN106960011A (en) | Metadata of distributed type file system management system and method | |
US10057348B2 (en) | Storage fabric address based data block retrieval | |
US9069471B2 (en) | Passing hint of page allocation of thin provisioning with multiple virtual volumes fit to parallel data access | |
US8504764B2 (en) | Method and apparatus to manage object-based tiers | |
US8725979B1 (en) | Efficient methods and systems for allocating storage volumes | |
CN105354294A (en) | Nested file management system and method | |
CN105404471A (en) | Distributed cloud storage cache layer realization method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
C10 | Entry into substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |