CN104571952B - A kind of method for separating processing data and metadata read-write requests - Google Patents
A kind of method for separating processing data and metadata read-write requests Download PDFInfo
- Publication number
- CN104571952B CN104571952B CN201410827456.5A CN201410827456A CN104571952B CN 104571952 B CN104571952 B CN 104571952B CN 201410827456 A CN201410827456 A CN 201410827456A CN 104571952 B CN104571952 B CN 104571952B
- Authority
- CN
- China
- Prior art keywords
- request
- metadata
- data
- queue
- requests
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
- G06F3/0631—Configuration or reconfiguration of storage systems by allocating resources to storage systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of method for separating processing data and metadata request.Including：The metadata for increasing request is semantic；Request of data queue and metadata request queue are set for each scsi device, the latter is specifically used to handle metadata request, and uses different processing modes to different requests.According to the characteristics of metadata request granularity is small, real-time, distribution is scattered, metadata request queue is asked without I/O dispatch layers by the way of preferential answering processing, is postponed not in request queue, is directly issued to disc driver；Big to request granularity, frequency is relatively low, is distributed the request of data concentrated, and using I/O dispatch layers are passed through, the mode for being issued to disc driver after stream a period of time, aerial drainage again is stored in request queue.The bottleneck problem of single request queue is alleviated, the concurrency of different levels request processing is enhanced, the needs that are particularly suitable for use in are frequent, substantial amounts of storage small documents, the business high to critical data requirement of real-time.
Description
Technical field
The invention belongs to computer memory technical field, read more particularly, to a kind of processing data that separates with metadata
The method of write request.
Background technology
Data society needs to access daily a large amount of data preserved within the storage system, but many times user and need not
The data preserved in access system in itself, and only need to the metadata in access system.In addition, being preserved in access system
Data before also must first access metadata, according to information inquiries such as filenames and obtain size of data, save location, access right
After the metadata informations such as limit, the data deposited in a device could be read.The data access of system has more than half to be all and first number
According to correlation.And be that granularity is small, frequency is high, it is more scattered etc. to be distributed the characteristics of metadata I/O.Therefore to the access process of metadata
There are enough attention and different processing.
The storage area network (Storage Area Network, SAN) especially now set up on ip networks has become
The storage solution of many enterprises, the storage device of target end system is mounted in initiation end system by IP agreement, made
Conducted interviews for a block device.But initiate the storage device only one of which request team of the target end system of end system carry
Row, while I/O requests at present are all without semanteme, all I/O requests all mutually exclusive enter request queue, complete I/O conjunction
And dispatch etc. processing.This mode exists following not enough：1) for storage environment heavy I/O, single queue can turn into request
The bottleneck of processing, it is impossible to fully call the service ability of actual multiple equipment；2) all I/O requests are without semanteme, it is impossible to reach
The metadata high to requirement of real-time is distinguished and treated.Especially can be tight in needing frequently, largely storing this kind of application of small documents
Ghost image rings the access performance of data.
The content of the invention
For the disadvantages described above or Improvement requirement of prior art, processing data and metadata are separated the invention provides a kind of
The method of read-write requests, realizes and does different processing to the request that importance is different, requirement of real-time is different, alleviate single
The bottleneck problem of request queue, enhances the concurrency of different levels request processing, and being particularly suitable for use in, needs are frequent, substantial amounts of to be deposited
Store up small documents, the business high to critical data requirement of real-time.
To achieve the above object, the invention provides a kind of method for separating processing data and metadata read-write requests, its
It is characterised by, comprises the following steps：(1) metadata request queue is created in target storage device, makes target storage device same
The queue of Shi Hanyou metadata requests and data request queue；(2) semanteme of increase I/O requests, distinguishes request of data and metadata
Request；(3) judge whether the request received is metadata request, is to be sent to metadata request queue, and order is performed
Step (4)；Otherwise it is request of data to illustrate the request received, and the request is sent into request of data queue, step (5) is skipped to；
(4) preferential answering processing mode is used to metadata request, metadata request is not entered I/O dispatch layers, please not in metadata
Ask and postpone to wait in queue, be directly issued to disc driver, be then back to step (3)；(5) request of data is dispatched into I/O
Layer makes requests on scheduling, postpones to wait in request of data queue, and expectation merges with subsequent data request；(6) request of data
After queue aerial drainage, request of data is issued to disc driver, return to step (3).
Preferably, the step (1) is specially：First by sysfs file system, set to each storage of kernel objects
Remarks volume is used for the interface for sending establishment/revocation metadata request queue command；Sent again to target storage device and create first number
According to request queue order, the establishment of metadata request queue is completed.
Preferably, the step (2) is specially：The buffer data structure buffer_head of file system layer is changed,
Increase represents the semantic position meta of metadata in buffer_head mode field state, and increases meta set and reset
Handling function；The request function of file system layer is changed, buffering area corresponding to all metadata requests adds meta mark,
Generate with bio requests semantic meta.
Preferably, in the step (4), when a upper metadata request is not returned also, it is temporarily stored in metadata request team
Metadata request in row does adjacent I/O merging.
In general, by the contemplated above technical scheme of the present invention compared with prior art, with following beneficial effect
Really：
1st, compared to existing without semantic I/O requests, the present invention adds semanteme for I/O requests, has distinguished request type, be
It is subsequently any to want to do different processing there is provided possible to different types of request.
2nd, compared to existing each small computer system interface (Small Computer System Interface,
SCSI) storage device only one of which request queue, the present invention is provided with request of data queue and metadata to SCSI storage device
Request queue, is respectively intended to processing data request and metadata request, alleviates the bottleneck problem of single request queue, enhance
The concurrency of different levels request processing, for example, I/O dispatch layers may do scheduling union operation, SCSI plans for request of data
Slightly handling routine is doing the operation of construction scsi command for metadata request.
3 ask feature according to different I/O, and different processing modes is used to different request queues.According to metadata
Ask granularity small, frequency is high, the characteristics of distribution relatively disperses, please to metadata request queue by the way of preferential answering processing
Ask without I/O dispatch layers, postpone not in request queue, but be directly issued to disc driver；It is big to request granularity, frequency
Degree is relatively low, is distributed the request of data more concentrated, and using I/O dispatch layers are passed through, postpones a period of time, phase in request queue
Treat to merge with subsequent request, be issued to the mode of disc driver after aerial drainage again.Realize, requirement of real-time different to importance
Different processing are done in different requests.
Brief description of the drawings
Fig. 1 is method flow diagram of the separated processing data with metadata read-write requests of the embodiment of the present invention；
Fig. 2 is the typical application scenarios frame of the separated processing data and the method for metadata read-write requests of the embodiment of the present invention
Composition.
Embodiment
In order to make the purpose , technical scheme and advantage of the present invention be clearer, it is right below in conjunction with drawings and Examples
The present invention is further elaborated.It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, and
It is not used in the restriction present invention.As long as in addition, technical characteristic involved in each embodiment of invention described below
Not constituting conflict each other can just be mutually combined.
As shown in figure 1, the method for the separated processing data of the embodiment of the present invention and metadata read-write requests includes following step
Suddenly：
(1) metadata request queue is created in target storage device, makes target storage device simultaneously please containing metadata
Ask queue and data request queue.
Set request of data queue with metadata request queue to distinguish processing data request and member for target storage device
Request of data, directly regard the existing request queue of each storage device in system as request of data queue.
The specific establishment process of metadata request queue is：By sysfs file system, to each storage of kernel objects
Facility registration is used for the interface for sending establishment/revocation metadata request queue command；Sent to target storage device and create first number
According to request queue order, the establishment of metadata request queue is completed.
Equally, when we no longer need to use metadata request queue, we can be by sending revocation metadata
Metadata request queue resource is reclaimed in request queue order.
(2) semanteme of increase I/O requests, distinguishes request of data and metadata request.
Specifically, the buffer data structure buffer_head of file system layer is changed, in buffer_head state
Increase represents the semantic position meta of metadata in field state, and increases meta set with resetting handling function；Change file
The request function of system layer, buffering area corresponding to all metadata requests adds meta mark, generates with meta semantemes
Bio is asked, for it is follow-up it is any want to do different processing to different types of request provide possibility.
(3) judge whether the request received is metadata request, is to be sent to metadata request in generic block layer
Queue, order performs step (4)；Otherwise it is request of data to illustrate the request received, and the request is sent into request of data queue
(i.e. system originally requested content queue), skips to step (5).
(4) preferential answering processing mode is used to metadata request, metadata request is not entered I/O dispatch layers, do not exist
Postpone to wait in metadata request queue, be directly issued to disc driver, be then back to step (3).
Wherein, when a upper metadata request is not returned also, it is temporarily stored in the metadata request in metadata request queue
Do adjacent I/O merging.
(5) request of data makes requests on scheduling into I/O dispatch layers, exists according to stream/aerial drainage (plug/unplug) mechanism of storage
Postpone to wait in request of data queue, expectation merges with subsequent data request.
(6) after request of data queue aerial drainage, request of data is issued to disc driver, return to step (3).
To make those skilled in the art more fully understand the present invention, with reference to separated place of the specific embodiment to the present invention
The method of reason data and metadata read-write requests is described in detail.
As shown in Fig. 2 the SAN now set up on ip networks has become the storage solution of many enterprises, by mesh
The storage device for marking end system is mounted in initiation end system by IP agreement, is conducted interviews as a block device, originating end
As access local device access target end carry originating end equipment, as shown in Fig. 2 101.Set up for this equipment
Two queues, metadata request queue and request of data queue, wherein, directly ask each storage device in system is existing
Ask queue as request of data queue, request of data queue can be scheduled by I/O dispatch layers, store stream/aerial drainage；And metadata
Request queue does not enter I/O scheduling then, and stream/aerial drainage is not stored, but directly issues processing and arrive scsi device driver, certainly, temporarily
There is the request for having not enough time to handle in metadata request queue can do simplest adjacent merging.Request is driven by ISCSI
Destination end is sent to by network after dynamic encapsulation.And target end data can be stored separately in not with metadata in many cases now
With in equipment, as shown in Fig. 2 102.The two classes request that so originating end is sended over can be directed to two groups of different equipment
In.On the one hand so do will not only destroy an original queue scheduling union operation in originating end to the classification of request, simultaneously
Also the service ability of the multiple different physical devices of destination end has been given full play to.
As it will be easily appreciated by one skilled in the art that the foregoing is merely illustrative of the preferred embodiments of the present invention, it is not used to
The limitation present invention, any modifications, equivalent substitutions and improvements made within the spirit and principles of the invention etc., it all should include
Within protection scope of the present invention.
Claims (4)
1. a kind of method for separating processing data and metadata read-write requests, it is characterised in that comprise the following steps：
(1) metadata request queue is created for target storage device, makes target storage device while possessing metadata request queue
With data request queue；
(2) semanteme of increase I/O requests, distinguishes request of data and metadata request；
(3) judge whether the request received is metadata request, is to be sent to metadata request queue, and order performs step
Suddenly (4)；Otherwise it is request of data to illustrate the request received, and the request is sent into request of data queue, step (5) is skipped to；
(4) preferential answering processing mode is used to metadata request, metadata request is not entered I/O dispatch layers, not in first number
According to postponing to wait in request queue, disc driver is directly issued to, step (3) is then back to；
(5) request of data makes requests on scheduling into I/O dispatch layers, postpones to wait in request of data queue, expects and follow-up
The merging of request of data；
(6) after request of data queue aerial drainage, request of data is issued to disc driver, return to step (3).
2. the method for processing data and metadata read-write requests is separated as claimed in claim 1, it is characterised in that the step
(1) it is specially：First by sysfs file system, registered to each storage device of kernel objects for sending establishment/revocation
The interface of metadata request queue command；Sent again to target storage device and create metadata request queue command, complete first number
According to the establishment of request queue.
3. the method for processing data and metadata read-write requests is separated as claimed in claim 1 or 2, it is characterised in that described
Step (2) is specially：The buffer data structure buffer_head of file system layer is changed, in buffer_head status word
Increase represents the semantic position meta of metadata in section state, and increases meta set with resetting handling function；Change file system
The request function of system layer, buffering area corresponding to all metadata requests adds meta mark, generates with bio semantic meta
Request.
4. the method for processing data and metadata read-write requests is separated as claimed in claim 1 or 2, it is characterised in that described
In step (4), when a upper metadata request is not returned also, the metadata request being temporarily stored in metadata request queue does phase
Adjacent I/O merging.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410827456.5A CN104571952B (en) | 2014-12-25 | 2014-12-25 | A kind of method for separating processing data and metadata read-write requests |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410827456.5A CN104571952B (en) | 2014-12-25 | 2014-12-25 | A kind of method for separating processing data and metadata read-write requests |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104571952A CN104571952A (en) | 2015-04-29 |
| CN104571952B true CN104571952B (en) | 2017-08-01 |
Family
ID=53088137
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410827456.5A Active CN104571952B (en) | 2014-12-25 | 2014-12-25 | A kind of method for separating processing data and metadata read-write requests |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104571952B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3293625B1 (en) * | 2015-05-27 | 2021-07-07 | Honor Device Co., Ltd. | Method and device for accessing file, and storage system |
| CN109445685B (en) * | 2018-09-26 | 2020-06-30 | 华中科技大学 | User mode file system processing method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101247417A (en) * | 2008-03-07 | 2008-08-20 | 中国科学院计算技术研究所 | Double-layer metadata processing system and method |
| CN101788961A (en) * | 2009-01-23 | 2010-07-28 | 英业达集团(天津)电子技术有限公司 | Data access method for carrying out asynchronous request on block equipment |
| CN102073605A (en) * | 2010-12-27 | 2011-05-25 | 深圳市创新科信息技术有限公司 | Method for storage interface bypassing Bio layer to access disk drive |
| CN102164161A (en) * | 2011-01-10 | 2011-08-24 | 清华大学 | Method and device for performing file layout extraction on parallel file system |
| CN102523258A (en) * | 2011-11-30 | 2012-06-27 | 广东电子工业研究院有限公司 | Data storage framework facing cloud operation system and load balancing method thereof |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8635415B2 (en) * | 2009-09-30 | 2014-01-21 | Intel Corporation | Managing and implementing metadata in central processing unit using register extensions |
-
2014
- 2014-12-25 CN CN201410827456.5A patent/CN104571952B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101247417A (en) * | 2008-03-07 | 2008-08-20 | 中国科学院计算技术研究所 | Double-layer metadata processing system and method |
| CN101788961A (en) * | 2009-01-23 | 2010-07-28 | 英业达集团(天津)电子技术有限公司 | Data access method for carrying out asynchronous request on block equipment |
| CN102073605A (en) * | 2010-12-27 | 2011-05-25 | 深圳市创新科信息技术有限公司 | Method for storage interface bypassing Bio layer to access disk drive |
| CN102164161A (en) * | 2011-01-10 | 2011-08-24 | 清华大学 | Method and device for performing file layout extraction on parallel file system |
| CN102523258A (en) * | 2011-11-30 | 2012-06-27 | 广东电子工业研究院有限公司 | Data storage framework facing cloud operation system and load balancing method thereof |
Non-Patent Citations (2)
| Title |
|---|
| 基于层次结构的元数据动态管理方法的研究;刘群 等;《计算机研究与发展》;20100505;第46卷(第z2期);第78-82页 * |
| 高可靠性元数据服务器研究;刘群 等;《计算机工程》;20081215;第34卷(第17期);第88-90页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104571952A (en) | 2015-04-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20200409798A1 (en) | Tiered cloud storage for different availability and performance requirements | |
| CN106603598B (en) | Method and device for processing service request | |
| US9621688B2 (en) | Custom error page enabled via networked computing service | |
| US10541936B1 (en) | Method and system for distributed analysis | |
| US10498681B1 (en) | Storage management for ephemeral messages | |
| US9836516B2 (en) | Parallel scanners for log based replication | |
| US20150379026A1 (en) | Content fabric for a distributed file system | |
| US20120221744A1 (en) | Migrating Virtual Machines with Adaptive Compression | |
| CN104932932A (en) | Asynchronous business processing method, device and system | |
| CN110383764A (en) | The system and method for usage history data processing event in serverless backup system | |
| US10142195B1 (en) | Partitioned performance tracking core resource consumption independently | |
| CN106534308B (en) | Method and device for solving data block access hot spot in distributed storage system | |
| CN103312624A (en) | Message queue service system and method | |
| CN104571952B (en) | A kind of method for separating processing data and metadata read-write requests | |
| CN110168513B (en) | Partial storage of large files in different storage systems | |
| US9247033B2 (en) | Accessing payload portions of client requests from client memory storage hardware using remote direct memory access | |
| CN109478126A (en) | The I/O starvation in external management calculated in equipment avoids | |
| CN107229424B (en) | Data writing method for distributed storage system and distributed storage system | |
| US20180349038A1 (en) | Method of Reordering a Queue of Write Requests | |
| US9659041B2 (en) | Model for capturing audit trail data with reduced probability of loss of critical data | |
| Hsu et al. | Effective memory reusability based on user distributions in a cloud architecture to support manufacturing ubiquitous computing | |
| WO2015058628A1 (en) | File access method and device | |
| US20140280347A1 (en) | Managing Digital Files with Shared Locks | |
| US10148588B1 (en) | Partitioned performance: using resource account aggregates to throttle at the granular level | |
| US11994956B2 (en) | Adaptive throttling of metadata requests |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |