CN103235754B - The treating method and apparatus of request in distributed file system - Google Patents
The treating method and apparatus of request in distributed file system Download PDFInfo
- Publication number
- CN103235754B CN103235754B CN201310145208.8A CN201310145208A CN103235754B CN 103235754 B CN103235754 B CN 103235754B CN 201310145208 A CN201310145208 A CN 201310145208A CN 103235754 B CN103235754 B CN 103235754B
- Authority
- CN
- China
- Prior art keywords
- node
- request
- type
- queue
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Multi Processors (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention discloses the treating method and apparatus of request in a kind of distributed file system, the method includes: judge whether request is pending request；In the case of the request that request is pending, request is resolved, obtains the node that this request is comprised；For each node of this request, according to the type of this node, send the node to the node queue corresponding with type.The present invention, by resolving request, makes each node in request send to corresponding node queue according to its type, it can be made to use rapidly and accurately on node, it is ensured that allows the request on different node the most concurrently perform, improves system operating efficiency.
Description
Technical field
The present invention relates to computer realm, in particular it relates to the place of request in a kind of distributed file system
Reason method and apparatus.
Background technology
In existing distributed file system, in order to ensure the data integrity of client, have is distributed
System can be the most real to distributed first by operation log recording daily record dish on meta data server
The memory element write data of file system.Which operation can ensure to greatest extent the request of client with
And the data of user do not lose.
But, if metadata system breaks down after restarting, owing to the writing speed of daily record dish is more than daily record
The speed resolved, may cause partial log (request) needs to be reformed, and therefore can reduce the process effect of system
Rate, increases the burden of system.
At present, the main processing ways for centralized daily record is daily record to be read from daily record dish, then will
Request is re-applied on node to be restored one by one.Although the program is capable of the consistent of data, but needs
Take the longer time, and the bandwidth of system and handling capacity can be made to be affected, extend the available machine time,
It is unfavorable for the work efficiency of raising system equally.
Low to the treatment effeciency of request for system in correlation technique, cause system to operate the most more asking
Topic, the most not yet proposes effective solution.
Summary of the invention
Low to the treatment effeciency of request for system in correlation technique, cause system to operate the most more asking
Topic, the present invention proposes the treating method and apparatus of request in a kind of distributed file system, it is possible in asking
Each node accurately and timely send to corresponding node queue, improve system operating efficiency.
The technical scheme is that and be achieved in that:
According to an aspect of the invention, it is provided the processing method of request in a kind of distributed file system.
In this distributed file system, the processing method of request includes:
Judge whether request is pending request；
In the case of the request that request is pending, request is resolved, obtains what this request was comprised
Node；
For each node of this request, according to the type of this node, send the node to corresponding with type
Node queue.
Wherein, request is carried out parsing to include:
Determine the node in request, and the node in request is classified.
Further, the type of node includes at least one of:
Metadata node, back end.
Further, after sending the node to the node queue corresponding with type, this distributed document
In system, the processing method of request includes:
Concurrently the node in each node queue is processed between multiple node queues.
Additionally, after the node processing that this request is corresponding in node queue completes, this request is deleted.
According to an aspect of the invention, it is provided the processing means of request in a kind of distributed file system.
In this distributed file system, the processing means of request includes:
Judge module, for judging whether ask is pending request；
Parsing module, for, in the case of the request that request is pending, resolving request, obtain
The node that this request is comprised；
Sending module, for each node to this request, according to the type of this node, sends this node
To the node queue corresponding with type.
Wherein, parsing module is for determining the node in request, and classifies the node in request.
Further, the type of node includes at least one of:
Metadata node, back end.
Further, after sending the node to the node queue corresponding with type, this distributed document
In system, the processing means of request also includes:
Processing module, for carrying out the node in each node queue concurrently between multiple node queues
Process.
Additionally, the processing means of request also includes in this distributed file system:
Removing module, after the node processing that this request is corresponding in node queue completes, by this request
Delete.
Pending situation, by screening request, is only resolved by the present invention, and make in request is every
Individual node sends to corresponding node queue according to its type, it is possible to avoid entering the most treated request completed
Row repeats to process, and is effectively increased system operating efficiency, decreases the workload of system.
Accompanying drawing explanation
In order to be illustrated more clearly that the embodiment of the present invention or technical scheme of the prior art, below will be to enforcement
In example, the required accompanying drawing used is briefly described, it should be apparent that, the accompanying drawing in describing below is only
Some embodiments of the present invention, for those of ordinary skill in the art, are not paying creative work
Under premise, it is also possible to obtain other accompanying drawing according to these accompanying drawings.
Fig. 1 is the flow chart of the processing method of request in distributed file system according to embodiments of the present invention；
Fig. 2 is the block diagram of the processing means of request in distributed file system according to embodiments of the present invention；
Fig. 3 is that distributed file system according to embodiments of the present invention processes step interior joint in log services
The schematic diagram of the state in device；
Fig. 4 is that distributed file system according to embodiments of the present invention processes the transmission of difference node in step
Schematic diagram to node queue.
Detailed description of the invention
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clearly
Chu, be fully described by, it is clear that described embodiment be only a part of embodiment of the present invention rather than
Whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art obtained all its
His embodiment, broadly falls into the scope of protection of the invention.
According to embodiments of the invention, it is provided that the processing method of request in a kind of distributed file system.
As it is shown in figure 1, the processing method bag of request in distributed file system according to embodiments of the present invention
Include:
Step S101, it is judged that whether request is pending request；
Step S103, in the case of the request that request is pending, resolves request, is somebody's turn to do
The node that request is comprised；
Step S105, for each node of this request, according to the type of this node, sends this node
To the node queue corresponding with type.
Wherein, when request is resolved, it may be determined that the node in request, and to the node in request
Classify.
Further, the type of node includes at least one of:
Metadata node, back end.
Further, after sending the node to the node queue corresponding with type, this distributed document
In system, the processing method of request includes:
Concurrently the node in each node queue is processed between multiple node queues.Compared to existing
Having in technology when again processing request after the system failure and be merely able to sequentially resolve Operation Log, the present invention adopts
Scheme with Parallel processing demands interior joint, it is to avoid the problem that system can not maximally utilise bandwidth.
Additionally, after the node processing that this request is corresponding in node queue completes, this request is deleted, from
Avoid taking memory space.
Below according to the processing method of request in the distributed file system of the present invention, can use in force will
Daily record performs in the way of concurrently performing.
When meta data server occurs extremely needing to restart, in order to make whole system reach consistent point, it is necessary to
The data that recorded on daily record dish being first applied on corresponding disk, system could normally provide clothes
Business.If all of daily record all recorded on a centrostigma, then can reduce the efficiency of system.In order to anti-
Locking system efficiency reduces, and the present invention uses the distributed execution of daily record of record, it is thus possible to make system to institute
There is the request serial for a certain disk to perform, and for other different nodes, the most concurrently perform request,
The recovery time that can make system shortens.
According to the processing method of request in the distributed file system of the present invention, can implement in actual applications
For the following mode that daily record is concurrently performed.
Hereinafter describe according to processing method daily record is concurrent of request in the distributed file system of the present invention
The mode performed, specifically comprises the following steps that
Step S1, when client sends a request, generates one continuously on meta data server
Tid (that is, transaction number, in specific implementation method, this tid include meta data server sequence number, should
The content destination node number corresponding with this request of request), and record this request and which will be write save
Point (meta data server or data server), then will ask lower wall, i.e. set up daily record (operation day
Will or data logging).
Step S2, after daily record lower wall, resolves request on node, the operation day corresponding by request
Will or data logging are write in corresponding object and (are operated in internal memory), (unit's number in i.e. corresponding node
According to server or data server).After writing internal memory success, return to client, inform this time operation
Success.
Step S3, data timing in internal memory is (according to certain time interval or in real time, by client
Regulation) lower wall, after lower wall, it is notified that this request of log server is complete, the request completed can be in day
Timing on will server (according to certain time interval or in real time, client specify) is deleted.
Step S4, as it is shown on figure 3, when after whole system abnormal restarting, first obtain from log server
Request tid (that is, untreated request, the log services as shown in Figure 3 taken to daily record dish
In device preserve tid5 to tidn) and have been applied to disk request (the most processed request, as scheme
The tid1 to tid4 having application to disk shown in 3), request is read out from log server.
Step S5, analysis request, request is sent to different daily record application according to different running node
In queue.As shown in Figure 4, (i.e. tid5 to tidn in Fig. 4) request that will preserve in log server
It is sent to different daily record application queue (node queue the most herein) according to different running node, as
According to the destination node number that this request in tid is corresponding, tid5 and tid6 is assigned to node 1, tid7 is distributed
To node 2 etc..
Step S6, the request in respective queue is sent to the back end of correspondence by each request application queue
On, then by respective node, market demand is completed.
Step S7, all of request all should be with after completing, and whole system unanimously (i.e. completes to have preserved institute
Have tid daily record), can continue externally to provide business.
According to embodiments of the invention, it is provided that the processing means of request in a kind of distributed file system.
As in figure 2 it is shown, according to the processing means bag of request in the distributed file system of the bright embodiment of this law
Include:
Judge module 21, for judging whether ask is pending request；
Parsing module 22, for, in the case of the request that request is pending, request being resolved,
Obtain the node that this request is comprised；
Sending module 23, for each node to this request, according to the type of this node, by this node
Send to the node queue corresponding with type.
Wherein, parsing module 22 is for determining the node in request, and classifies the node in request.
Further, the type of node includes at least one of:
Metadata node, back end.
Further, after sending the node to the node queue corresponding with type, this distributed document
In system, the processing means of request also includes:
Processing module (not shown), is used between multiple node queues concurrently in each node queue
Node process.
Additionally, the processing means of request also includes in this distributed file system:
Removing module (not shown), after the node processing that this request is corresponding in node queue completes,
This request is deleted.
According to technical scheme, it is proposed that different in the model generation whole system of centralised storage daily record
Restoration methods after often restarting again, the method will be applied to different joints according to request when using log
Classified on Dian.The request read from log server is added directly into different request queues, allows not
With the parallel execution of the request of server, improve efficiency.
In sum, by means of the technique scheme of the present invention, by request is resolved, request is made
In each node send to corresponding node queue according to its type, can make should use node rapidly and accurately
On (meta data server or data server), it is ensured that allow the request on different node the most concurrently perform,
Ensure that the handling capacity of system, shorten the time of startup, and for request correspondence in node queue
Node processing complete after, request is deleted, takies reducing unnecessary memory space, improve and be
System operating efficiency, makes Consumer's Experience become more preferable.
The foregoing is only presently preferred embodiments of the present invention, not in order to limit the present invention, all at this
Within bright spirit and principle, any modification, equivalent substitution and improvement etc. made, should be included in this
Within bright protection domain.
Claims (8)
1. the processing method of request in a distributed file system, it is characterised in that described process side
Method includes:
Judge whether request is pending request；
In the case of the request that described request is pending, described request is resolved, is somebody's turn to do
The node that request is comprised；
For each node of this request, according to the type of this node, send the node to described
The node queue that type is corresponding；
After sending the node to the node queue corresponding with described type, in multiple node queues
Between concurrently the node in each node queue is processed.
Processing method the most according to claim 1, it is characterised in that described request is solved
Analysis includes:
Determine the node in described request, and the node in described request is classified.
Processing method the most according to claim 1, it is characterised in that the type bag of described node
Include at least one of:
Metadata node, back end.
Processing method the most according to claim 1, it is characterised in that should in node queue
After the node processing asking corresponding completes, this request is deleted.
5. the processing means of request in a distributed file system, it is characterised in that described process fills
Put and include:
Judge module, for judging whether ask is pending request；
Parsing module, in the case of the request that described request is pending, to described ask into
Row resolves, and obtains the node that this request is comprised；
Sending module, for each node to this request, according to the type of this node, by this node
Send to the node queue corresponding with described type；
Processing module, is used for after sending the node to the node queue corresponding with described type,
Concurrently the node in each node queue is processed between multiple node queues.
Processing means the most according to claim 5, it is characterised in that described parsing module is used for
Determine the node in described request, and the node in described request is classified.
Processing means the most according to claim 5, it is characterised in that the type bag of described node
Include at least one of:
Metadata node, back end.
Processing means the most according to claim 5, it is characterised in that described processing means is also wrapped
Include:
Removing module, after the node processing that this request is corresponding in node queue completes, should
Request is deleted.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310145208.8A CN103235754B (en) | 2013-04-24 | 2013-04-24 | The treating method and apparatus of request in distributed file system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310145208.8A CN103235754B (en) | 2013-04-24 | 2013-04-24 | The treating method and apparatus of request in distributed file system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103235754A CN103235754A (en) | 2013-08-07 |
| CN103235754B true CN103235754B (en) | 2016-10-05 |
Family
ID=48883799
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310145208.8A Active CN103235754B (en) | 2013-04-24 | 2013-04-24 | The treating method and apparatus of request in distributed file system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103235754B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105843813A (en) * | 2015-01-14 | 2016-08-10 | ***通信集团重庆有限公司 | Method and device for processing big data |
| CN106815251B (en) * | 2015-11-30 | 2019-09-03 | 成都华为技术有限公司 | Distributed data base system, data bank access method and device |
| CN107770205A (en) * | 2016-08-16 | 2018-03-06 | 瓦戈科技（上海）有限公司 | It is remotely located the method for equipment |
| CN106331153B (en) * | 2016-09-20 | 2019-10-25 | 北京华为数字技术有限公司 | A kind of filter method of service request, apparatus and system |
| CN114928574B (en) * | 2022-05-16 | 2023-06-13 | 多点生活(成都)科技有限公司 | Information transmission method, apparatus, electronic device, and computer-readable medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1298536A1 (en) * | 2001-10-01 | 2003-04-02 | Partec AG | Distributed file system and method of operating a distributed file system |
| CN101324892A (en) * | 2007-06-12 | 2008-12-17 | 三星电子株式会社 | System and method of checking integrity of content and metadata |
| CN102025758A (en) * | 2009-09-18 | 2011-04-20 | 成都市华为赛门铁克科技有限公司 | Method, device and system fore recovering data copy in distributed system |
| CN102368210A (en) * | 2011-10-25 | 2012-03-07 | 曙光信息产业(北京)有限公司 | Pipeline based metadata journaling method for distributed system |
| CN102567445A (en) * | 2011-10-25 | 2012-07-11 | 无锡城市云计算中心有限公司 | Method for guaranteeing consistency of metadata in distributed file system |
| CN102662795A (en) * | 2012-03-20 | 2012-09-12 | 浪潮电子信息产业股份有限公司 | Metadata fault-tolerant recovery method in distributed storage system |
| CN102819570A (en) * | 2012-07-19 | 2012-12-12 | 华为技术有限公司 | Data access method, device and system |
| CN103051681A (en) * | 2012-12-06 | 2013-04-17 | 华中科技大学 | Collaborative type log system facing to distribution-type file system |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102236851B (en) * | 2010-04-21 | 2016-03-09 | 百度在线网络技术（北京）有限公司 | The method and system that the multidimensional credit system composing power based on user calculates in real time |
| CN102419766B (en) * | 2011-11-01 | 2013-11-20 | 西安电子科技大学 | Data redundancy and file operation methods based on Hadoop distributed file system (HDFS) |
| CN103150394B (en) * | 2013-03-25 | 2014-07-23 | 中国人民解放军国防科学技术大学 | Distributed file system metadata management method facing to high-performance calculation |
-
2013
- 2013-04-24 CN CN201310145208.8A patent/CN103235754B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1298536A1 (en) * | 2001-10-01 | 2003-04-02 | Partec AG | Distributed file system and method of operating a distributed file system |
| CN101324892A (en) * | 2007-06-12 | 2008-12-17 | 三星电子株式会社 | System and method of checking integrity of content and metadata |
| CN102025758A (en) * | 2009-09-18 | 2011-04-20 | 成都市华为赛门铁克科技有限公司 | Method, device and system fore recovering data copy in distributed system |
| CN102368210A (en) * | 2011-10-25 | 2012-03-07 | 曙光信息产业(北京)有限公司 | Pipeline based metadata journaling method for distributed system |
| CN102567445A (en) * | 2011-10-25 | 2012-07-11 | 无锡城市云计算中心有限公司 | Method for guaranteeing consistency of metadata in distributed file system |
| CN102662795A (en) * | 2012-03-20 | 2012-09-12 | 浪潮电子信息产业股份有限公司 | Metadata fault-tolerant recovery method in distributed storage system |
| CN102819570A (en) * | 2012-07-19 | 2012-12-12 | 华为技术有限公司 | Data access method, device and system |
| CN103051681A (en) * | 2012-12-06 | 2013-04-17 | 华中科技大学 | Collaborative type log system facing to distribution-type file system |
Non-Patent Citations (1)
| Title |
|---|
| 分布式并行文件***的日志管理;吴华;《中国优秀硕士学位论文全文数据库 信息科技辑》;20040315(第1期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103235754A (en) | 2013-08-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103235754B (en) | The treating method and apparatus of request in distributed file system | |
| US10261853B1 (en) | Dynamic replication error retry and recovery | |
| US20170116084A1 (en) | Method and System for Monitoring Virtual Machine Cluster | |
| US10467048B2 (en) | Techniques for virtual machine migration | |
| US8381016B2 (en) | Fault tolerance for map/reduce computing | |
| CN107870845A (en) | Towards the management method and system of micro services framework applications | |
| CN105635252B (en) | Hadoop distributed file system HDFS erasure code redundancy backup method | |
| CN103310460A (en) | Image characteristic extraction method and system | |
| CN104077328B (en) | The operation diagnostic method and equipment of MapReduce distributed system | |
| CN107153643B (en) | Data table connection method and device | |
| CN109033814A (en) | intelligent contract triggering method, device, equipment and storage medium | |
| CN103152391A (en) | Journal output method and device | |
| CN115004156A (en) | Real-time multi-tenant workload tracking and automatic throttling | |
| CN105302641A (en) | Node scheduling method and apparatus in virtual cluster | |
| CN110908837B (en) | Application program exception handling method and device, electronic equipment and storage medium | |
| CN111198754A (en) | Task scheduling method and device | |
| CN105049524B (en) | A method of the large-scale dataset based on HDFS loads | |
| CN105069029B (en) | A kind of real-time ETL system and method | |
| CN105068875A (en) | Intelligence data processing method and apparatus | |
| US10789087B2 (en) | Insight usage across computing nodes running containerized analytics | |
| CN105574008A (en) | Task scheduling method and equipment applied to distributed file system | |
| CN106649000A (en) | Fault recovery method for real-time processing engine, and corresponding server | |
| WO2023165512A1 (en) | Fault file storage method and related apparatus | |
| CN108089930A (en) | The sending method and device of data | |
| US9430338B2 (en) | Method and computing device for recording log entries |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220722 Address after: 100089 building 36, courtyard 8, Dongbeiwang West Road, Haidian District, Beijing Patentee after: Dawning Information Industry (Beijing) Co.,Ltd. Patentee after: DAWNING INFORMATION INDUSTRY Co.,Ltd. Address before: 100193 No. 36 Building, No. 8 Hospital, Wangxi Road, Haidian District, Beijing Patentee before: Dawning Information Industry (Beijing) Co.,Ltd. |