CN109815204A - A kind of metadata request distribution method and equipment based on congestion aware - Google Patents
A kind of metadata request distribution method and equipment based on congestion aware Download PDFInfo
- Publication number
- CN109815204A CN109815204A CN201811505304.8A CN201811505304A CN109815204A CN 109815204 A CN109815204 A CN 109815204A CN 201811505304 A CN201811505304 A CN 201811505304A CN 109815204 A CN109815204 A CN 109815204A
- Authority
- CN
- China
- Prior art keywords
- server
- metadata request
- transmission rate
- sent
- destination server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Abstract
The embodiment of the present invention provides a kind of metadata request distribution method and equipment based on congestion aware, it include: to receive metadata request and all server state informations for being sent to server, according to server state information, Performance Evaluation is carried out to all servers that are sent to by preset cost estimation function, obtains all performance costs for being sent to each server in server；The minimum server of performance cost is chosen as destination server, judgement knows that the number of requests that destination server is sent in actual time window is less than preset threshold, sends metadata request to destination server；Client equivalence determines the congestion state of destination server, and according to the congestion state of destination server, adjusts the transmission rate of metadata request.Method provided by the invention improves the processing speed of metadata request and the throughput of system entirety, and by the congestion condition of aware services device, adjustment request transmission rate saves system maintenance cost and time overhead.
Description
Technical field
The present embodiments relate to video technique field more particularly to a kind of metadata request distributions based on congestion aware
Method and apparatus.
Background technique
In recent years, with the arrival of big data era, it is common that storage and the utilization of data have been increasingly becoming many industries
The topic of concern.In order to the data for preferably these being utilized uninterruptedly to generate in real time, distributed field system is often required to use
It unites and Organization And Management is carried out to the data on many multiple storage devices.All due to request most of in distributed file system
It is the request to metadata, therefore most of file system often use science metadata server cluster to mention for client
For Metadata Service with load sharing, while being also avoided that single node failure.But since distributed file system is to client
It is not instructed heuristically and is constrained, client is in selection target server often because the reasons such as Herd Behavior are made
It is shaken at the congestion and performance of server end, causes the performance of metadata server cluster to decline, even result in part server
Delay machine.
In the prior art, many solutions are all to go to optimize in server end trial and avoid part server negative
Carry excessive problem.For example, Ceph distributed file system does not use science metadata server cluster to mention for client
For Metadata Service, the entire NameSpace of file system is divided into several height using the method that dynamic subtree divides and is ordered
The name space, and each server is allowed to undertake the metadata storage of a part of NameSpace respectively and service.Such method is no doubt
Can in server end dynamically equally loaded, but it is corresponding and come be metadata migration process bring performance decline with it is each
Kind expense.Simultaneously because no longer there is redundancy, so there are the risk of single node failure, and metadata is also required to write back in real time firmly
Disk, maintenance cost are got higher.Another method is the reception upper limit in server end setting metadata request, once load reaches
This upper limit just no longer receives new metadata request.Such methods are the problem is that the load when part server reaches
After limit, the metadata request for being sent to these servers can all lose, and client needs again after one section of longer waiting time-out
The request of these loss is sent, and all requests for being related to same metadata require to be blocked during this, finally make client
The experience at end is deteriorated, while also bringing many problems to the maintenance of consistency.Synchronization, often there are also at many servers
In the state gently loaded, only client goes to perceive these information there is no the algorithm using enlightenment, then causes service
Device end loads unbalanced.
In the prior art, metadata distribution optimization often is carried out from server side de, avoids part server load excessive,
However only there is significant limitation to server optimization, under efficiency is relatively low, while certain performance is also lost, has increased
Additional expense, can not go from the root the fluctuation of load of settlement server end, performance decline the problem of.
Summary of the invention
In view of the deficiencies of the prior art, the embodiment of the present invention provides a kind of metadata request distributor based on congestion aware
Method and equipment, this method are run in client-side, are capable of the congestion situation at aware services device end, and adaptive customer end itself
The metadata request transmission rate upper limit is collapsed, together with preventing destination server from receiving number of requests that is excessive and can not handling
When be also prevented from client and immediately block all subsequent requests after perceiving congestion, the fluctuation of load for eliminating server end is existing
As improving the stability and throughput of system entirety.
In a first aspect, the embodiment of the present invention provides a kind of metadata request distribution method based on congestion aware, comprising:
Metadata request is received to be sent to by preset cost estimation function to all according to server state information
Server carries out Performance Evaluation, obtains all performance costs for being sent to each server in server；
The minimum server of the performance cost is chosen as destination server, judgement, which is known in actual time window, to be sent to
The number of requests of destination server is less than preset threshold, and Xiang Suoshu destination server sends metadata request；
The size relation of speed is received according to the metadata request transmission rate upper limit of the destination server and response, with
And last transmission rate upper limit rise time receives feedback information elapsed time to this, equivalence determines the target clothes
The jam state information of business device adjusts the transmission of the metadata request according to the jam state information of the destination server
Rate；
Wherein, described to be sent to server specifically: the science metadata server cluster in distributed file system
In all meta data servers that can be accessed.
Wherein, described that Performance Evaluation is carried out to Servers-all by preset cost estimation function, obtain all services
In device the step of the performance cost of each server, specifically include: the history of server can be sent to described in each by, which counting, asks
The exponentially weighted moving average for the response delay asked and the waiting list and processing speed that server can be sent to described in each
The exponentially weighted moving average of rate estimates the performance cost that server can be sent to described in each.
Wherein, after the step of minimum server of the selection performance cost is as destination server, further includes:
If judgement knows that being sent to destination server number of requests in actual time window reaches preset threshold, reselects unselected
That crosses is sent to the minimum server of cost in server as new destination server.
Wherein, the method also includes: if judgement know the request for being sent to server all in actual time window
Quantity reaches preset threshold, then blocks the metadata request, handles the metadata request again in future time window.Its
In, the congestion state according to the destination server, the step of adjusting the transmission rate of the metadata request, specifically wraps
Include: judgement knows that the metadata request transmission rate upper limit is less than response receiving velocity, then hair is improved referring to propagation process three times
Toward the metadata request transmission rate upper limit of the destination server.
Wherein, the step of transmission rate of the adjustment metadata request further include:
Judgement knows that the metadata request transmission rate upper limit is greater than response receiving velocity, while from the last time transmission rate upper limit
Growth event to the time of passing through for receiving feedback this moment is greater than a preset time, then lowers the metadata request and send speed
Rate.
Wherein, propagation process improves the metadata request transmission rate for being sent to the destination server three times for the reference
The step of upper limit, specifically includes: respectively in convex function region, plateau and concave function region, according to the metadata request
The transmission rate upper limit and the destination server enter the peak rate of saturation state, promote the transmission speed of the metadata request
The rate upper limit.Wherein, the method also includes: according to the feedback information of the destination server, in the cost estimation function
Required parameter is updated.
Second aspect, the embodiment of the present invention provides a kind of electronic equipment, including memory, processor and is stored in memory
Computer program that is upper and can running on a processor, the processor are realized when executing described program such as above-mentioned first aspect institute
The step of metadata request distribution method based on congestion aware provided.
The third aspect, the embodiment of the present invention provide a kind of non-transient computer readable storage medium, are stored thereon with calculating
Machine program realizes first number as provided by above-mentioned first aspect based on congestion aware when the computer program is executed by processor
The step of according to request distribution method.
Metadata request distribution method and equipment provided in an embodiment of the present invention based on congestion aware, is estimated using cost
Function carrys out selection target server, calculate certain server cost it is lower, then it is assumed that the request currently reached is sent to the service
Device can be responded faster, improve the processing speed of metadata request and the throughput of system entirety, while passing through perception
The congestion condition of server, adjustment request transmission rate prevent the fluctuation of load and collapse of server end, dramatically save and are
The maintenance cost and time overhead of system administrative staff, also improves the throughput of cluster entirety, while client also obtains more
Good experience.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is this hair
Bright some embodiments for those of ordinary skill in the art without creative efforts, can be with root
Other attached drawings are obtained according to these attached drawings.
Fig. 1 is the process signal for the metadata request distribution method based on congestion aware that one embodiment of the invention provides
Figure；
Fig. 2 is destination service in the metadata request distribution method based on congestion aware that one embodiment of the invention provides
Device selects flow diagram；
Fig. 3 is metadata in the metadata request distribution method based on congestion aware that further embodiment of this invention provides
Transmission rate is requested to adjust flow diagram；
Fig. 4 is the structural schematic diagram for the electronic equipment that one embodiment of the invention provides.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with the embodiment of the present invention
In attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is
A part of the embodiment of the present invention, instead of all the embodiments.Based on the embodiments of the present invention, those of ordinary skill in the art
Every other embodiment obtained without creative efforts, shall fall within the protection scope of the present invention.
With reference to Fig. 1, Fig. 1 is the stream for the metadata request distribution method based on congestion aware that one embodiment of the invention provides
Journey schematic diagram, provided method include:
S1 receives metadata request and all server state informations for being sent to server, according to the server
Status information carries out Performance Evaluation to all servers that are sent to by preset cost estimation function, obtains all be sent to
The performance cost of each server in server.
S2 chooses the minimum server of the performance cost as destination server, and judgement is known in actual time window
The number of requests for being sent to destination server is less than preset threshold, and Xiang Suoshu destination server sends metadata request.
S3 is closed according to the size that the metadata request transmission rate upper limit of the destination server and response receive speed
System, and last transmission rate upper limit rise time receive feedback information elapsed time to this, described in judgement of equal value
The jam state information of destination server adjusts the metadata request according to the jam state information of the destination server
Transmission rate.
Wherein, described to be sent to server specifically: the science metadata server cluster in distributed file system
In all meta data servers that can be accessed.
Specifically, client is by cost estimation function from all up to service when reaching a metadata request
Destination server of the server for selecting cost minimum in device as the request, wherein can be sent to server is distributed document
All meta data servers that can be accessed, every server all have in science metadata server cluster in system
Portion includes the duplication of all metadata of entire file system, and can externally provide metadata request service, and by consistent
Property agreement maintenance metadata consistency.
Client judges whether the number of requests that destination server is sent in actual time window reaches the set upper limit and come really
Recognize whether the request can send, if the number of requests for being sent to destination server in actual time window is not up to limited
System, then client sends the metadata request
After client sends metadata request, client can receive the response and feedback that destination server is sent back to, visitor
Family end according to the size relation of both its metadata request transmission rate upper limit between the server and response receiving velocity with
And increase event to the process time for receiving feedback this moment from the last time transmission rate upper limit, carry out the judgement destination server of equal value
Congestion condition, if destination server congestion, client lowers the metadata request transmission rate upper limit for being sent to the server
To alleviate the packing pressure of the server；If congestion and destination server performance do not have using space, client is referring to one
A propagation process three times is sent to the request transmission rate upper limit of the server to improve, to realize higher throughput.
By the method, using cost estimation function come selection target server, calculate certain server cost it is lower,
Then think that the request currently reached is sent to the server and can faster be responded, improve metadata request processing speed and
The throughput of system entirety, while passing through the congestion condition of aware services device, adjustment request transmission rate prevents server end
The fluctuation of load and collapse dramatically save the maintenance cost and time overhead of system manager, also improve cluster entirety
Throughput, while client also obtains better experience.
On the basis of the above embodiments, described that Servers-all progress performance is commented by preset cost estimation function
The step of estimating, obtaining the performance cost of each server in Servers-all, specifically includes: counting can be sent to described in each
The historical requests of server response delay exponentially weighted moving average and can be sent to described in each server etc.
To the exponentially weighted moving average of queue and processing speed, the performance cost that server can be sent to described in each is estimated.
Specifically, can be sent to server be distributed file system in science metadata server cluster in it is all can
With accessed meta data server, every server all has a duplication comprising all metadata of entire file system,
And metadata request service can be externally provided, and by the consistency of consistency protocol maintenance metadata；On the other hand, cost
Estimation function is sent to the exponentially weighted moving average of the response delay of the historical requests of each server and each by statistics
The waiting list of a server and the exponentially weighted moving average of processing speed carry out the performance generation of each server of Synthesize estimation
Valence, if cost estimation function calculate certain server cost it is lower, then it is assumed that the request currently reached is sent to the server can be with
Faster responded.
On the basis of the above embodiments, the minimum server of the performance cost of choosing is as destination server
After step, further includes: if judgement knows that being sent to destination server number of requests in actual time window reaches preset threshold,
Reselect unselected mistake is sent to the minimum server of cost in server as new destination server.
If judgement knows that the number of requests for being sent to server all in actual time window reaches preset threshold, hinder
The metadata request is filled in, handles the metadata request again in future time window.
Specifically, refering to what is shown in Fig. 2, Fig. 2 is the metadata request based on congestion aware that one embodiment of the invention provides
In distribution method, destination server selects flow diagram, and client judges to be sent to destination server in actual time window
Whether number of requests reaches the set upper limit to confirm whether the request can send, if being sent to target clothes in actual time window
The number of requests of business device is not up to institute's restriction.In this step, the number of requests of destination server is sent in unit time window
The upper limit is equivalent to be sent to the metadata request transmission rate upper limit of destination server in definition, and numerical values recited is to receive
It is true that transmission rate is requested to can only be less than or be equal to what is regulated and controled when response and feedback by congestion aware and control algolithm
The upper limit；And it is the number of responses from destination server received in unit time window that response, which receives rate then, is real
The data that Shi Tongji is obtained；The order of magnitude of time window size only need it is sufficiently small, it is ensured that metadata request send feelings
The monitoring of condition is effective in real time with regulation.The order of magnitude needs of the time window size reach a millisecond rank, to guarantee to first number
The monitoring of situation is sent according to request and is regulated and controled effective in real time.Client judge in the time window Servers-all whether all by
It selected and quantity forwarded is requested to reach the upper limit, and blocked the metadata request if it is client until next time window arrives
Come.
Method provided by the invention using cost estimation function come selection target server, calculate certain server cost
It is lower, then it is assumed that the request currently reached is sent to the server and can faster be responded, and improves the processing of metadata request
The throughput of speed and system entirety；As for the relevant parameter renewal process of cost estimation function, it is only necessary to according to feedback information
Update exponentially weighted moving average and the service that the response for the historical requests that the client is sent to respective server is delayed
The waiting list of device and the exponentially weighted moving average of processing speed are simple and efficient.On the basis of above-described embodiment
On, the congestion state according to the destination server, the step of adjusting the transmission rate of the metadata request, specifically wraps
Include: judgement knows that the metadata request transmission rate upper limit is less than response receiving velocity, then hair is improved referring to propagation process three times
Toward the metadata request transmission rate upper limit of the destination server.
The step of transmission rate of the adjustment metadata request further include: judgement knows that metadata request sends speed
The rate upper limit is greater than response receiving velocity, while increasing event to the process time for receiving feedback this moment from the last time transmission rate upper limit
Greater than one preset time, then lower the metadata request transmission rate.
Propagation process improves the metadata request transmission rate upper limit for being sent to the destination server three times for the reference
The step of specifically include: respectively in convex function region, plateau and concave function region, according to the transmission of the metadata request
Rate-limit and the destination server enter the peak rate of saturation state, in the transmission rate for promoting the metadata request
Limit.
Specifically, if the metadata request transmission rate upper limit between client and the server is less than response receiving velocity,
Then think destination server not congestion, client referring to one three times propagation process to improve be sent to the request of the server and send
Rate-limit；
If the metadata request transmission rate upper limit between client and the server is greater than response receiving velocity, while from upper
The secondary transmission rate upper limit increases event and is greater than a given lag time (i.e. at one section by the time to receive feedback this moment
In the given time, the processing speed of destination server does not catch up with the request transmission rate of client still), then it is assumed that the target
Server congestion, and then downward is sent to the metadata request transmission rate upper limit of the server to alleviate the subsequent pressure of the server
Power；If other situations, then do not deal with.
In the present embodiment, to determine that the lag time of congestion is to shift to an earlier date specified value γ, each duplication can be specifically regarded
Depending on the situation of type metadata server cluster.
Client lowers the process for being sent to the metadata request transmission rate upper limit of destination server after perceiving congestion
It include: that client records the metadata request transmission rate upper limit this moment, to make destination server enter saturation shape
The peak rate of state；The metadata request transmission rate upper limit of oneself is reduced β times by client, and slow according to propagation process three times
It is slow to increase the transmission rate upper limit, if client experiences server end congestion again in the process, equally by metadata this moment
The request transmission rate upper limit is recorded as new peak rate, and above procedure is repeated.
Wherein, propagation process includes: to be recorded in convex function region, the request transmission rate upper limit currently than it three times
The peak rate for making destination server enter saturation state is low, therefore client can quickly increase the transmission rate upper limit of oneself；
In plateau, the close peak rate for making destination server enter saturation state of the current request transmission rate upper limit, client
The gradually stable transmission rate upper limit at this time, carefully improves the transmission rate upper limit with minimum amplification；It is in concave function region, i.e., objective
Family end has stopped a period long enough in plateau, the history peak value recorded at this time no longer make destination server into
Enter saturation state, therefore client is again introduced into the mode of the rapid growth transmission rate upper limit, to make full use of destination server
Computing resource.
With reference to Fig. 3, Fig. 3 is the metadata request distribution method based on congestion aware that further embodiment of this invention provides
In, metadata request transmission rate adjusts flow diagram,
Step 301: client updates the historical requests for the respective server that cost estimation function is related to according to feedback information
Response delay exponentially weighted moving average and the waiting list of the server and the exponential weighting of processing speed it is mobile
Average value.
Step 302: client judges whether its metadata request transmission rate upper limit between the server is not less than sound
Receiving velocity is answered, if it is not, thening follow the steps 303；Otherwise, step 304 is executed.
Step 303: client judges to be sent to the server request transmission rate upper limit from last time and increases event to receiving this moment
Whether it is greater than γ by the time to feedback, if it is greater, then executing step 305；Otherwise, step 306 is executed.
In this step, to determine that the lag time of congestion is to shift to an earlier date specified value γ, each science can be specifically regarded
Depending on the situation of metadata server cluster, so that system manager be allow to build and manage for greater flexibility distributed field system
System.
Step 304: client calculates the request transmission rate that future time window is sent to the server referring to following formula
The upper limit:
In formula, SijThe request transmission rate upper limit that client i is sent to destination server j is represented, alpha parameter is to control
The duration of plateau；In addition, SmaxThe peak value for making destination server enter saturation state recorded when after occurring for congestion
Rate, S after the rate-limit ratio that β parameter is lowered after being occurred by congestion, i.e. congestion occurij=Smax·(1-β)；Δ t is certainly
The last time transmission rate upper limit lowers event to the process time for receiving feedback this moment.
In this step, three times propagation process be a first rapid growth transmission rate to server may congestion the case where,
Then slowing down speedup process keeps server to stablize, until being further continued for increasing after one section according to the plateau of alpha parameter control
Transmission rate is to make full use of the process of server performance, which ensure that destination server is able to maintain higher throughput always,
It will not can't accept request because of the congestion at a certain moment before and lose a large amount of performance, avoid the negative of destination server
Carrier wave is dynamic.
Step 305: client records the metadata request transmission rate upper limit this moment, to make destination server
Into the peak rate of saturation state；The metadata request transmission rate upper limit of oneself is reduced β times by client.
In this step, the metadata request transmission rate upper limit of oneself is reduced β times by client, guarantees destination server not
It can be collapsed because of excessive metadata request, while new request will not be can't accept again and reduce overall throughput, made whole
A science metadata server cluster keeps high stability and high robustness.
Step 306: terminating feedback and update and speed adjust process.
Method provided in an embodiment of the present invention increased after judgement destination server not congestion using one three times
Journey is sent to the request transmission rate upper limit of the server to improve, and propagation process is a first rapid growth in simple terms three times for this
Transmission rate to server may congestion the case where, then slow down speedup process keep server stablize, until pass through one section of root
It is further continued for increasing process of the transmission rate to make full use of server performance after plateau according to alpha parameter control, which ensure that mesh
Mark server is able to maintain higher throughput always, will not can't accept request because of the congestion at a certain moment before and lose
A large amount of performance avoids the fluctuation of load of destination server.
On the other hand, after determining destination server congestion, instruct client by metadata request transmission rate this moment
The upper limit is recorded as the peak rate for making destination server enter saturation state, meanwhile, client sends out the metadata request of oneself
The transmission rate upper limit reduces β times, and slowly increases the transmission rate upper limit according to propagation process three times whenever, if client is again
It is secondary to experience server end congestion, then it will continue to repeat above procedure, guarantee that destination server will not be because of excessive metadata
It requests and collapses, while new request will not be can't accept again and reduce overall throughput, make entire science Metadata Service
Device cluster keeps high stability and high robustness.
On the basis of the above embodiments, the method also includes: according to the feedback information of the destination server, to institute
Required parameter in cost estimation function is stated to be updated.
Specifically, client is according to feedback information when client receives the response and feedback that destination server is sent back to
Relevant parameter needed for updating cost estimation function.The relevant parameter renewal process of cost estimation function, as updates the client
End is sent to the exponentially weighted moving average of the response delay of the historical requests of respective server and the waiting team of the server
The exponentially weighted moving average of column and processing speed.
In conclusion method provided by the invention is adaptive algorithm, it is only necessary in client maintenance historical feedback information
And count Real time request transmission rate and response receives rate, just can automatic sensing congestion, adjustment request transmission rate prevents from taking
The fluctuation of load and collapse at business device end, dramatically save the maintenance cost and time overhead of system manager, also improve
The throughput of cluster entirety, while client also obtains better experience.
Fig. 4 is the structural schematic diagram of a kind of electronic equipment of the embodiment of the present invention, as shown in figure 4, provided equipment packet
It includes: processor (processor) 401, communication interface (Communications Interface) 402, memory (memory)
403 and bus 404, wherein processor 401, communication interface 402, memory 403 complete mutual communication by bus 404.
Processor 401 can call the logical order in memory 403, to execute following method, for example, receive metadata request
And all server state informations for being sent to server are estimated according to the server state information by preset cost
Meter function carries out Performance Evaluation to all servers that are sent to, and obtains all performances for being sent to each server in server
Cost；The minimum server of the performance cost is chosen as destination server, judgement, which is known, is sent to mesh in actual time window
The number of requests for marking server is less than preset threshold, and Xiang Suoshu destination server sends metadata request；It is taken according to the target
The metadata request transmission rate upper limit of device of being engaged in and the size relation of response reception speed, and the last transmission rate upper limit increase
Feedback information elapsed time is received to this for a long time, equivalence determines the jam state information of the destination server, root
According to the jam state information of the destination server, the transmission rate of the metadata request is adjusted.
The embodiment of the present invention discloses a kind of computer program product, and computer program product includes being stored in non-transient calculating
Computer program on machine readable storage medium storing program for executing, computer program include program instruction, when program instruction is computer-executed,
Computer is able to carry out method provided by above-mentioned each method embodiment, for example, receive metadata request and it is all can
It is sent to the server state information of server, according to the server state information, by preset cost estimation function to institute
Server can be sent to by, which having, carries out Performance Evaluation, obtains all performance costs for being sent to each server in server；It chooses
For the minimum server of the performance cost as destination server, judgement, which is known, is sent to destination server in actual time window
Number of requests is less than preset threshold, and Xiang Suoshu destination server sends metadata request；According to first number of the destination server
According to the size relation for requesting the transmission rate upper limit and response reception speed, and last transmission rate upper limit rise time is to originally
Secondary to receive feedback information elapsed time, equivalence determines the jam state information of the destination server, according to the target
The jam state information of server adjusts the transmission rate of the metadata request.
The present embodiment provides a kind of non-transient computer readable storage medium, non-transient computer readable storage medium storages
Computer instruction, computer instruction make computer execute method provided by above-mentioned each method embodiment, for example, receive member
Request of data and all server state informations for being sent to server, according to the server state information, by default
Cost estimation function Performance Evaluation is carried out to all servers that are sent to, obtain and all are sent in server each and service
The performance cost of device；The minimum server of the performance cost is chosen as destination server, actual time window is known in judgement
The number of requests for being inside sent to destination server is less than preset threshold, and Xiang Suoshu destination server sends metadata request；According to institute
The metadata request transmission rate upper limit and response of stating destination server receive the size relation of speed, and last transmission speed
Rate upper limit rise time receives feedback information elapsed time to this, and equivalence determines the congestion state of the destination server
Information adjusts the transmission rate of the metadata request according to the jam state information of the destination server.
The apparatus embodiments described above are merely exemplary, wherein described, unit can as illustrated by the separation member
It is physically separated with being or may not be, component shown as a unit may or may not be physics list
Member, it can it is in one place, or may be distributed over multiple network units.It can be selected according to the actual needs
In some or all of the modules achieve the purpose of the solution of this embodiment.Those of ordinary skill in the art are not paying creativeness
Labour in the case where, it can understand and implement.
Through the above description of the embodiments, those skilled in the art can be understood that each embodiment can
It realizes by means of software and necessary general hardware platform, naturally it is also possible to pass through hardware.Based on this understanding, on
Stating technical solution, substantially the part that contributes to existing technology can be embodied in the form of software products in other words, should
Computer software product may be stored in a computer readable storage medium, such as ROM/RAM, magnetic disk, CD, including several fingers
It enables and using so that a computer equipment (can be personal computer, server or the network equipment etc.) executes each implementation
Method described in certain parts of example or embodiment.
Finally, it should be noted that the above embodiments are merely illustrative of the technical solutions of the present invention, rather than its limitations；Although
Present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that: it still may be used
To modify the technical solutions described in the foregoing embodiments or equivalent replacement of some of the technical features；
And these are modified or replaceed, technical solution of various embodiments of the present invention that it does not separate the essence of the corresponding technical solution spirit and
Range.
Claims (10)
1. a kind of metadata request distribution method based on congestion aware characterized by comprising
Metadata request is received according to server state information, server is sent to all by preset cost estimation function
Performance Evaluation is carried out, all performance costs for being sent to each server in server are obtained；
The minimum server of the performance cost is chosen as destination server, judgement, which is known, is sent to target in actual time window
The number of requests of server is less than preset threshold, and Xiang Suoshu destination server sends metadata request；
The size relation of speed, Yi Jishang are received according to the metadata request transmission rate upper limit of the destination server and response
Transmission rate upper limit rise time receives feedback information elapsed time to this, and equivalence determines the destination server
Jam state information the transmission rate of the metadata request is adjusted according to the jam state information of the destination server；
Wherein, described to be sent to server specifically: institute in the science metadata server cluster in distributed file system
There is the meta data server that can be accessed.
2. the method according to claim 1, wherein it is described by preset cost estimation function to all services
The step of device carries out Performance Evaluation, obtains the performance cost of each server in Servers-all, specifically includes:
The exponentially weighted moving average of the response delay for the historical requests that server can be sent to described in each is counted, and every
The waiting list of server and the exponentially weighted moving average of processing speed can be sent to described in one, estimating can described in each
It is sent to the performance cost of server.
3. the method according to claim 1, wherein described choose the minimum server conduct of the performance cost
After the step of destination server, further includes:
If judgement know that destination server number of requests is sent in actual time window reaches preset threshold, reselect not by
What is selected is sent to the minimum server of cost in server as new destination server.
4. according to the method described in claim 3, it is characterized in that, the method also includes:
If judgement knows that the number of requests for being sent to server all in actual time window reaches preset threshold, block institute
Metadata request is stated, handles the metadata request again in future time window.
5. the method according to claim 1, wherein the congestion state according to the destination server, is adjusted
The step of transmission rate of the whole metadata request, specifically includes:
Judgement knows that the metadata request transmission rate upper limit is less than response receiving velocity, then hair is improved referring to propagation process three times
Toward the metadata request transmission rate upper limit of the destination server.
6. according to the method described in claim 4, it is characterized in that, the step of the transmission rate of the adjustment metadata request
Suddenly further include:
Judgement knows that the metadata request transmission rate upper limit is greater than response receiving velocity, while increasing from the last time transmission rate upper limit
Event to the time of passing through for receiving feedback this moment is greater than a preset time, then lowers the metadata request transmission rate.
7. according to the method described in claim 4, it is characterized in that, the reference propagation process to improve is sent to the mesh three times
The step of marking the metadata request transmission rate upper limit of server specifically includes:
Respectively in convex function region, plateau and concave function region, according to the transmission rate upper limit of the metadata request and
The destination server enters the peak rate of saturation state, promotes the transmission rate upper limit of the metadata request.
8. according to any method of claim 5-7, which is characterized in that the method also includes:
According to the feedback information of the destination server, required parameter in the cost estimation function is updated.
9. a kind of electronic equipment including memory, processor and stores the calculating that can be run on a memory and on a processor
Machine program, which is characterized in that realized when the processor executes described program and be based on gathering around as described in any one of claim 1 to 8
The step of filling in the metadata request distribution method of perception.
10. a kind of non-transient computer readable storage medium, is stored thereon with computer program, which is characterized in that the computer
The metadata request distributor as described in any one of claim 1 to 8 based on congestion aware is realized when program is executed by processor
The step of method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811505304.8A CN109815204B (en) | 2018-12-10 | 2018-12-10 | Metadata request distribution method and equipment based on congestion perception |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811505304.8A CN109815204B (en) | 2018-12-10 | 2018-12-10 | Metadata request distribution method and equipment based on congestion perception |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109815204A true CN109815204A (en) | 2019-05-28 |
| CN109815204B CN109815204B (en) | 2021-03-02 |
Family
ID=66601910
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811505304.8A Active CN109815204B (en) | 2018-12-10 | 2018-12-10 | Metadata request distribution method and equipment based on congestion perception |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109815204B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112764931A (en) * | 2021-01-25 | 2021-05-07 | 北京字跳网络技术有限公司 | Request processing method, device, equipment, storage medium and program product |
| CN113259428A (en) * | 2021-05-11 | 2021-08-13 | 鸬鹚科技(深圳)有限公司 | Data access request processing method and device, computer equipment and medium |
| CN116582492A (en) * | 2023-07-14 | 2023-08-11 | 珠海星云智联科技有限公司 | Congestion control method, system and storage medium for optimizing RDMA reading |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101030926A (en) * | 2006-02-28 | 2007-09-05 | 华为技术有限公司 | Method for controlling network data flow of global microwave access inter-operation |
| CN101808118A (en) * | 2010-03-02 | 2010-08-18 | 浪潮(北京)电子信息产业有限公司 | Access method, device and system of server |
| CN102224722A (en) * | 2008-11-25 | 2011-10-19 | 思杰***有限公司 | Systems and methods for object rate limiting |
| JP2012073820A (en) * | 2010-09-29 | 2012-04-12 | Panasonic Corp | Wireless communication system, content list update method, server apparatus and client apparatus |
| CN103200208A (en) * | 2012-01-06 | 2013-07-10 | 湖南智源信息网络技术开发有限公司 | Method and device of data demand distribution based on frequency |
| CN105471760A (en) * | 2014-09-12 | 2016-04-06 | 华为技术有限公司 | Routing method, load balancing device and data communication system |
| US9712448B2 (en) * | 2011-03-30 | 2017-07-18 | International Business Machines Corporation | Proxy server, hierarchical network system, and distributed workload management method |
| CN107196865A (en) * | 2017-06-08 | 2017-09-22 | 中国民航大学 | A kind of adaptive threshold overload moving method of Load-aware |
| CN107196869A (en) * | 2017-07-14 | 2017-09-22 | 北京知道创宇信息技术有限公司 | The adaptive load balancing method, apparatus and system of Intrusion Detection based on host actual loading |
-
2018
- 2018-12-10 CN CN201811505304.8A patent/CN109815204B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101030926A (en) * | 2006-02-28 | 2007-09-05 | 华为技术有限公司 | Method for controlling network data flow of global microwave access inter-operation |
| CN102224722A (en) * | 2008-11-25 | 2011-10-19 | 思杰***有限公司 | Systems and methods for object rate limiting |
| CN101808118A (en) * | 2010-03-02 | 2010-08-18 | 浪潮(北京)电子信息产业有限公司 | Access method, device and system of server |
| JP2012073820A (en) * | 2010-09-29 | 2012-04-12 | Panasonic Corp | Wireless communication system, content list update method, server apparatus and client apparatus |
| US9712448B2 (en) * | 2011-03-30 | 2017-07-18 | International Business Machines Corporation | Proxy server, hierarchical network system, and distributed workload management method |
| CN103200208A (en) * | 2012-01-06 | 2013-07-10 | 湖南智源信息网络技术开发有限公司 | Method and device of data demand distribution based on frequency |
| CN105471760A (en) * | 2014-09-12 | 2016-04-06 | 华为技术有限公司 | Routing method, load balancing device and data communication system |
| CN107196865A (en) * | 2017-06-08 | 2017-09-22 | 中国民航大学 | A kind of adaptive threshold overload moving method of Load-aware |
| CN107196869A (en) * | 2017-07-14 | 2017-09-22 | 北京知道创宇信息技术有限公司 | The adaptive load balancing method, apparatus and system of Intrusion Detection based on host actual loading |
Non-Patent Citations (1)
| Title |
|---|
| CHIHEB BEN AMEUR等: "Combining traffic-shaping methods with congestion control variants for HTTP adaptive streaming", 《MULTIMEDIA SYSTEMS》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112764931A (en) * | 2021-01-25 | 2021-05-07 | 北京字跳网络技术有限公司 | Request processing method, device, equipment, storage medium and program product |
| CN112764931B (en) * | 2021-01-25 | 2023-07-28 | 北京字跳网络技术有限公司 | Request processing method, apparatus, device, storage medium, and program product |
| CN113259428A (en) * | 2021-05-11 | 2021-08-13 | 鸬鹚科技(深圳)有限公司 | Data access request processing method and device, computer equipment and medium |
| CN116582492A (en) * | 2023-07-14 | 2023-08-11 | 珠海星云智联科技有限公司 | Congestion control method, system and storage medium for optimizing RDMA reading |
| CN116582492B (en) * | 2023-07-14 | 2023-09-26 | 珠海星云智联科技有限公司 | Congestion control method, system and storage medium for optimizing RDMA reading |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109815204B (en) | 2021-03-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109788046B (en) | Multi-strategy edge computing resource scheduling method based on improved bee colony algorithm | |
| EP2710470B1 (en) | Extensible centralized dynamic resource distribution in a clustered data grid | |
| US20180018293A1 (en) | Method, controller, and system for service flow control in object-based storage system | |
| JP6380110B2 (en) | Resource control system, control pattern generation device, control device, resource control method, and program | |
| US20120221730A1 (en) | Resource control system and resource control method | |
| US11356338B2 (en) | Dynamic inter-cloud placement of virtual network functions for a slice | |
| US11178220B2 (en) | Method and system for transmitting a data resource acquisition request | |
| US11329901B2 (en) | Dynamic inter-cloud placement of virtual network functions for a slice | |
| US20150199219A1 (en) | Method and apparatus for server cluster management | |
| CN109815204A (en) | A kind of metadata request distribution method and equipment based on congestion aware | |
| CN104243405A (en) | Request processing method, device and system | |
| US20220075665A1 (en) | Scheduling method for selecting optimal cluster within cluster of distributed collaboration type | |
| CN105703927A (en) | Resource allocation method, network device and network system | |
| CN102480502B (en) | I/O load equilibrium method and I/O server | |
| CN108376103A (en) | A kind of the equilibrium of stock control method and server of cloud platform | |
| US11354164B1 (en) | Robotic process automation system with quality of service based automation | |
| CN113543160B (en) | 5G slice resource allocation method, device, computing equipment and computer storage medium | |
| CN115190121B (en) | Cross-region-based micro-service overload scheduling system, method and equipment | |
| CN109587068B (en) | Flow switching method, device, equipment and computer readable storage medium | |
| JP5997659B2 (en) | Distributed processing system and distributed processing method | |
| CN108228323B (en) | Hadoop task scheduling method and device based on data locality | |
| JP6280237B2 (en) | Computer system and data processing method | |
| CN110636104B (en) | Resource request method, electronic device and storage medium | |
| US20210136007A1 (en) | Method and apparatus for orchestrating resources in multi-access edge computing (mec) network | |
| CN113596146A (en) | Resource scheduling method and device based on big data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |