CN108712305B - Flow scheduling method based on substream flow value estimation method - Google Patents
Flow scheduling method based on substream flow value estimation method Download PDFInfo
- Publication number
- CN108712305B CN108712305B CN201810420239.2A CN201810420239A CN108712305B CN 108712305 B CN108712305 B CN 108712305B CN 201810420239 A CN201810420239 A CN 201810420239A CN 108712305 B CN108712305 B CN 108712305B
- Authority
- CN
- China
- Prior art keywords
- flow
- priority
- message
- host
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0876—Network utilisation, e.g. volume of load or congestion level
- H04L43/0882—Utilisation of link capacity
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/50—Queue scheduling
- H04L47/52—Queue scheduling by attributing bandwidth to queues
- H04L47/527—Quantum based scheduling, e.g. credit or deficit based scheduling or token bank
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/50—Queue scheduling
- H04L47/62—Queue scheduling characterised by scheduling criteria
- H04L47/625—Queue scheduling characterised by scheduling criteria for service slots or service orders
- H04L47/6275—Queue scheduling characterised by scheduling criteria for service slots or service orders based on priority
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Environmental & Geological Engineering (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a flow scheduling method based on a substream flow value estimation method, which comprises the following steps: s1, periodically reporting an estimated value of the flow sub-flow to the controller; s2: calculating the corresponding priority of the Coflow by a controller side Coflow priority formulation method, and sending a message to each host; s3, the host side updates the flow priority map and the virtual transmission completion time corresponding to each flow priority according to the flow priority information to generate a data message; s4, the host side enqueues the data message to obtain a storage queue of the flow message; and S5, dequeuing the data message to obtain a scheduling message, and realizing the scheduling of the flow. The invention solves the problems of lack of accuracy, influence on the optimization effect of the scheduling strategy, lack of rationality of the scheme and low optimization and processing efficiency in the prior art.
Description
Technical Field
The invention belongs to the technical field of internet, and particularly relates to a flow scheduling method based on a substream flow value estimation method.
Background
Data centers are large dedicated clusters of computers owned by specialized institutions, often of different sizes, to meet diverse usage needs. Data centers with tens of thousands of servers are often used by some large online service providers (such as Google, Microsoft, and Amazon) to run some large-scale data-intensive tasks, such as web indexing or large data set analysis. These data intensive tasks typically utilize the MapReduce paradigm (or variants) to perform parallel computational processing among the clusters of the data center and generate parallel cooperative data streams during the processing, and such cooperative data streams are often described by a Coflow model. Coflow is defined as a set of cooperative data flows with a common performance goal, and generally refers to a set of cooperative data flows generated by the same parallel computing application in a communication phase. Each individual flow in the set is referred to as a sub-flow of this Coflow. The completion time of the flow depends on the maximum value of the completion time of the transmission of the substreams therein. Because the transmission completion time of the Coflow is coordinated, the traditional network flow scheduling related means cannot be well applied to the Coflow, and therefore, a traffic scheduling scheme applicable to the Coflow needs to be researched so as to optimize the transmission completion time of the Coflow, thereby improving the processing capacity of the data center for parallel computing application. In fact, the flow scheduling scheme mainly needs to solve two problems: (1) how each traffic in the same flow allocates bandwidth. (2) How traffic in different flow allocates bandwidth.
In recent years, the problem of optimizing the flow completion time of the data center attracts the attention of a great number of researchers. Based on the intelligibility of the flow related information, the research in this aspect can be further divided into two categories — scheduling mechanism with known a priori knowledge and scheduling mechanism with unknown a priori knowledge. The scheduling mechanism known by a priori knowledge refers to that the complete flow characteristics can be obtained during decision making, including the number of sub-flows in the flow, the size and duration of each sub-flow, and the like. The scheduling mechanism with unknown a priori knowledge means that the complete flow characteristic cannot be obtained during decision making, or any characteristic information of the flow cannot be obtained at all.
In the two different research scenarios, the scheduling mechanism with known priori knowledge has sufficient information, and the idea is to establish a constraint condition according to network topology, network capacity and service requirements and an optimization problem model by taking the minimum flow average completion time as an objective function. The scheduling scheme in this scenario generally obtains a more detailed scheduling policy, including routing assignment of each substream of the Coflow and bandwidth allocation on a corresponding path, and has a better optimization effect. However, the flow scheduling mechanism in this scenario typically requires the assumption that the host is powerful enough to provide the full flow information, which is often impractical and not applicable to the actual data center network environment.
The problems of the prior art are as follows:
(1) in the prior art, a method for only sending the accumulated size of the finished message as the estimated value of the flow size is too rough, known information is not fully utilized, the current accumulated condition of the flow in a network is ignored, so that the estimation of the flow size is lack of accuracy, and the optimization effect of a scheduling strategy is influenced;
(2) in the prior art, the flow priority is set only according to the size of the flow, and the scheme is lack of rationality;
(3) in the prior art, the priority level is set for the flow according to the flow size, so that the average flow completion time is not reduced in some scenes, and the optimization and treatment efficiency are low.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides the flow scheduling method based on the estimation method of the flow value of the substream, which has high accuracy, high optimization, high rationality, high efficiency and high processing efficiency, and solves the problems of the prior art that the accuracy is lacked, the optimization effect of the scheduling strategy is influenced, the scheme is lacked in rationality and the optimization and processing efficiency are low.
In order to achieve the purpose of the invention, the invention adopts the technical scheme that:
a flow scheduling method based on a substream flow value estimation method comprises the following steps:
s1: periodically estimating each flow value of the flow of the Coflow on the host side by using a Coflow value estimation method, and periodically reporting the estimated value of the flow of the Coflow to a controller;
s2: according to the estimated value of the flow substream, calculating the corresponding priority of the flow by a controller side flow priority formulation method, and issuing a message carrying flow priority information to each host;
s3: adjusting the corresponding relation of local flow priority by using a controller message processing method on the host side according to the flow priority information carried in the message sent by the controller, updating the flow priority map and the virtual transmission completion time corresponding to each flow priority, and generating a data message;
s4: enqueuing the data message generated in the step S3 on the host side to obtain a storage queue of the flow message;
s5: and according to the storage queue obtained in the step S4 and the virtual transmission completion time corresponding to each flow priority, dequeuing the data message to obtain a scheduling message, and realizing the scheduling of the flow.
Further, in step S1, the method for estimating the flow rate value of the flow substream includes the following steps:
s1-1: judging whether a preset reporting period of the host side is reached, if so, entering a step S1-2, and executing statistics and reporting of the host side, otherwise, entering a step S1-4;
s1-2: calculating an estimated value of the local flow substream;
s1-3: the accumulated value of the arriving messages in the flow sub-flow period is reset to zero, a preset reporting period is reset, and the step S1-1 is entered;
s1-4: obtaining an accumulated value of the arriving message flow in a Coflow sub-flow period of the host side, a message flow accumulated value of the Coflow sub-flow in a transmission queue and a message flow accumulated value of the Coflow sub-flow transmission completion message;
s1-5: and judging whether the preset reporting simulation time length is reached, if so, ending the reporting of the host side, and otherwise, entering the step S1-1.
Further, in step S1-2, the calculation formula of the estimated value of the local Coflow substream is:
in the formula (I), the compound is shown in the specification,
flow estimation values of local flow sub-flows counted by a host at a sending end;
the accumulated message flow value of the flow sub-flow in the transmission queue of the sending end host is obtained;
accumulating the flow value of the transmission completion message counted by the host at the sending end for the flow sub-flow;
the flow sub-flow reaches the accumulated value of the message in the period of the sending end host.
Further, in step S2, the method for creating the controller-side flow priority includes the following steps:
s2-1: judging whether a preset execution period of the controller algorithm is reached, if so, entering a step S2-2, otherwise, entering a step S2-4;
s2-2: calculating a bottleneck value corresponding to the Coflow according to an estimated value of the flow of the local Coflow substream, and dividing the priority of each Coflow by combining a bottleneck threshold to obtain the corresponding priority of each Coflow;
s2-3: encoding the corresponding priority of each flow into a message, issuing the message to each host, resetting a preset execution period, and entering the step S2-1;
s2-4: counting estimated values of local flow substream flow reported by each host to obtain a total transmitting estimated value of the local flow substream and a total receiving estimated value of the local flow substream;
s2-5: and judging whether the preset simulation time of the algorithm is reached, if so, ending the control algorithm of the controller side, and otherwise, entering the step S2-1.
Further, in step S2-2, the calculation formula of the bottleneck value corresponding to the Coflow is:
wherein, bottleeck (c) is the bottleneck value corresponding to Coflow;
the total estimated value of the local flow sub-flow is sent;
the total receiving estimated value of the local flow sub-flow is obtained;
the calculation formula of the total receiving estimation value of the local flow substream is as follows:
in the formula (I), the compound is shown in the specification,
the total receiving estimated value of the local flow sub-flow is obtained;
receiving estimation value of local flow sub-flow; i belongs to the set of flows belonging to the flow c in the receiving end host n;
the calculation formula of the total transmission estimation value of the local flow substream is as follows:
in the formula (I), the compound is shown in the specification,
the total estimated value of the local flow sub-flow is sent;
sending estimation value of local flow sub-flow; i (n, c) is the set of flows belonging to Coflow c in the sending end host n.
Further, in step S3, the method for processing a controller packet includes the following steps:
s3-1: analyzing the content of the message in the step S2 at the host side to obtain the priority corresponding to each flow;
s3-2: obtaining the Coflow ID contained in each priority according to the corresponding relation between the Coflow and the priority thereof, and updating the Coflow priority map according to the sequence of the Coflow arrival time;
s3-3: judging whether the flow priority map is changed, if so, entering a step S3-4, otherwise, not updating the virtual transmission completion time corresponding to the flow priority, and ending the method;
s3-4: and calculating the virtual transmission completion time corresponding to each flow priority according to a WFQ scheduling method.
Further, in step S4, the method for enqueuing data packets includes the following steps:
s4-1: judging whether the host locally has a record of the flow to which the data message belongs, if so, entering step S4-5, otherwise, entering step S4-2;
s4-2: judging whether the queue corresponding to the highest priority in the flow priority map is empty, if so, entering a step S4-3, otherwise, not updating the virtual transmission completion time corresponding to the highest priority of the flow, and entering a step S4-4;
s4-3: calculating the virtual transmission completion time corresponding to the highest priority of the flow according to a WFQ scheduling method;
s4-4: setting the priority of the flow as the highest priority, and updating the local flow priority map of the host;
s4-5: and storing the data message and the arrival time of the flow data message into a storage queue corresponding to the flow number, and outputting the storage queue.
8. The flow scheduling method based on the estimation method of the flow rate of substreams according to claim 7, wherein in the step S4-3, the calculation formula of the virtual transmission completion time corresponding to the highest priority of the flow is:
in the formula, V _ finish [ i ] is the virtual transmission completion time corresponding to the flow priority; a [ i ] is the arrival time of the first message in the storage queue corresponding to the flow positioned at the head; p [ i ] is the message size value of the first message in the storage queue corresponding to the flow positioned at the head; w [ i ] is the weight corresponding to the flow priority i; w [ l ] is the corresponding weight of the flow priority l; r is the actual link rate.
Further, in step S5, the method for dequeuing data packets includes the following steps:
s5-1: selecting the priority corresponding to the minimum virtual transmission completion time as the priority of the to-be-scheduled flow in the current round according to the WFQ scheduling method and the virtual transmission completion time corresponding to each flow priority;
s5-2: according to the Coflow arrival time in the Coflow contained in the priority to be scheduled in the current round, obtaining the Coflow with the earliest arrival time as the Coflow to be scheduled, and taking the number of the Coflow as TRAN _ ID;
s5-3: selecting a flow storage queue with the number of TRAN _ ID, selecting a queue head message to pop up, and outputting a scheduling message.
S5-4: judging whether the corresponding flow message in the local round of to-be-scheduled flow priority obtained in the step S5-1 is empty, if so, not updating the virtual transmission completion time of the local round of to-be-scheduled flow priority, and ending the method, otherwise, entering the step S5-5;
s5-5: and calculating the virtual transmission completion time of the flow priority to be scheduled in the current round according to the WFQ scheduling method.
Further, the calculation formula of the virtual transmission completion time of the flow priority is as follows:
in the formula, V _ finish [ i ] is the virtual transmission completion time corresponding to the flow priority; v _ start [ i ] is the virtual transmission starting time corresponding to the flow priority; p [ i ] is the message size value of the first message in the storage queue corresponding to the flow positioned at the head; w [ i ] is the weight corresponding to the flow priority i; w [ l ] is the corresponding weight of the flow priority l; r is the actual link rate;
V_start[i]＝max{A[i],V_pre[i]}
in the formula, V _ start [ i ] is the virtual transmission starting time corresponding to the flow priority; a [ i ] is the arrival time of the first message in the storage queue corresponding to the flow positioned at the head; v _ pre [ i ] is the virtual transmission completion time last calculated by the priority i in the flow priority map.
The beneficial effect of this scheme does:
(1) according to the method, the flow priority is set according to the size of the flow bottleneck, known information is fully utilized, and the accuracy is improved;
(2) the estimation method of the flow sub-flow size is improved, and the reasonability of the scheme is improved;
(3) the average flow completion time is effectively reduced, and the processing efficiency of parallel computing application is improved;
(4) the flow scheduling mechanism provided by the invention is simple to deploy and low in algorithm complexity, can be applied to large-traffic scenes, and realizes efficient flow scheduling of a data center.
Drawings
FIG. 1 is a flow chart of a flow scheduling method based on a substream flow value estimation method;
FIG. 2 is a flow chart of a flow value estimation method for a flow substream;
FIG. 3 is a flow chart of a controller-side flow priority formulation method;
FIG. 4 is a flow chart of a method of controller message processing;
FIG. 5 is a flow chart of a method of data message enqueue processing;
fig. 6 is a flow chart of a method of dequeue processing of data packets.
Detailed Description
The following description of the embodiments of the present invention is provided to facilitate the understanding of the present invention by those skilled in the art, but it should be understood that the present invention is not limited to the scope of the embodiments, and it will be apparent to those skilled in the art that various changes may be made without departing from the spirit and scope of the invention as defined and defined in the appended claims, and all matters produced by the invention using the inventive concept are protected.
In the embodiment of the present invention, a flow scheduling method based on a substream flow value estimation method, as shown in fig. 1, includes the following steps:
s1: periodically estimating each flow value of the flow of the Coflow on the host side by using a Coflow value estimation method, and periodically reporting the estimated value of the flow of the Coflow to a controller;
the flow value estimation method of the flow substream, as shown in fig. 2, comprises the following steps:
s1-1: judging whether a preset reporting period of the host side is reached, if so, entering a step S1-2, and executing statistics and reporting of the host side, otherwise, entering a step S1-4;
s1-2: calculating an estimated value of the local flow substream;
the calculation formula of the estimation value of the local flow substream is as follows:
in the formula (I), the compound is shown in the specification,
flow estimation values of local flow sub-flows counted by a host at a sending end;
the accumulated message flow value of the flow sub-flow in the transmission queue of the sending end host is obtained;
accumulating the flow value of the transmission completion message counted by the host at the sending end for the flow sub-flow;
the flow sub-flow reaches the accumulated value of the message in the period of the sending end host;
s1-3: the accumulated value of the arriving messages in the flow sub-flow period is reset to zero, a preset reporting period is reset, and the step S1-1 is entered;
s1-4: obtaining an accumulated value of the arriving message flow in a Coflow sub-flow period of the host side, a message flow accumulated value of the Coflow sub-flow in a transmission queue and a message flow accumulated value of the Coflow sub-flow transmission completion message;
s1-5: judging whether the preset reporting simulation duration is reached, if so, ending the reporting of the host side, and otherwise, entering the step S1-1;
when the method for estimating the flow value of the flow substream runs, the preset reporting period cannot be reached in the first cycle, so that the method directly enters the step S1-4 from the step S1-1 to obtain an accumulated value of the reached message flow in the flow substream period of the host side, an accumulated value of the message flow of the flow substream in a transmission queue and an accumulated value of the message flow of the flow substream transmission completion message, wherein the preset reporting period is less than the preset reporting simulation duration, the method reenters the step S1-1, and when the preset reporting period of the host side is reached, the estimated value of the local flow substream is calculated;
s2: according to the estimated value of the flow substream, calculating the corresponding priority of the flow by a controller side flow priority formulation method, and issuing a message carrying flow priority information to each host;
the controller-side flow priority setting method, as shown in fig. 3, includes the following steps:
s2-1: judging whether a preset execution period of the controller algorithm is reached, if so, entering a step S2-2, otherwise, entering a step S2-4;
s2-2: calculating a bottleneck value corresponding to the Coflow according to an estimated value of the flow of the local Coflow substream, and dividing the priority of each Coflow by combining a bottleneck threshold to obtain the corresponding priority of each Coflow;
the calculation formula of the bottleneck value corresponding to the flow is as follows:
wherein, bottleeck (c) is the bottleneck value corresponding to Coflow;
the total estimated value of the local flow sub-flow is sent;
the total receiving estimated value of the local flow sub-flow is obtained;
the calculation formula of the total receiving estimation value of the local flow substream is as follows:
in the formula (I), the compound is shown in the specification,
the total receiving estimated value of the local flow sub-flow is obtained;
is a local flow sonA received estimate of the stream; i belongs to the set of flows belonging to the flow c in the receiving end host n;
the calculation formula of the total transmission estimation value of the local flow substream is as follows:
in the formula (I), the compound is shown in the specification,
the total estimated value of the local flow sub-flow is sent;
sending estimation value of local flow sub-flow; i (n, c) is a set of flows belonging to the flow c in the sending end host n;
s2-3: encoding the corresponding priority of each flow into a message, issuing the message to each host, resetting a preset execution period, and entering the step S2-1;
s2-4: counting estimated values of local flow substream flow reported by each host to obtain a total transmitting estimated value of the local flow substream and a total receiving estimated value of the local flow substream;
s2-5: judging whether the preset simulation duration of the algorithm is reached, if so, ending the control algorithm of the controller side, otherwise, entering the step S2-1;
when the controller-side flow priority level formulation method runs, the preset execution period of the controller algorithm cannot be reached in the first cycle, so that the method directly enters the step S2-4 from the step S2-1, estimates of the flow rates of the local flow sub-flows reported by all hosts are counted to obtain a total transmission estimation value of the local flow sub-flows and a total receiving estimation value of the local flow sub-flows, the preset execution period is shorter than the preset simulation duration of the algorithm, the step S2-1 is re-entered, when the preset execution period of the controller algorithm is reached, a bottleneck value corresponding to the flow is calculated according to the estimation values of the flow rates of the local flow sub-flows, and priority levels of all the flow sub-flows are divided by combining with bottleneck threshold values to obtain corresponding priority levels of all the flow sub-flows;
s3: adjusting the corresponding relation of local flow priority by using a controller message processing method on the host side according to the flow priority information carried in the message sent by the controller, updating the flow priority map and the virtual transmission completion time corresponding to each flow priority, and generating a data message;
as shown in fig. 4, the method for processing a controller message includes the following steps:
s3-1: analyzing the content of the message in the step S2 at the host side to obtain the priority corresponding to each flow;
s3-2: obtaining the Coflow ID contained in each priority according to the corresponding relation between the Coflow and the priority thereof, and updating the Coflow priority map according to the sequence of the Coflow arrival time;
s3-3: judging whether the flow priority map is changed, if so, entering a step S3-4, otherwise, not updating the virtual transmission completion time corresponding to the flow priority, and ending the method;
s3-4: calculating the virtual transmission completion time corresponding to each flow priority according to a WFQ scheduling method;
the calculation formula of the virtual transmission completion time of the flow priority is as follows:
in the formula, V _ finish [ i ] is the virtual transmission completion time corresponding to the flow priority; v _ start [ i ] is the virtual transmission starting time corresponding to the flow priority; p [ i ] is the message size value of the first message in the storage queue corresponding to the flow positioned at the head; w [ i ] is the weight corresponding to the flow priority i; w [ l ] is the corresponding weight of the flow priority l; r is the actual link rate;
V_start[i]＝max{A[i],V_pre[i]}
in the formula, V _ start [ i ] is the virtual transmission starting time corresponding to the flow priority; a [ i ] is the arrival time of the first message in the storage queue corresponding to the flow positioned at the head; v _ pre [ i ] is the virtual transmission completion time of the previous calculation of the priority i in the flow priority map;
s4: enqueuing the data message generated in the step S3 on the host side to obtain a storage queue of the flow message;
as shown in fig. 5, the method for enqueuing data packets includes the following steps:
s4-1: judging whether the host locally has a record of the flow to which the data message belongs, if so, entering step S4-5, otherwise, entering step S4-2;
s4-2: judging whether the queue corresponding to the highest priority in the flow priority map is empty, if so, entering a step S4-3, otherwise, not updating the virtual transmission completion time corresponding to the highest priority of the flow, and entering a step S4-4;
s4-3: calculating the virtual transmission completion time corresponding to the highest priority of the flow according to a WFQ scheduling method;
the calculation formula of the virtual transmission completion time corresponding to the highest priority of the flow is as follows:
in the formula, V _ finish [ i ] is the virtual transmission completion time corresponding to the flow priority; a [ i ] is the arrival time of the first message in the storage queue corresponding to the flow positioned at the head; p [ i ] is the message size value of the first message in the storage queue corresponding to the flow positioned at the head; w [ i ] is the weight corresponding to the flow priority i; w [ l ] is the corresponding weight of the flow priority l; r is the actual link rate;
s4-4: setting the priority of the flow as the highest priority, and updating the local flow priority map of the host;
s4-5: storing the data message and the arrival time of the flow data message into a storage queue corresponding to the flow number, and outputting the storage queue;
s5: according to the storage queue obtained in the step S4 and the virtual transmission completion time corresponding to each flow priority, dequeuing the data message to obtain a scheduling message, and realizing the scheduling of the flow;
as shown in fig. 6, the method for dequeuing data packets includes the following steps:
s5-1: selecting the priority corresponding to the minimum virtual transmission completion time as the priority of the to-be-scheduled flow in the current round according to the WFQ scheduling method and the virtual transmission completion time corresponding to each flow priority;
s5-2: according to the Coflow arrival time in the Coflow contained in the priority to be scheduled in the current round, obtaining the Coflow with the earliest arrival time as the Coflow to be scheduled, and taking the number of the Coflow as TRAN _ ID;
s5-3: selecting a flow storage queue with the number of TRAN _ ID, selecting a queue head message to pop up, and outputting a scheduling message.
S5-4: judging whether the corresponding flow message in the local round of to-be-scheduled flow priority obtained in the step S5-1 is empty, if so, not updating the virtual transmission completion time of the local round of to-be-scheduled flow priority, and ending the method, otherwise, entering the step S5-5;
s5-5: calculating the virtual transmission completion time of the flow priority to be scheduled in the current round according to a WFQ scheduling method;
the calculation formula of the virtual transmission completion time of the flow priority is as follows:
in the formula, V _ finish [ i ] is the virtual transmission completion time corresponding to the flow priority; v _ start [ i ] is the virtual transmission starting time corresponding to the flow priority; p [ i ] is the message size value of the first message in the storage queue corresponding to the flow positioned at the head; w [ i ] is the weight corresponding to the flow priority i; w [ l ] is the corresponding weight of the flow priority l; r is the actual link rate;
V_start[i]＝max{A[i],V_pre[i]}
in the formula, V _ start [ i ] is the virtual transmission starting time corresponding to the flow priority; a [ i ] is the arrival time of the first message in the storage queue corresponding to the flow positioned at the head; v _ pre [ i ] is the virtual transmission completion time last calculated by the priority i in the flow priority map.
The invention provides the flow scheduling method based on the estimation method of the flow value of the substream, which has high accuracy, high optimization, high rationality, high efficiency and high processing efficiency, and solves the problems of the prior art that the accuracy is lacked, the optimization effect of a scheduling strategy is influenced, the scheme is lacked in rationality and the optimization and processing efficiency are low.
Claims (9)
1. A flow scheduling method based on a substream flow value estimation method is characterized by comprising the following steps:
s1: periodically estimating each flow value of the flow of the Coflow on the host side by using a Coflow value estimation method, and periodically reporting the estimated value of the flow of the Coflow to a controller;
s2: according to the estimated value of the flow of the subflow flow, calculating the priority corresponding to the flow by a controller side flow priority formulating method, and issuing a message carrying the flow priority information to each host, wherein in the step S2, the controller side flow priority formulating method comprises the following steps:
s2-1: judging whether a preset execution period of the controller algorithm is reached, if so, entering a step S2-2, otherwise, entering a step S2-4;
s2-2: calculating a bottleneck value corresponding to the Coflow according to an estimated value of the local Coflow sub-flow of the host, and dividing the priority of each Coflow by combining a bottleneck threshold to obtain the corresponding priority of each Coflow;
s2-3: encoding the corresponding priority of each flow into a message, issuing the message to each host, resetting a preset execution period, and entering the step S2-1;
s2-4: counting estimated values of host local flow substream flow reported by each host to obtain a total transmitted estimated value of the host local flow substream and a total received estimated value of the host local flow substream;
s2-5: judging whether the preset simulation duration of the algorithm is reached, if so, ending the control algorithm of the controller side, otherwise, entering the step S2-1;
s3: adjusting the corresponding relation of local flow priority of the host by using a controller message processing method on the host side according to flow priority information carried in a message sent by a controller, updating a flow priority map and virtual transmission completion time corresponding to each flow priority, and generating a data message, wherein the map represents mapping;
s4: enqueuing the data message generated in the step S3 on the host side to obtain a storage queue of the flow message;
s5: and according to the storage queue obtained in the step S4 and the virtual transmission completion time corresponding to each flow priority, dequeuing the data message to obtain a scheduling message, and realizing the scheduling of the flow.
2. The flow scheduling method based on the substream flow value estimation method of claim 1, wherein in the step S1, the flow substream flow value estimation method comprises the following steps:
s1-1: judging whether a preset reporting period of the host side is reached, if so, entering a step S1-2, and executing statistics and reporting of the host side, otherwise, entering a step S1-4;
s1-2: calculating an estimated value of the local flow substream of the host;
s1-3: the accumulated value of the arriving message flow in the flow sub-flow period is reset to zero, a preset reporting period is reset, and the step S1-1 is entered;
s1-4: obtaining an accumulated value of the arriving message flow in a Coflow sub-flow period of the host side, a message flow accumulated value of the Coflow sub-flow in a transmission queue and a message flow accumulated value of the Coflow sub-flow transmission completion message;
s1-5: and judging whether the preset reporting simulation time length is reached, if so, ending the reporting of the host side, and otherwise, entering the step S1-1.
3. The flow scheduling method based on the estimation method of flow rate of substreams as claimed in claim 2, wherein in step S1-2, the calculation formula of the estimation value of the host local flow substream is:
in the formula (I), the compound is shown in the specification,
local flow of flow for host atFlow estimation values counted by a sending end host;
the accumulated message flow value of the flow sub-flow in the transmission queue of the sending end host is obtained;
accumulating the flow value of the transmission completion message counted by the host at the sending end for the flow sub-flow;
the flow sub-flow reaches the accumulated value of the message flow in the period of the sending end host.
4. The flow scheduling method based on the estimation method of the flow rate of the substream, as recited in claim 1, wherein in the step S2-2, the calculation formula of the bottleneck value corresponding to the flow is:
wherein, bottleeck (c) is the bottleneck value corresponding to Coflow;
the total estimated value of the local flow sub-flow is sent;
the total receiving estimated value of the local flow sub-flow of the host is obtained;
the calculation formula of the total receiving estimation value of the local flow substream is as follows:
in the formula (I), the compound is shown in the specification,
the total receiving estimated value of the local flow sub-flow of the host is obtained;
receiving estimation value of local flow sub-flow of the host; i belongs to the set of flows belonging to the flow c in the receiving end host n;
the calculation formula of the total sending estimation value of the host local flow subflows is as follows:
in the formula (I), the compound is shown in the specification,
sending a total estimated value of the local flow sub-flow of the host;
sending estimation value of local flow sub-flow of host; i (n, c) is the set of flows belonging to Coflow c in the sending end host n.
5. The flow scheduling method based on the estimation method of the flow rate of substreams according to claim 1, wherein in the step S3, the controller message processing method comprises the following steps:
s3-1: analyzing the content of the message in the step S2 at the host side to obtain the priority corresponding to each flow;
s3-2: obtaining the flow ID contained in each priority according to the corresponding relation between the flow and the priority thereof, and updating the flow priority map according to the sequence of the flow arrival time;
s3-3: judging whether the flow priority map is changed, if so, entering a step S3-4, otherwise, not updating the virtual transmission completion time corresponding to the flow priority, and ending the method, wherein the map represents mapping;
s3-4: and calculating the virtual transmission completion time corresponding to each flow priority according to a WFQ scheduling method.
6. The flow scheduling method based on the substream flow value estimation method of claim 1, wherein in the step S4, the method for enqueuing data packets comprises the following steps:
s4-1: judging whether the host locally has a record of the flow to which the data message belongs, if so, entering step S4-5, otherwise, entering step S4-2;
s4-2: judging whether a queue corresponding to the highest priority in a flow priority map is empty, if so, entering a step S4-3, otherwise, not updating virtual transmission completion time corresponding to the highest priority of the flow, and entering a step S4-4, wherein the map represents mapping;
s4-3: calculating the virtual transmission completion time corresponding to the highest priority of the flow according to a WFQ scheduling method;
s4-4: setting the priority of the flow as the highest priority, and updating the local flow priority map of the host;
s4-5: and storing the data message and the arrival time of the flow data message into a storage queue corresponding to the flow number, and outputting the storage queue.
7. The flow scheduling method based on the estimation method of the flow rate of substreams according to claim 6, wherein in the step S4-3, the calculation formula of the virtual transmission completion time corresponding to the highest priority of the flow is:
in the formula, V _ finish [ i ] is the virtual transmission completion time corresponding to the flow priority; a [ i ] is the arrival time of the first message in the storage queue corresponding to the flow positioned at the head; p [ i ] is the message size value of the first message in the storage queue corresponding to the flow positioned at the head; w [ i ] is the weight corresponding to the flow priority i; w [ l ] is the corresponding weight of the flow priority l; r is the actual link rate.
8. The flow scheduling method based on the substream flow value estimation method of claim 1, wherein in the step S5, the method for dequeuing the data packets comprises the following steps:
s5-1: selecting the priority corresponding to the minimum virtual transmission completion time as the priority of the to-be-scheduled flow in the current round according to the WFQ scheduling method and the virtual transmission completion time corresponding to each flow priority;
s5-2: according to the Coflow arrival time in the Coflow contained in the priority to be scheduled in the current round, obtaining the Coflow with the earliest arrival time as the Coflow to be scheduled, and taking the number of the Coflow as TRAN _ ID;
s5-3: selecting a flow storage queue with the number of TRAN _ ID, selecting a queue head message from the flow storage queue to pop up, and outputting a scheduling message;
s5-4: judging whether the corresponding flow message in the local round of to-be-scheduled flow priority obtained in the step S5-1 is empty, if so, not updating the virtual transmission completion time of the local round of to-be-scheduled flow priority, and ending the method, otherwise, entering the step S5-5;
s5-5: and calculating the virtual transmission completion time of the flow priority to be scheduled in the current round according to the WFQ scheduling method.
9. The flow scheduling method based on the estimation method of the flow rate value of the substream, according to claim 5 or 8, characterized in that the calculation formula of the virtual transmission completion time of the flow priority is:
in the formula, V _ finish [ i ] is the virtual transmission completion time corresponding to the flow priority; v _ start [ i ] is the virtual transmission starting time corresponding to the flow priority; p [ i ] is the message size value of the first message in the storage queue corresponding to the flow positioned at the head; w [ i ] is the weight corresponding to the flow priority i; w [ l ] is the corresponding weight of the flow priority l; r is the actual link rate;
V_start[i]＝max{A[i],V_pre[i]}
in the formula, V _ start [ i ] is the virtual transmission starting time corresponding to the flow priority; a [ i ] is the arrival time of the first message in the storage queue corresponding to the flow positioned at the head; v _ pre [ i ] is the virtual transmission completion time calculated last by the priority i in the flow priority map, and the map represents the mapping.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810420239.2A CN108712305B (en) | 2018-05-04 | 2018-05-04 | Flow scheduling method based on substream flow value estimation method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810420239.2A CN108712305B (en) | 2018-05-04 | 2018-05-04 | Flow scheduling method based on substream flow value estimation method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108712305A CN108712305A (en) | 2018-10-26 |
| CN108712305B true CN108712305B (en) | 2020-03-31 |
Family
ID=63867849
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810420239.2A Active CN108712305B (en) | 2018-05-04 | 2018-05-04 | Flow scheduling method based on substream flow value estimation method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108712305B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110708259B (en) * | 2019-09-25 | 2023-07-07 | 江苏省未来网络创新研究院 | Information-unaware Coflow scheduling system capable of automatically adjusting queue threshold value and scheduling method thereof |
| CN111756653B (en) * | 2020-06-04 | 2022-03-04 | 北京理工大学 | Multi-coflow scheduling method based on deep reinforcement learning of graph neural network |
| WO2022074415A1 (en) * | 2020-10-06 | 2022-04-14 | Huawei Technologies Co., Ltd. | Device and method for coflow tracking and scheduling |
| CN117221126B (en) * | 2023-11-09 | 2024-02-13 | 之江实验室 | Network collaboration flow-oriented route scheduling method and system |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105227488A (en) * | 2015-08-25 | 2016-01-06 | 上海交通大学 | A kind of network flow group scheduling method for distributed computer platforms |
| CN105827545A (en) * | 2016-04-21 | 2016-08-03 | 中国科学院信息工程研究所 | Scheduling method and device of TCP co-flows in data center network |
-
2018
- 2018-05-04 CN CN201810420239.2A patent/CN108712305B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105227488A (en) * | 2015-08-25 | 2016-01-06 | 上海交通大学 | A kind of network flow group scheduling method for distributed computer platforms |
| CN105827545A (en) * | 2016-04-21 | 2016-08-03 | 中国科学院信息工程研究所 | Scheduling method and device of TCP co-flows in data center network |
Non-Patent Citations (1)
| Title |
|---|
| Leveraging multiple coflow attributes for information-agnostic coflow scheduling;Shuo Wang 等;《2017 IEEE International Conference on Communications (ICC)》;20170731;第II-III节 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108712305A (en) | 2018-10-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108712305B (en) | Flow scheduling method based on substream flow value estimation method | |
| US10764215B2 (en) | Programmable broadband gateway hierarchical output queueing | |
| Sharma et al. | Programmable calendar queues for high-speed packet scheduling | |
| Ghodsi et al. | Multi-resource fair queueing for packet processing | |
| Guo et al. | On efficient bandwidth allocation for traffic variability in datacenters | |
| CN108768876B (en) | Traffic scheduling method facing machine learning framework | |
| Liu et al. | eBA: Efficient bandwidth guarantee under traffic variability in datacenters | |
| CN101834786B (en) | Queue scheduling method and device | |
| WO2018233425A1 (en) | Network congestion processing method, device, and system | |
| US20220124039A1 (en) | Fine Grain Traffic Shaping Offload For A Network Interface Card | |
| EP2740245A1 (en) | A scalable packet scheduling policy for vast number of sessions | |
| Oral et al. | Evaluation of the shortest processing time scheduling rule with truncation process | |
| Li et al. | Endpoint-flexible coflow scheduling across geo-distributed datacenters | |
| CN113132265A (en) | Multi-stage scheduling method and device for multi-path Ethernet | |
| CN110177056B (en) | Automatic adaptive bandwidth control method | |
| Gao et al. | Flash: Joint flow scheduling and congestion control in data center networks | |
| CN101374109A (en) | Method and apparatus for scheduling packets | |
| Gao et al. | DemePro: DEcouple packet Marking from Enqueuing for multiple services with PROactive congestion control | |
| Jajoo et al. | Saath: Speeding up coflows by exploiting the spatial dimension | |
| da Silva et al. | Algorithm for the placement of groups of virtual machines in data centers | |
| CN106209683B (en) | Data transmission method and system based on data center's wide area network | |
| Zhang et al. | Scheduling coflows with incomplete information | |
| Uchechukwu et al. | Scalable analytic models for performance efficiency in the cloud | |
| Wang et al. | Efficient and fair: Information-agnostic online coflow scheduling by combining limited multiplexing with drl | |
| Shi et al. | On the extreme parallelism inside next-generation network processors |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |