CN112073496B - Load balancing-based data placement method in geographically distributed cloud - Google Patents

Load balancing-based data placement method in geographically distributed cloud Download PDF

Info

Publication number
CN112073496B
CN112073496B CN202010906169.9A CN202010906169A CN112073496B CN 112073496 B CN112073496 B CN 112073496B CN 202010906169 A CN202010906169 A CN 202010906169A CN 112073496 B CN112073496 B CN 112073496B
Authority
CN
China
Prior art keywords
data
cloud
geographically distributed
data center
load balancing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010906169.9A
Other languages
Chinese (zh)
Other versions
CN112073496A (en
Inventor
李春林
鄢金卫
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Wuhan University of Technology WUT
Original Assignee
Wuhan University of Technology WUT
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wuhan University of Technology WUT filed Critical Wuhan University of Technology WUT
Priority to CN202010906169.9A priority Critical patent/CN112073496B/en
Publication of CN112073496A publication Critical patent/CN112073496A/en
Application granted granted Critical
Publication of CN112073496B publication Critical patent/CN112073496B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
    • H04L67/1004—Server selection for load balancing
    • H04L67/1008—Server selection for load balancing based on parameters of servers, e.g. available memory or workload
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10—Complex mathematical operations
    • G06F17/15—Correlation function computation including computation of convolution operations
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
    • H04L67/1004—Server selection for load balancing
    • H04L67/1023—Server selection for load balancing based on a hash applied to IP addresses or costs

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Signal Processing (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Mathematical Analysis (AREA)
  • Mathematical Optimization (AREA)
  • Pure & Applied Mathematics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Algebra (AREA)
  • Computer Hardware Design (AREA)
  • Databases & Information Systems (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

The invention relates to a data placement method based on load balancing in a geographically distributed cloud, which comprises the following steps: acquiring the free capacity and the number of servers of each cloud data center; calculating according to the idle capacity to obtain the capacity limit of the cloud data center; obtaining load balancing limitation of the geographically distributed cloud according to the number of the servers; calculating to obtain data transmission bandwidth cost limit in the geographically distributed cloud; constructing an objective function about data placement of the geographically distributed cloud according to the load balancing limit, the capacity limit of the cloud data center and the data transmission bandwidth cost limit; the objective function is to minimize the data transfer time for data placement; solving the objective function by adopting a Lagrange relaxation method to obtain the data transmission time for placing the minimized data; and outputting the data placement scheme matrix when the data transmission time is minimum. The method and the device can effectively reduce the data transmission time in the data placement process and ensure the load balance of the geographically distributed cloud.

Description

Load balancing-based data placement method in geographically distributed cloud
Technical Field
The invention relates to the technical field of computer cloud storage, in particular to a data placement method based on load balancing in a geographically distributed cloud.
Background
The birth of cloud computing is the product of the information technology revolution. The cloud computing applies a virtualization technology to integrate a large number of IT resources such as servers, computing clusters, storage devices and the like distributed in different areas into a logically unified virtual resource pool, and provides various safe, reliable, low-cost and highly-extensible computing or storage services for users. Currently, the amount of data from users, sensors and algorithms is exponentially growing, and most of the data is stored in data centers distributed in different geographical locations around the world. In order to truly realize globalization of cloud computing, it is a necessary trend to jointly use different cloud services of different cloud providers, that is, different clouds are connected to each other according to general standards and policies to form a geographically distributed cloud, so as to provide a general environment for cloud computing. Cloud data center systems that span multiple geographic locations are currently very popular, aiming to bring services close to users, reduce power consumption costs, and achieve stability of services in the face of network or power failures. Facebook, Google, microsoft and Amazon have built large-scale data centers worldwide, investing large amounts of money in building their services.
In the big data era, data centers must manage the amount of data from TB level to PB level. For large geographically distributed data processing, each regional data center can only accommodate a portion of the data set, subject to storage capacity constraints. Job processing in a geographically distributed cloud requires transferring all data to be processed to a cloud data center where a job is located, and too long time for placing data may result in increased job delay, thereby increasing response time of a data stream application. Therefore, how to effectively place data in a geographically distributed cloud becomes an urgent issue to be solved.
The research or application of data placement in the current mainstream geographically distributed cloud environment focuses on considering both transmission time and bandwidth cost in the data placement process, and no consideration is given to load balancing of the geographically distributed cloud system after data placement.
The current mainstream technology has the following defects:
(1) because cluster configurations of the cloud data centers in the geographically distributed cloud are different, the computing capacities of the cloud data centers are different;
(2) due to different loads generated by processing different data, the consumption of computing resources of the cloud data center is different.
One obvious manifestation of the above defects in practical applications is that most of the computing resources of the cloud data center with a small load may be kept idle; and the cloud data center processing load may be large, which may result in inefficient job execution. Obviously, the 'idle and dead, accumulated and dead' condition is not favorable for reasonable allocation of resources, and needs to be solved urgently, but no solution considering load balancing limitation, capacity limitation of cloud data centers and data transmission bandwidth cost limitation is available in the market at present.
Disclosure of Invention
The invention aims to solve the problems and provides a data placement method based on load balancing in a geographically distributed cloud, which aims to effectively reduce data transmission time in the data placement process and ensure load balancing of the geographically distributed cloud.
In order to solve the problems, the technical scheme provided by the invention is as follows:
the data placement method based on load balancing in the geographically distributed cloud comprises the following steps:
s100, acquiring the idle capacity of each cloud data center and the number of servers of each cloud data center; then calculating according to the free capacity of each cloud data center to obtain the capacity limit of the cloud data center; obtaining load balancing limitation of the geographically distributed cloud according to the number of the servers of each cloud data center;
s200, calculating to obtain data transmission bandwidth cost limit in the geographically distributed cloud;
the calculating to obtain the data transmission bandwidth cost limit in the geographically distributed cloud comprises the following steps:
s210, constructing a weighted directed graph of transmission bandwidth cost of data placement in the geographically distributed cloud;
s220, solving the multisource shortest path of the weighted directed graph by adopting a Floyd algorithm;
s230, calculating according to the result of solving the multisource shortest path of the weighted directed graph to obtain the cost limit of the data transmission bandwidth in the geographically distributed cloud;
s300, constructing an objective function related to data placement of the geographically distributed cloud according to the load balancing limit, the capacity limit of the cloud data center and the data transmission bandwidth cost limit; the objective function is to minimize the data transfer time for the data placement;
s400, solving the objective function by adopting a Lagrange relaxation method to obtain the minimized data transmission time for placing the data; outputting a data placement scheme matrix when the data transmission time is minimum;
the solving of the objective function by using a Lagrange relaxation method to obtain the minimized data transmission time for the data placement comprises the following steps:
s410, converting the target function into a linear programming target function;
s420, absorbing the complex constraint condition of the linear programming objective function into the linear programming objective function by adopting a Lagrange relaxation method to obtain a Lagrange relaxation function;
s430, initializing penalty factors, sub-gradients, step sizes and iteration times of the Lagrange relaxation function;
s440, calculating the minimum value of the Lagrangian relaxation function according to the current value of the penalty factor; calculating to obtain the data placement scheme matrix according to the current value of the penalty factor;
s450, updating the secondary gradient and the step length; then updating the penalty factor according to the updated value of the secondary gradient and the updated value of the step length;
s460, judging whether the data placement scheme matrix meets the target of the target function or not, and judging whether the iteration times are equal to an artificially preset iteration time threshold or not; according to the judgment result, the following operations are carried out:
if the data placement scheme matrix meets the target of the objective function, or the iteration times are equal to an artificially preset iteration time threshold value, outputting the data placement scheme matrix;
otherwise, return to S440.
Preferably, the capacity limit of the cloud data center is expressed by the following formula:
Figure GDA0003118813030000041
wherein: ckThe remaining free capacity of the kth cloud data center; m is the number of the jobs to be scheduled; the job to be scheduled is represented as a set { T }1,T2,T3,...,TmWhere T ismThe mth job in the job set to be scheduled; n is the number of blocks of data to be placed; the data to be placed is represented as a set { D }1,D2,D3,...,DnIn which D isnThe nth data block in the data set to be placed is obtained; alpha is alphamnFor selecting variables, the value range is alphamnE {0,1}, when processing operation TmData D of neednTaking 1 when the current value is zero, or taking 0 when the current value is zero; gamma raymkFor selecting variables, the value range is gammamk∈{0,1}，GkFor the kth cloud data center, when the operation TmIs scheduled to cloud data center GkTaking 1 when the current value is zero, or taking 0 when the current value is zero; gnTo contain the data D to be placednThe cloud data center of (1);
Figure GDA0003118813030000042
for the slave cloud data center GnTo a data center GkThe amount of data of (a); g is the geographically distributed cloud, represented as a set { G }1,G2,G3,...,Gk}。
Preferably, the load balancing constraint is expressed as:
Figure GDA0003118813030000043
wherein: eta1Is a minimum load balancing parameter, η1≤1；η2Is a maximum load balancing parameter, η2Not less than 1; k is the total number of cloud data centers in the geographically distributed cloud; skThe number of servers of the kth cloud data center; l is the total load of the geographically distributed cloud; l iskIs the load of the kth cloud data center.
Preferably, the solving the multisource shortest path of the weighted directed graph by using the Floyd algorithm includes the following steps:
s221, converting the geographically distributed cloud into the weighted directed graph; the weighted directed graph is represented by:
DG＝(V,E)
wherein: DG is the weighted directed graph; v is the set of vertices; each of the vertices in V characterizes one of the cloud data centers in the geographically distributed cloud; e is a set of edges; each edge in E represents that a direct transmission path exists between two cloud data centers;
s222, constructing a weighted adjacency matrix of the weighted directed graph; the weighted adjacency matrix is expressed by the following formula:
A＝[aij]e×e
wherein: e is the number of vertexes;
s223, constructing to obtain a shortest distance matrix according to a state transfer equation; the shortest distance matrix is denoted as d (e); wherein D (0) ═ a; the state transition equation is calculated as follows:
map[i][j]＝min{map[i][l]+map[l][j],map[i][j]}
wherein: map [ i ] [ j ] is the shortest distance from vertex i to vertex j; l is the breakpoint of the exhaustive i, j;
s224, updating h times recursively, and finally constructing a matrix D (h) by D (h-1); the element D [ i ] [ j ] of the matrix D (h) is the shortest path length from vertex i to vertex j;
s225, calculating to obtain a data block D according to the shortest distance matrix D (h)nTransmitting to cloud data center GkMinimum transmission cost of (c); the calculation method is as follows:
Figure GDA0003118813030000051
wherein: bCMOSnkRepresenting a data block DnTransmitting to cloud data center GkMinimum transmission cost of (c);
Figure GDA0003118813030000052
representing slave cloud data center GnTransmitting Unit data volume to cloud data center GkA minimum transmission bandwidth unit price;
s226, according to the data block DnTransmitting to cloud data center GkThe minimum value of the transmission bandwidth cost of the data placement is calculated and obtained, and the minimum value is calculated according to the following formula:
Figure GDA0003118813030000053
wherein: opt is the minimum value of the transmission bandwidth cost for the data placement.
Preferably, the objective function is expressed by:
Figure GDA0003118813030000061
wherein: target characterizes the Target function; t is tmkCalculated as follows:
Figure GDA0003118813030000062
wherein:
Figure GDA0003118813030000063
as a cloud data center GnAnd data center GkThe data transmission rate therebetween.
Compared with the prior art, the invention has the following advantages:
the load balance of the geographically distributed cloud can be guaranteed while the data transmission time in the data placement process is effectively reduced.
Drawings
FIG. 1 is a general flow diagram of an embodiment of the present invention;
fig. 2 is a schematic flow chart of solving a multisource shortest path of a weighted directed graph by using a Floyd algorithm according to an embodiment of the present invention;
fig. 3 is a schematic flow chart of solving an objective function by using a lagrangian relaxation method to obtain data transmission time for minimizing data placement according to an embodiment of the present invention.
Detailed Description
The present invention is further illustrated by the following examples, which are intended to be purely exemplary and are not intended to limit the scope of the invention, as various equivalent modifications of the invention will occur to those skilled in the art upon reading the present disclosure and fall within the scope of the appended claims.
It should be noted in advance that, a cloud data center executing a job in a geographically distributed cloud environment must contain all data required by the job, that is, data corresponding to the job must be placed in the same cloud data center, and therefore how to perform data placement is a key issue.
The principle of the invention is as follows: the capacity limit and the service performance of each cloud data center in the geographically distributed cloud are considered, and meanwhile, the load balance of each cloud data center in the geographically distributed cloud is guaranteed. Specifically, the method comprises the following steps:
firstly, mathematical modeling is carried out on the data placement problem in the geographically distributed cloud under three limiting conditions of load balance limitation, capacity limitation of a cloud data center and data transmission bandwidth cost limitation.
And then, converting the data transmission cost problem in the geographically distributed cloud into a multi-source shortest path problem of a directed weighted graph, modeling the data transmission broadband cost by adopting a Floyd algorithm, and solving the minimum value of the transmission bandwidth cost.
And finally, converting the data placement objective function from a complex integer programming problem into a linear programming problem, and solving the data placement scheme with the minimum transmission time by a Lagrange relaxation method.
The data placement scheme for solving the minimum transmission time by the Lagrange relaxation method mainly comprises three parts:
the data placement problem in the geographically distributed cloud is modeled into an integer programming problem to minimize data transmission time to optimize delay in job scheduling, and simultaneously, the capacity limit and data transmission cost of each data center are considered, and load balance of the geographically distributed cloud system is maintained.
And secondly, constraining transmission cost in an integer programming model for data placement, in the specific embodiment, converting the geographically distributed cloud into a directed weighted graph, then performing multisource shortest path solving on the directed weighted graph by using a Floyd algorithm, and searching for the minimum data transmission cost.
And finally, converting the integer programming problem into a linear programming problem according to the property of the full single mode matrix, solving the linear programming model by adopting a Lagrange relaxation method, and solving the data placement scheme with the minimum data transmission time when the constraint condition is met.
As shown in fig. 1, the load balancing-based data placement method in the geographically distributed cloud includes the steps of:
s100, acquiring the idle capacity of each cloud data center and the number of servers of each cloud data center; then calculating according to the free capacity of each cloud data center to obtain the capacity limit of the cloud data center; and obtaining load balancing limitation of the geographically distributed cloud according to the number of the servers of each cloud data center.
In this embodiment, the capacity limit of the cloud data center is expressed by formula (1):
Figure GDA0003118813030000081
wherein: ckThe remaining free capacity of the kth cloud data center; m is the number of the jobs to be scheduled; the job to be scheduled is represented as a set { T }1,T2,T3,...,TmWhere T ismThe mth job in the job set to be scheduled; n is the number of blocks of data to be placed; the data to be placed is represented as a set { D }1,D2,D3,...,DnIn which D isnThe nth data block in the data set to be placed is obtained; alpha is alphamnFor selecting variables, the value range is alphamnE {0,1}, when processing operation TmData D of neednTaking 1 when the current value is zero, or taking 0 when the current value is zero; gamma raymkFor selecting variables, the value range is gammamk∈{0,1}，GkFor the kth cloud data center, when the operation TmIs scheduled to cloud data center GkTaking 1 when the current value is zero, or taking 0 when the current value is zero; gnTo contain data D to be placednThe cloud data center of (1);
Figure GDA0003118813030000082
for the slave cloud data center GnTo a data center GkThe amount of data of (a); g is a geographically distributed cloud, represented as a set { G }1,G2,G3,...,Gk}。
In this embodiment, the load balancing constraint is expressed by equation (2):
Figure GDA0003118813030000083
wherein: eta1Is a minimum load balancing parameter, η1≤1；η2Is a maximum load balancing parameter, η2Not less than 1; k is the total number of cloud data centers in the geographically distributed cloud; skThe number of servers of the kth cloud data center; l is the total load of the geographically distributed cloud; l iskIs the load of the kth cloud data center.
And S200, calculating to obtain the cost limit of the data transmission bandwidth in the geographically distributed cloud.
In this specific embodiment, the step of calculating the cost limit of the data transmission bandwidth in the geographically distributed cloud includes:
s210, constructing a weighted directed graph of transmission bandwidth costs for data placement in a geographically distributed cloud.
And S220, solving the multisource shortest path of the weighted directed graph by adopting a Floyd algorithm.
As shown in fig. 2, in this embodiment, the step of solving the multi-source shortest path of the weighted directed graph by using the Floyd algorithm includes the following steps:
s221, converting the geographically distributed cloud into a weighted directed graph; the weighted directed graph is expressed by equation (3):
DG＝(V,E) (3)
wherein: DG is a weighted directed graph; v is the set of vertices; each vertex in V characterizes one cloud data center in the geographically distributed cloud; e is a set of edges; each edge in E characterizes the existence of a direct transmission path between two cloud data centers.
S222, constructing a weighted adjacent matrix of a weighted directed graph; the weighted adjacency matrix is expressed by equation (4):
A＝[aij]e×e (4)
wherein: e is the number of vertices.
S223, constructing to obtain a shortest distance matrix according to a state transfer equation; the shortest distance matrix is denoted as d (e); wherein D (0) ═ a; the state transition equation is calculated as equation (5):
map[i][j]＝min{map[i][l]+map[l][j],map[i][j]} (5)
wherein: map [ i ] [ j ] is the shortest distance from vertex i to vertex j; l is the breakpoint for the exhaustive i, j.
S224, updating h times recursively, and finally constructing a matrix D (h) by D (h-1); the element D [ i ] [ j ] of matrix D (h) is the shortest path length from vertex i to vertex j.
S225, according to the shortest distance matrix D (h), calculating to obtain a data block DnTransmitting to cloud data center GkMinimum transmission cost of (c); the calculation method is as follows (6):
Figure GDA0003118813030000091
wherein: bCMOSnkRepresenting a data block DnTransmitting to cloud data center GkMinimum transmission cost of (c);
Figure GDA0003118813030000092
representing slave cloud data center GnTransmitting Unit data volume to cloud data center GkMinimum transmission bandwidth unit price of
S226, according to the data block DnTransmitting to cloud data center GkThe minimum value of the transmission bandwidth cost of data placement is calculated, and the minimum value is calculated according to the formula (7):
Figure GDA0003118813030000093
wherein: opt is the minimum value of the transmission bandwidth cost for data placement.
And S230, calculating to obtain the cost limit of the data transmission bandwidth in the geographically distributed cloud according to the result of solving the multisource shortest path of the weighted directed graph.
S300, constructing a target function related to data placement of the geographically distributed cloud according to load balance limitation, capacity limitation of a cloud data center and data transmission bandwidth cost limitation; the objective function is to minimize the data transfer time for data placement.
The objective function is expressed by equation (8):
Figure GDA0003118813030000101
wherein: target characterizes the Target function; t is tmkCalculated according to equation (9):
Figure GDA0003118813030000102
wherein:
Figure GDA0003118813030000103
as a cloud data center GnAnd data center GkThe data transmission rate therebetween.
S400, solving the objective function by adopting a Lagrange relaxation method to obtain the data transmission time for placing the minimized data; and outputting the data placement scheme matrix when the data transmission time is minimum.
As shown in fig. 3, in this embodiment, solving the objective function by using a lagrangian relaxation method to obtain the data transmission time for placing the minimized data includes the following steps:
and S410, converting the objective function into a linear programming objective function.
And S420, absorbing the complex constraint condition of the linear programming objective function into the linear programming objective function by adopting a Lagrange relaxation method to obtain the Lagrange relaxation function.
And S430, initializing penalty factors, sub-gradients, step sizes and iteration times of the Lagrange relaxation function.
In this embodiment, two penalty factors are set:
Figure GDA0003118813030000104
and phi; setting the secondary gradient as g; let the step size be ψ.
S440, according to the penalty factor
Figure GDA0003118813030000105
And calculating the minimum value of the Lagrangian relaxation function for the current value of phi; and calculating to obtain a data placement scheme matrix according to the current value of the penalty factor.
In this embodiment, the data placement scheme matrix is set to γ*。
S450, updating the secondary gradient and the step length; the updated value of the sub-gradient is
Figure GDA0003118813030000106
And gφ(ii) a The updated step size has a value of
Figure GDA0003118813030000111
And psiφ(ii) a Then updating the penalty factor according to the updated value of the secondary gradient and the value of the step length
Figure GDA0003118813030000112
And the current value of phi.
S460, judging whether the data placement scheme matrix meets the target of the target function or not, and judging whether the iteration times are equal to an artificial preset iteration time threshold num or not; according to the judgment result, the following operations are carried out:
and if the data placement scheme matrix meets the target of the target function, or the iteration number is equal to an artificially preset iteration number threshold num, outputting the data placement scheme matrix.
Otherwise, return to S440.
In this embodiment, the data placement method may be described by the following pseudo code:
Figure GDA0003118813030000113
Figure GDA0003118813030000121
pseudo code description of the algorithm can be obtained, lines 1 to 10, and the problem of minimum transmission bandwidth cost limitation of data placement in the geographically distributed cloud is converted into the problem of the multisource shortest path of the weighted directed graph; solving the multisource shortest path problem of the weighted directed graph by using a Floyd algorithm in the shortest path algorithm to obtain data transmission bandwidth cost limit; the 11 th line obtains a target function according to load balance limitation, capacity limitation of a cloud data center and data transmission bandwidth cost limitation, and obtains a Lagrange relaxation function through a Lagrange relaxation method; lines 12 to 22 solve the relaxed objective function.
In the foregoing detailed description, various features are grouped together in a single embodiment for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments of the subject matter require more features than are expressly recited in each claim. Rather, as the following claims reflect, invention lies in less than all features of a single disclosed embodiment. Thus, the following claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. To those skilled in the art; various modifications to these embodiments will be readily apparent, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
What has been described above includes examples of one or more embodiments. It is, of course, not possible to describe every conceivable combination of components or methodologies for purposes of describing the aforementioned embodiments, but one of ordinary skill in the art may recognize that many further combinations and permutations of various embodiments are possible. Accordingly, the embodiments described herein are intended to embrace all such alterations, modifications and variations that fall within the scope of the appended claims. Furthermore, to the extent that the term "includes" is used in either the detailed description or the claims, such term is intended to be inclusive in a manner similar to the term "comprising" as "comprising" is interpreted when employed as a transitional word in a claim. Furthermore, any use of the term "or" in the specification of the claims is intended to mean a "non-exclusive or".
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are merely exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.

Claims (5)

1. A data placement method based on load balancing in a geographically distributed cloud is characterized in that: comprises the following steps:
s100, acquiring the idle capacity of each cloud data center and the number of servers of each cloud data center; then calculating according to the free capacity of each cloud data center to obtain the capacity limit of the cloud data center; obtaining load balancing limitation of the geographically distributed cloud according to the number of the servers of each cloud data center;
s200, calculating to obtain data transmission bandwidth cost limit in the geographically distributed cloud;
the calculating to obtain the data transmission bandwidth cost limit in the geographically distributed cloud comprises the following steps:
s210, constructing a weighted directed graph of transmission bandwidth cost of data placement in the geographically distributed cloud;
s220, solving the multisource shortest path of the weighted directed graph by adopting a Floyd algorithm;
s230, calculating according to the result of solving the multisource shortest path of the weighted directed graph to obtain the cost limit of the data transmission bandwidth in the geographically distributed cloud;
s300, constructing an objective function related to data placement of the geographically distributed cloud according to the load balancing limit, the capacity limit of the cloud data center and the data transmission bandwidth cost limit; the objective function is to minimize the data transfer time for the data placement;
s400, solving the objective function by adopting a Lagrange relaxation method to obtain the minimized data transmission time for placing the data; outputting a data placement scheme matrix when the data transmission time is minimum;
the solving of the objective function by using a Lagrange relaxation method to obtain the minimized data transmission time for the data placement comprises the following steps:
s410, converting the target function into a linear programming target function;
s420, absorbing the complex constraint condition of the linear programming objective function into the linear programming objective function by adopting a Lagrange relaxation method to obtain a Lagrange relaxation function;
s430, initializing penalty factors, sub-gradients, step sizes and iteration times of the Lagrange relaxation function;
s440, calculating the minimum value of the Lagrangian relaxation function according to the current value of the penalty factor; calculating to obtain the data placement scheme matrix according to the current value of the penalty factor;
s450, updating the secondary gradient and the step length; then updating the penalty factor according to the updated value of the secondary gradient and the updated value of the step length;
s460, judging whether the data placement scheme matrix meets the target of the target function or not, and judging whether the iteration times are equal to an artificially preset iteration time threshold or not; according to the judgment result, the following operations are carried out:
if the data placement scheme matrix meets the target of the objective function, or the iteration times are equal to an artificially preset iteration time threshold value, outputting the data placement scheme matrix;
otherwise, return to S440.
2. The method for load balancing-based data placement in geographically distributed clouds of claim 1, wherein: the capacity limit of the cloud data center is expressed by the following formula:
Figure FDA0003118813020000021
wherein: ckThe remaining free capacity of the kth cloud data center; m is the number of the jobs to be scheduled; the job to be scheduled is represented as a set { T }1,T2,T3,...,TmWhere T ismThe mth job in the job set to be scheduled; n is the number of blocks of data to be placed; the data to be placed is represented as a set { D }1,D2,D3,...,DnIn which D isnThe nth data block in the data set to be placed is obtained; alpha is alphamnFor selecting variables, the value range is alphamnE {0,1}, when processing operation TmData D of neednTaking 1 when the current value is zero, or taking 0 when the current value is zero; gamma raymkFor selecting variables, the value range is gammamk∈{0,1}，GkFor the kth cloud data center, when the operation TmIs scheduled to cloud data center GkTaking 1 when the current value is zero, or taking 0 when the current value is zero; gnTo contain the data D to be placednThe cloud data center of (1);
Figure FDA0003118813020000022
for the slave cloud data center GnTo a data center GkThe amount of data of (a); g is the geographically distributed cloud, represented as a set { G }1,G2,G3,...,Gk}。
3. The method for load balancing-based data placement in geographically distributed clouds of claim 2, wherein: the load balancing constraint is expressed as:
Figure FDA0003118813020000031
wherein: eta1Is a minimum load balancing parameter, η1≤1；η2Is a maximum load balancing parameter, η2Not less than 1; k is the total number of cloud data centers in the geographically distributed cloud; skThe number of servers of the kth cloud data center; l is the total load of the geographically distributed cloud; l iskIs the load of the kth cloud data center.
4. The method for load balancing-based data placement in geographically distributed clouds of claim 3, wherein: the method for solving the multisource shortest path of the weighted directed graph by adopting the Floyd algorithm comprises the following steps:
s221, converting the geographically distributed cloud into the weighted directed graph; the weighted directed graph is represented by:
DG＝(V,E)
wherein: DG is the weighted directed graph; v is the set of vertices; each of the vertices in V characterizes one of the cloud data centers in the geographically distributed cloud; e is a set of edges; each edge in E represents that a direct transmission path exists between two cloud data centers;
s222, constructing a weighted adjacency matrix of the weighted directed graph; the weighted adjacency matrix is expressed by the following formula:
A＝[aij]e×e
wherein: e is the number of vertexes;
s223, constructing to obtain a shortest distance matrix according to a state transfer equation; the shortest distance matrix is denoted as d (e); wherein D (0) ═ a; the state transition equation is calculated as follows:
map[i][j]＝min{map[i][l]+map[l][j],map[i][j]}
wherein: map [ i ] [ j ] is the shortest distance from vertex i to vertex j; l is the breakpoint of the exhaustive i, j;
s224, updating h times recursively, and finally constructing a matrix D (h) by D (h-1); the element D [ i ] [ j ] of the matrix D (h) is the shortest path length from vertex i to vertex j;
s225, according to the shortest distance matrix D (h)Calculating to obtain data block DnTransmitting to cloud data center GkMinimum transmission cost of (c); the calculation method is as follows:
Figure FDA0003118813020000041
wherein: bCMOSnkRepresenting a data block DnTransmitting to cloud data center GkMinimum transmission cost of (c); a isGn,kRepresenting slave cloud data center GnTransmitting Unit data volume to cloud data center GkA minimum transmission bandwidth unit price;
s226, according to the data block DnTransmitting to cloud data center GkThe minimum value of the transmission bandwidth cost of the data placement is calculated and obtained, and the minimum value is calculated according to the following formula:
Figure FDA0003118813020000042
wherein: opt is the minimum value of the transmission bandwidth cost for the data placement.
5. The method for load balancing-based data placement in geographically distributed clouds of claim 4, wherein: the objective function is expressed as follows:
Figure FDA0003118813020000043
wherein: target characterizes the Target function; t is tmkCalculated as follows:
Figure FDA0003118813020000044
wherein:
Figure FDA0003118813020000045
as a cloud data center GnAnd data center GkThe data transmission rate therebetween.
CN202010906169.9A 2020-09-01 2020-09-01 Load balancing-based data placement method in geographically distributed cloud Active CN112073496B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010906169.9A CN112073496B (en) 2020-09-01 2020-09-01 Load balancing-based data placement method in geographically distributed cloud

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010906169.9A CN112073496B (en) 2020-09-01 2020-09-01 Load balancing-based data placement method in geographically distributed cloud

Publications (2)

Publication Number Publication Date
CN112073496A CN112073496A (en) 2020-12-11
CN112073496B true CN112073496B (en) 2021-09-10

Family

ID=73664827

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010906169.9A Active CN112073496B (en) 2020-09-01 2020-09-01 Load balancing-based data placement method in geographically distributed cloud

Country Status (1)

Country Link
CN (1) CN112073496B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114118437B (en) * 2021-09-30 2023-04-18 电子科技大学 Model updating synchronization method for distributed machine learning in micro cloud

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102497394A (en) * 2011-11-28 2012-06-13 中国科学院研究生院 Duplicate file placement method in content distribution network based on optimized model
CN102984280A (en) * 2012-12-18 2013-03-20 北京工业大学 Data backup system and method for social cloud storage network application
CN108920282A (en) * 2018-08-03 2018-11-30 北京科技大学 A kind of copy of content generation, placement and the update method of holding load equilibrium
CN109889573A (en) * 2019-01-14 2019-06-14 武汉理工大学 Based on the Replica placement method of NGSA multiple target in mixed cloud

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8515915B2 (en) * 2010-09-24 2013-08-20 Hitachi Data Systems Corporation System and method for enhancing availability of a distributed object storage system during a partial database outage
US10198447B2 (en) * 2015-09-14 2019-02-05 Komprise Inc. Electronic file migration system and various methods of transparent data migration management

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102497394A (en) * 2011-11-28 2012-06-13 中国科学院研究生院 Duplicate file placement method in content distribution network based on optimized model
CN102984280A (en) * 2012-12-18 2013-03-20 北京工业大学 Data backup system and method for social cloud storage network application
CN108920282A (en) * 2018-08-03 2018-11-30 北京科技大学 A kind of copy of content generation, placement and the update method of holding load equilibrium
CN109889573A (en) * 2019-01-14 2019-06-14 武汉理工大学 Based on the Replica placement method of NGSA multiple target in mixed cloud

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
The Real-time Scheduling Strategy Based on Traffic;Jing Zhang;《IEEE》;20161230;全文 *
基于人工蜂群算法的存储负载副本放置均衡算法;郭佳;《北京交通大学学报》;20200707;全文 *

Also Published As

Publication number Publication date
CN112073496A (en) 2020-12-11

Similar Documents

Publication Publication Date Title
US11562213B2 (en) Methods and arrangements to manage memory in cascaded neural networks
US8972983B2 (en) Efficient execution of jobs in a shared pool of resources
US9277003B2 (en) Automated cloud workload management in a map-reduce environment
US20210092069A1 (en) Accelerating multi-node performance of machine learning workloads
US9015708B2 (en) System for improving the performance of high performance computing applications on cloud using integrated load balancing
US8959135B2 (en) Data structure for tiling and packetizing a sparse matrix
CN106933669B (en) Apparatus and method for data processing
US11010313B2 (en) Method, apparatus, and system for an architecture for machine learning acceleration
US9229777B2 (en) Dynamically relocating workloads in a networked computing environment
US20140372609A1 (en) Dynamically expanding computing resources in a networked computing environment
WO2023051505A1 (en) Job solving method and apparatus
CN111752691B (en) Method, device, equipment and storage medium for sorting AI (advanced technology attachment) calculation graphs
CN113449839A (en) Distributed training method, gradient communication device and computing equipment
CN112073496B (en) Load balancing-based data placement method in geographically distributed cloud
CN109416688B (en) Method and system for flexible high performance structured data processing
CN113076224B (en) Data backup method, data backup system, electronic device and readable storage medium
CN113448714B (en) Computing resource control system based on cloud platform
US8627039B2 (en) Effective memory clustering to minimize page fault and optimize memory utilization
CN114399035A (en) Method for transferring data, direct memory access device and computer system
CN104679575A (en) Control system and control method for input and output flow
US11954518B2 (en) User-defined metered priority queues
CN107797870A (en) A kind of cloud computing data resource dispatching method
Mikram et al. Server consolidation algorithms for cloud computing: taxonomies and systematic analysis of literature
CN116010020A (en) Container pool management
US11886981B2 (en) Inter-processor data transfer in a machine learning accelerator, using statically scheduled instructions

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant