CN106095572B - distributed scheduling system and method for big data processing - Google Patents
distributed scheduling system and method for big data processing Download PDFInfo
- Publication number
- CN106095572B CN106095572B CN201610404437.0A CN201610404437A CN106095572B CN 106095572 B CN106095572 B CN 106095572B CN 201610404437 A CN201610404437 A CN 201610404437A CN 106095572 B CN106095572 B CN 106095572B
- Authority
- CN
- China
- Prior art keywords
- node
- computing node
- task
- computing
- load rate
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5044—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering hardware capabilities
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/501—Performance criteria
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/5022—Workload threshold
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer And Data Communications (AREA)
Abstract
the invention provides a distributed scheduling system and a method for big data processing, wherein the system comprises: the system comprises a client, a plurality of dispatching centers and a plurality of computing nodes; the method comprises the steps that a dispatching center receives and stores node information sent by each computing node, wherein the node information comprises node position information and performance characteristic information; the client acquires node information of each computing node from the scheduling center, respectively calculates a load rate corresponding to each computing node according to the number of tasks currently allocated to each computing node and performance characteristic information of each computing node, and allocates a task to be scheduled to the computing node with the minimum load rate according to the load rate corresponding to each computing node; and the computing node sends node information to each scheduling center and receives and processes the task to be scheduled sent by the client. The invention carries out scheduling by the client, and the task data does not pass through the scheduling center, thereby avoiding the bottleneck limit of the scheduling center. And the performance of the computing nodes is considered during load balancing, and uniform scheduling can be realized.
Description
Technical Field
the invention relates to the technical field of computers, in particular to a distributed scheduling system and method for big data processing.
background
With the rapid development of information technology, data generated by industrial application is explosively increased, and large data of hundreds of tb (terabyte) scales and even tens of to hundreds of pb (petabbyte) scales are generated, and the generated large data requires a computer and an information system to have higher processing capacity.
Currently, related technologies provide a big data processing method, where a network architecture for big data processing is shown in fig. 1, a client sends task data to a scheduling center, and the scheduling center schedules the task data to a server with the smallest number of tasks to be processed according to the number of tasks being processed by each server in a server cluster.
The task data must be scheduled by the scheduling center, and the scheduling capability of the scheduling center is limited, so that the processing capability of the whole data processing architecture is limited. Even if the number of the tasks being processed is the same, the total calculated amount may be different, and the load is different for the servers with different calculating capacities, so that scheduling based on the number of the tasks being processed is likely to cause uneven scheduling, and load balance cannot be effectively guaranteed.
disclosure of Invention
In view of this, an object of the embodiments of the present invention is to provide a distributed scheduling system and method for big data processing, where a client performs scheduling, task data does not pass through a scheduling center, and bottleneck limitation of the scheduling center is avoided. And the performance of the computing nodes is considered during load balancing, and uniform scheduling can be realized.
in a first aspect, an embodiment of the present invention provides a distributed scheduling system for big data processing, where the system includes: the system comprises a client, a plurality of dispatching centers and a plurality of computing nodes;
The dispatching center is used for receiving and storing node information sent by each computing node, wherein the node information comprises node position information and performance characteristic information;
The client is used for acquiring node information of each computing node from the scheduling center, respectively calculating the load rate corresponding to each computing node according to the number of tasks currently allocated to each computing node and the performance characteristic information of each computing node, and allocating the tasks to be scheduled to the computing nodes with the minimum load rate according to the load rate corresponding to each computing node;
And the computing node is used for sending node information to each scheduling center and receiving and processing the task to be scheduled sent by the client.
With reference to the first aspect, an embodiment of the present invention provides a first possible implementation manner of the first aspect, where the client is configured to establish a communication connection with a scheduling center through an interface provided by a software development kit SDK, and obtain node information of each computing node from the scheduling center through the established communication connection.
with reference to the first aspect, an embodiment of the present invention provides a second possible implementation manner of the first aspect, where the client is configured to create a corresponding virtual node for each computing node through a software development kit SDK; storing node information of a computing node corresponding to the virtual node in the created virtual node, and constructing a waiting queue and a work queue; and establishing communication connection between the created virtual node and the corresponding computing node through the interface provided by the SDK.
with reference to the first aspect, an embodiment of the present invention provides a third possible implementation manner of the first aspect, where,
The client is used for counting the number of tasks in a waiting queue and a work queue corresponding to a first computing node, determining the counted number of tasks as the number of tasks currently allocated to the first computing node, and the first computing node is any one of the computing nodes; acquiring the number of CPU cores and dominant frequency of a central processing unit of the first computing node from the performance characteristic information of the first computing node; calculating the calculation capacity value of the first calculation node according to the CPU core number and the main frequency; and calculating the load rate corresponding to the first calculation node according to the number of tasks currently distributed to the first calculation node and the calculation capacity value.
With reference to the second possible implementation manner of the first aspect, an embodiment of the present invention provides a fourth possible implementation manner of the first aspect, where the client is configured to sort, according to a preset order, virtual nodes corresponding to each computing node according to a load rate corresponding to each computing node; selecting a virtual node with the minimum load rate from the sorted virtual nodes; and inserting the task to be scheduled into the tail of the waiting queue of the virtual node with the minimum load rate through a scheduler.
With reference to the fourth possible implementation manner of the first aspect, an embodiment of the present invention provides a fifth possible implementation manner of the first aspect, where,
The client is used for sending the task to be scheduled to the corresponding computing node through a task sender through the communication connection between the virtual node with the minimum load rate and the corresponding computing node; deleting the task to be scheduled from the waiting queue of the virtual node with the minimum load rate; and inserting the task to be scheduled into the tail of the work queue of the virtual node with the minimum load rate.
With reference to the fifth possible implementation manner of the first aspect, an embodiment of the present invention provides the sixth possible implementation manner of the first aspect, where the client is configured to receive a processing result, corresponding to the task to be scheduled, returned by the computing node with the smallest load rate; and deleting the task to be scheduled from the work queue of the virtual node with the minimum load rate.
With reference to the second possible implementation manner of the first aspect, an embodiment of the present invention provides a seventh possible implementation manner of the first aspect, where the client is configured to set a timer in a created virtual node; respectively recording the processing duration of each task in the work queue of the virtual node through the timer; and when detecting the task with the processing time length exceeding the preset time length, deleting the task with the processing time length exceeding the preset time length from the work queue, and displaying overtime error prompt information.
In a second aspect, an embodiment of the present invention provides a distributed scheduling method for big data processing, where the method includes:
acquiring node information of each computing node from a scheduling center, wherein the node information comprises node position information and performance characteristic information;
respectively calculating the load rate corresponding to each computing node according to the number of tasks currently distributed to each computing node and the performance characteristic information of each computing node;
and distributing the task to be scheduled to the computing node with the minimum load rate according to the load rate corresponding to each computing node.
With reference to the second aspect, an embodiment of the present invention provides a first possible implementation manner of the second aspect, where the calculating, according to the number of tasks currently allocated to each computing node and the performance characteristic information of each computing node, a load rate corresponding to each computing node respectively includes:
counting the number of tasks in a waiting queue and a work queue corresponding to a first computing node, and determining the counted number of tasks as the number of tasks currently allocated to the first computing node, wherein the first computing node is any one of the computing nodes;
acquiring the number of CPU cores and dominant frequency of a central processing unit of the first computing node from the performance characteristic information of the first computing node;
Calculating the calculation capacity value of the first calculation node according to the CPU core number and the main frequency;
And calculating the load rate corresponding to the first calculation node according to the number of tasks currently distributed to the first calculation node and the calculation capacity value.
With reference to the second aspect, an embodiment of the present invention provides a second possible implementation manner of the second aspect, where the obtaining node information of each computing node from a scheduling center includes:
establishing communication connection with a dispatching center through an interface provided by a Software Development Kit (SDK);
And acquiring the node information of each computing node from the dispatching center through the established communication connection.
With reference to the second aspect, an embodiment of the present invention provides a third possible implementation manner of the second aspect, where after the obtaining node information of each computing node from the scheduling center, the method further includes:
Respectively creating corresponding virtual nodes for each computing node through a Software Development Kit (SDK);
storing node information of a computing node corresponding to the virtual node in the created virtual node, and constructing a waiting queue and a work queue;
and establishing communication connection between the created virtual node and the corresponding computing node through the interface provided by the SDK.
With reference to the third possible implementation manner of the second aspect, an embodiment of the present invention provides a fourth possible implementation manner of the second aspect, where the allocating a task to be scheduled to a computing node with a smallest load rate according to the load rate corresponding to each computing node includes:
According to the load rate corresponding to each computing node, sequencing the virtual nodes corresponding to each computing node according to a preset sequence;
selecting a virtual node with the minimum load rate from the sorted virtual nodes;
And inserting the task to be scheduled into the tail of the waiting queue of the virtual node with the minimum load rate through the scheduler.
With reference to the fourth possible implementation manner of the second aspect, an embodiment of the present invention provides a fifth possible implementation manner of the second aspect, where the method further includes:
sending the task to be scheduled to the corresponding computing node through the communication connection between the virtual node with the minimum load rate and the corresponding computing node by a task sender;
Deleting the task to be scheduled from a waiting queue of the virtual node with the minimum load rate;
and inserting the task to be scheduled into the tail of the work queue in the virtual node with the minimum load rate.
with reference to the fifth possible implementation manner of the second aspect, an embodiment of the present invention provides a sixth possible implementation manner of the second aspect, where the method further includes:
Receiving a processing result corresponding to the task to be scheduled, which is returned by the computing node corresponding to the virtual node with the minimum load rate;
And deleting the task to be scheduled from the work queue of the virtual node with the minimum load rate.
With reference to the third possible implementation manner of the second aspect, an embodiment of the present invention provides a seventh possible implementation manner of the second aspect, where the method further includes:
Setting a timer in the created virtual node;
Respectively recording the processing duration of each task in the work queue of the virtual node through the timer;
and when detecting the task with the processing time length exceeding the preset time length, deleting the task with the processing time length exceeding the preset time length from the work queue, and displaying overtime error prompt information.
in the system and method provided by the embodiment of the invention, the system comprises: the system comprises a client, a plurality of dispatching centers and a plurality of computing nodes; the method comprises the steps that a dispatching center receives and stores node information sent by each computing node, wherein the node information comprises node position information and performance characteristic information; the client acquires node information of each computing node from the scheduling center, respectively calculates a load rate corresponding to each computing node according to the number of tasks currently allocated to each computing node and performance characteristic information of each computing node, and allocates a task to be scheduled to the computing node with the minimum load rate according to the load rate corresponding to each computing node; and the computing node sends node information to each scheduling center and receives and processes the task to be scheduled sent by the client. The invention carries out scheduling by the client, and the task data does not pass through the scheduling center, thereby avoiding the bottleneck limit of the scheduling center. And the performance of the computing nodes is considered during load balancing, and uniform scheduling can be realized.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
in order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.
FIG. 1 is a schematic diagram of a large data processing network architecture provided in the prior art;
fig. 2 is a schematic structural diagram illustrating a large data processing distributed scheduling system according to embodiment 1 of the present invention;
Fig. 3 is a schematic structural diagram illustrating another distributed scheduling system for big data processing according to embodiment 1 of the present invention;
Fig. 4 is a schematic diagram illustrating a process of scheduling by a scheduler in cooperation with a virtual node according to embodiment 1 of the present invention;
Fig. 5 is a flowchart illustrating a distributed scheduling method for big data processing according to embodiment 2 of the present invention.
Detailed Description
in order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present invention without making any creative effort, shall fall within the protection scope of the present invention.
considering that task data in the related art must be scheduled through a scheduling center, the scheduling capability of the scheduling center is limited, which limits the processing capability of the whole data processing architecture. Even if the number of the tasks being processed is the same, the total calculated amount may be different, and the load is different for the servers with different calculating capacities, so that scheduling based on the number of the tasks being processed is likely to cause uneven scheduling, and load balance cannot be effectively guaranteed. Based on this, embodiments of the present invention provide a distributed scheduling system and method for big data processing, which are described below by way of embodiments.
example 1
Referring to fig. 2, an embodiment of the present invention provides a distributed scheduling system for big data processing, where the system includes: client 1, a plurality of dispatch centers 2, and a plurality of compute nodes 3. In fig. 2, the scheduling center 2 is simply referred to as center 2.
Wherein, a plurality of scheduling centers 2 adopt a copy mechanism, that is, each scheduling center 2 in the plurality of scheduling centers 2 is a backup for each other. When the distributed scheduling system is used for processing big data, each scheduling center 2 is in communication connection with the plurality of computing nodes 3, but at most a preset number of scheduling centers 2 are in communication connection with the client 1 at the same time, and the preset number is greater than or equal to 1 and smaller than the total number of the scheduling centers 2. When a certain scheduling center 2 connected with the client 1 has a fault, the communication connection between the faulty scheduling center 2 and the client 1 can be disconnected, then one scheduling center 2 is randomly selected from the scheduling centers 2 which are not currently connected with the client 1, and the communication connection between the selected scheduling center 2 and the client 1 is established. Therefore, the problem that the whole distributed scheduling system cannot work when a single scheduling center 2 is abnormal can be avoided.
In the embodiment of the invention, the node information is registered with the dispatching center 2 by adopting the mode that the computing node 3 actively registers. Specifically, the computing node 3 is configured to send node information to each of the plurality of scheduling centers 2, where the node information includes node location information and performance characteristic information of the computing node 3. The node location information includes an IP (Internet Protocol, Protocol for interconnection between networks) address and port number of the computing node 3, and the like. The performance characteristic information is used to indicate the computing capability of the computing node 3, and includes the number of CPU (Central Processing Unit) cores, the number of threads, the dominant frequency, the memory capacity, the network bandwidth, and the like of the computing node 3.
Each of the plurality of scheduling centers 2 is configured to receive the node information sent by each of the computing nodes 3, and store the node information of each of the computing nodes 3.
in the embodiment of the present invention, an SDK (Software Development Kit) is installed in the client 1, the SDK provides an interface for the client 1, and the client 1 establishes communication connection with the scheduling center 2 and the computing node 3 through the interface provided by the SDK.
When the client 1 has a task to be scheduled to be processed, the client 1 first establishes a communication connection with one scheduling center 2 of the plurality of scheduling centers 2 through an interface provided by the SDK. Then, the client 1 is configured to acquire the node information of each computing node 3 from the scheduling center 2 through the established communication connection, and store the node information of each computing node 3.
As shown in fig. 3, the client 1 is further configured to create a corresponding virtual node for each computing node 3 through the SDK; storing node information of a computing node 3 corresponding to the virtual node in the created virtual node, and constructing a waiting queue and a work queue; the communication connection between the created virtual node and the corresponding computing node 3 is established through the interface provided by the SDK.
in the embodiment of the present invention, as shown in fig. 3, a scheduler is further disposed in an SDK installed in a client 1, a task to be scheduled, which needs to be processed by the client 1, is scheduled to a virtual node by the scheduler and then sent to a computing node 3 corresponding to the virtual node by the virtual node, after the processing by the computing node 3 is completed, a processing result corresponding to the task to be scheduled is returned to the corresponding virtual node, and the virtual node uploads the processing result to a processor of the client 1. Therefore, the client 1 is adopted to carry out task scheduling, task data do not need to pass through the scheduling center 2, and the bottleneck problem caused by the limited processing capacity of the scheduling center 2 is avoided.
when the client 1 schedules tasks, the client 1 is configured to calculate a load rate corresponding to each computing node 3 according to the number of tasks currently allocated to each computing node 3 and the performance characteristic information of each computing node 3, and allocate the task to be scheduled to the computing node 3 with the smallest load rate according to the load rate corresponding to each computing node 3.
In the embodiment of the present invention, the tasks that the client 1 has been allocated to the computing node 3 but not yet sent to the computing node 3 are all stored in the waiting queue of the virtual node corresponding to the computing node 3, and when the client 1 sends a task to the computing node 3, the task is deleted from the waiting queue, and the task is inserted into the work queue of the virtual node corresponding to the computing node 3. Therefore, the number of tasks currently allocated to the computing node 3 is the total number of tasks in the waiting queue and the work queue of the virtual node corresponding to the computing node 3.
When the client 1 calculates the load rate of each computing node 3, the client 1 is configured to count task numbers included in a waiting queue and a work queue corresponding to the first computing node 3, determine the counted task numbers as the task numbers currently allocated to the first computing node 3, and the first computing node 3 is any one computing node 3 in each computing node 3; acquiring the CPU core number and the dominant frequency of the first computing node 3 from the performance characteristic information of the first computing node 3; calculating the calculation capacity value of the first calculation node 3 according to the CPU core number and the dominant frequency; and calculating the load rate corresponding to the first computing node 3 according to the number of tasks currently distributed to the first computing node 3 and the computing capability value.
The computing capability value of the first computing node 3 is the product of the CPU core number of the first computing node 3 and the dominant frequency. In the embodiment of the present invention, if the CPU of the first computing node 3 is a hyper-threaded CPU, the computing capability value of the first computing node 3 is a product of a preset coefficient, the number of CPU cores of the first computing node 3, and the dominant frequency. The preset coefficient may be 1.25 or 1.5, and the specific value of the preset coefficient is not specifically limited in the embodiment of the present invention, and the value of the preset coefficient may be customized according to actual needs in actual applications. In addition, the above-mentioned CPU core number is a physical core number of the first computing node 3, not a thread number.
after the calculation capability value of the first calculation node 3 is obtained in the above manner, the ratio between the number of tasks currently allocated to the first calculation node 3 and the calculation capability value is calculated, and the calculated ratio is determined as the load rate corresponding to the first calculation node 3. For each of the other computing nodes 3 in the respective scheduling system, the load rate corresponding to each of the other computing nodes 3 is calculated as described above for the first computing node 3.
Since the number of tasks allocated to the computing node 3 changes from moment to moment, the load rate corresponding to the computing node 3 also changes from moment to moment, and in the embodiment of the present invention, the client 1 calculates the load rate corresponding to each computing node 3 in real time in the manner described above.
In the embodiment of the invention, when the load rate is calculated, the task amount distributed to the computing nodes 3 and the computing capacity of the computing nodes 3 are comprehensively considered, the problem of uneven scheduling caused by different computing capacities of the computing nodes 3 is avoided, and the load balance can be effectively realized.
After the load rate corresponding to each computing node 3 is calculated in the above manner, the client 1 is configured to sort the virtual nodes corresponding to each computing node 3 according to a preset sequence and according to the load rate corresponding to each computing node 3; selecting a virtual node with the minimum load rate from the sorted virtual nodes; and inserting the task to be scheduled into the tail of the waiting queue of the virtual node with the minimum load rate through the scheduler.
The above-mentioned sorting operation of the virtual nodes is performed in the scheduler, and the preset order may be an order from a load rate to a load rate, or an order from a load rate to a load rate.
The processor of the client 1 submits to the task to be scheduled of the SDK, and the scheduler allocates the task to be scheduled to the virtual node with the smallest load rate according to the above-mentioned manner, and inserts the task to be scheduled into the tail of the waiting queue of the virtual node with the smallest load rate.
in the embodiment of the present invention, the client 1 is further configured to send the task to be scheduled to the corresponding computing node 3 through the communication connection between the virtual node with the smallest load rate and the corresponding computing node 3 by the task sender; deleting the task to be scheduled from the waiting queue of the virtual node with the minimum load rate; and inserting the task to be scheduled into the tail of the work queue of the virtual node with the minimum load rate.
after the client 1 sends the task to be scheduled to the computing node 3 with the minimum load rate through the task sender, the computing node 3 is used for receiving and processing the task to be scheduled sent by the client 1. And after the computing node 3 finishes processing the task to be scheduled, sending a processing result corresponding to the task to be scheduled to the client 1.
At this time, the client 1 is configured to receive a processing result corresponding to the task to be scheduled, which is returned by the computing node 3 with the smallest load rate; and deleting the task to be scheduled from the work queue of the virtual node with the minimum load rate.
In order to facilitate understanding of the scheduling process of the scheduler and the virtual node, the following description will take a single computing node 3 and a virtual node corresponding to the computing node 3 as an example, with reference to the drawings. As shown in fig. 4, a diagram in fig. 4 represents that the scheduler inserts the tasks 12 submitted by the processor of the client 1 into the tail of the waiting queue of the virtual node with the smallest load rate. Fig. 4B shows that when the task transmitter transmits the task 8 to the computing node 3 with the smallest load rate, the task 8 is deleted from the waiting queue and the task 8 is inserted into the tail of the work queue. Fig. 4C shows that when the computing node 3 finishes processing the task 7 and returns the processing result corresponding to the task 7 to the result receiver, the task 7 is deleted from the work queue.
In the embodiment of the present invention, the task transmitter in the client 1 has an integrated transmission function, that is, a plurality of tasks can be simultaneously transmitted to the computing node 3 through the task transmitter, so as to improve the efficiency of task transmission.
in addition, because the problem that data packets are lost in the process of sending the task to be scheduled may exist, the computing node 3 cannot complete the processing of the task to be scheduled, and thus the client 1 cannot receive the processing result corresponding to the task to be scheduled, which is returned by the computing node 3, so that the task to be scheduled is always cached in the work queue, and a dead task is formed. In order to avoid dead tasks in the work queue, the embodiment of the invention times the tasks cached in the work queue and deletes overtime tasks. Specifically, the client 1 is configured to set a timer in the created virtual node; respectively recording the processing duration of each task in the work queue of the virtual node through a timer; and when detecting the task with the processing time length exceeding the preset time length, deleting the task with the processing time length exceeding the preset time length from the work queue, and displaying the overtime error prompt message.
The timer mechanism can avoid the problem of task loss caused by packet loss of network transmission data and the problem that computing resources cannot be fully utilized due to the fact that dead tasks exist in a work queue.
in the embodiment of the invention, the load rate comprehensively considers the number of tasks currently allocated to the computing node 3 and the computing capacity of the computing node 3, the problem of uniform scheduling of matching the task amount and the computing capacity is effectively solved, and meanwhile, the load state of the computing node 3 can be timely sensed through a processing result feedback mechanism of the computing node 3, so that the computing resources are fully utilized. In the scheme, threshold limitation can be set on the load rate of the computing node 3, when the load rate of the computing node 3 exceeds the threshold, the task is not dispatched to the computing node 3 any more, and the problem of insufficient memory caused by overlarge buffer amount of the distributed task is avoided.
In the embodiment of the present invention, although the load rates of the virtual nodes are dynamically changed, there are cases where the load rates of some of the computing nodes 3 are the same or similar. For example, when the distributed scheduling system is just started, all the clients 1 consider the load rates of all the computing nodes 3 to be 0, at this time, the virtual nodes in all the clients 1 are sorted identically, and if tasks to be scheduled are issued at the same time, the tasks are issued to the same computing node 3 by all the clients 1 at the same time. In order to avoid the problem, the scheduler performs task scheduling in a random mode when the load rates are the same or similar.
the embodiment of the invention can also effectively solve the problem of unbalanced simultaneous scheduling of the multiple clients 1, when the scheduling is just started, because the load rates of all the virtual nodes are all 0, the clients 1 can send tasks to all the computing nodes 3, even if the problem of imbalance exists at the moment, the load rate of the virtual node corresponding to the computing node 3 is increased due to the fact that the processing result returning speed of the busy computing node 3 is reduced, the number of tasks scheduled to the computing node 3 by the clients 1 is reduced, and the task scheduling of all the computing nodes 3 is gradually balanced. Even if the scheduling algorithms are different, the scheduling method and the scheduling device can effectively relieve the problem of uneven scheduling.
The distributed scheduling system provided by the embodiment of the invention has high timeliness of task scheduling, can quickly return the processing result corresponding to the task after the task is issued, and is also suitable for interactive application with high real-time requirement.
In an embodiment of the present invention, a distributed scheduling system includes: the system comprises a client, a plurality of dispatching centers and a plurality of computing nodes; the method comprises the steps that a dispatching center receives and stores node information sent by each computing node, wherein the node information comprises node position information and performance characteristic information; the client acquires node information of each computing node from the scheduling center, respectively calculates a load rate corresponding to each computing node according to the number of tasks currently allocated to each computing node and performance characteristic information of each computing node, and allocates a task to be scheduled to the computing node with the minimum load rate according to the load rate corresponding to each computing node; and the computing node sends node information to each scheduling center and receives and processes the task to be scheduled sent by the client. The invention carries out scheduling by the client, and the task data does not pass through the scheduling center, thereby avoiding the bottleneck limit of the scheduling center. And the performance of the computing nodes is considered during load balancing, and uniform scheduling can be realized.
Example 2
Referring to fig. 5, an embodiment of the present invention provides a distributed scheduling method for big data processing, where an execution subject of the method may be a client in the distributed scheduling system provided in embodiment 1. The method specifically comprises the following steps:
Step 201: and acquiring node information of each computing node from the scheduling center, wherein the node information comprises node position information and performance characteristic information.
The node location information includes an IP address, a port number, and the like of the computing node. The performance characteristic information is used for representing the computing capability of the computing node, and comprises the CPU core number, the thread number, the main frequency, the memory capacity, the network bandwidth and the like of the computing node.
In a distributed scheduling system for big data processing, all computing nodes register their own node information in each scheduling center. The client is provided with the SDK, the SDK provides an interface for the client, and the client establishes communication connection with the dispatching center and the computing node through the interface provided by the SDK.
the process of the client acquiring the node information of each computing node specifically includes: establishing communication connection with a dispatching center through an interface provided by the SDK; and acquiring the node information of each computing node from the dispatching center through the established communication connection. And after the client acquires the node information of each computing node, the node information of each computing node is also stored.
In the distributed scheduling system, a scheduling center adopting a copy mechanism is provided, that is, a plurality of scheduling centers are provided, and each of the plurality of scheduling centers is a backup for each other. The client establishes communication connection with one of the dispatching centers through an interface provided by the SDK, and obtains node information of each computing node from the dispatching center.
In the embodiment of the invention, the client also creates a corresponding virtual node for each computing node through the SDK; storing node information of a computing node corresponding to the virtual node in the created virtual node, and constructing a waiting queue and a work queue; and establishing communication connection between the created virtual node and the corresponding computing node through an interface provided by the SDK.
Step 202: respectively calculating the load rate corresponding to each computing node according to the number of tasks currently distributed to each computing node and the performance characteristic information of each computing node;
In the embodiment of the invention, the tasks which are distributed to the computing nodes by the client and are not sent to the computing nodes are all stored in the waiting queue of the virtual node corresponding to the computing nodes, when the client sends the tasks to the computing nodes, the tasks are deleted from the waiting queue, and the tasks are inserted into the work queue of the virtual node corresponding to the computing nodes. Therefore, the number of tasks currently allocated to the computing node is the total number of tasks in the waiting queue and the work queue of the virtual node corresponding to the computing node.
In the embodiment of the present invention, a scheduler is further disposed in the SDK installed in the client, and the client calculates the load rate corresponding to each computing node through the scheduler, where the specific calculation process includes:
Counting the number of tasks in a waiting queue and a work queue corresponding to a first computing node, and determining the counted number of tasks as the number of tasks currently allocated to the first computing node, wherein the first computing node is any one of the computing nodes; acquiring the core number and dominant frequency of a Central Processing Unit (CPU) of the first computing node from the performance characteristic information of the first computing node; calculating the calculation capacity value of the first calculation node according to the CPU core number and the dominant frequency; and calculating the load rate corresponding to the first calculation node according to the number of tasks currently allocated to the first calculation node and the calculation capacity value.
The computing capability value of the first computing node is the product of the CPU core number and the dominant frequency of the first computing node. In the embodiment of the present invention, if the CPU of the first computing node is a hyper-threaded CPU, the computing capability value of the first computing node is a product of a preset coefficient, the number of CPU cores of the first computing node, and the dominant frequency. The preset coefficient may be 1.25 or 1.5, and the specific value of the preset coefficient is not specifically limited in the embodiment of the present invention, and the value of the preset coefficient may be customized according to actual needs in actual applications. In addition, the number of the CPU cores is the number of physical cores of the first compute node, not the number of threads.
after the calculation capability value of the first calculation node is obtained in the above manner, the ratio between the number of tasks currently allocated to the first calculation node and the calculation capability value is calculated, and the calculated ratio is determined as the load rate corresponding to the first calculation node. And for each other computing node in the time-sharing scheduling system, the load rate corresponding to each other computing node is respectively calculated by the same method as the first computing node.
Because the number of tasks allocated to the computing nodes is changed from moment to moment, the load rate corresponding to the computing nodes is also changed from moment to moment, and the load rate corresponding to each computing node is calculated in real time through the mode.
When the load rate is calculated, the task amount distributed to the computing nodes and the computing capacity of the computing nodes are comprehensively considered, the problem of uneven scheduling caused by different computing capacities of the computing nodes is solved, and load balancing can be effectively realized.
The load rate corresponding to each computing node is computed in real time in the above manner, and when there is a task to be scheduled that needs to be processed at the client, the task to be scheduled is allocated to the computing node with the smallest load rate through the following step 103.
step 203: and distributing the task to be scheduled to the computing node with the minimum load rate according to the load rate corresponding to each computing node.
According to the load rate corresponding to each computing node, sequencing the virtual nodes corresponding to each computing node according to a preset sequence; selecting a virtual node with the minimum load rate from the sorted virtual nodes; and inserting the task to be scheduled into the tail of the waiting queue in the virtual node with the minimum load rate through the scheduler.
the above-mentioned sorting operation of the virtual nodes is performed in the scheduler, and the preset order may be an order from a load rate to a load rate, or an order from a load rate to a load rate.
And the processor of the client submits the task to be scheduled to the SDK, the task to be scheduled is distributed to the virtual node with the minimum load rate by the scheduler according to the mode, and the task to be scheduled is inserted into the tail of the waiting queue of the virtual node with the minimum load rate.
After the task to be scheduled is inserted into the waiting queue for waiting, the client further needs to send the task to be scheduled, which is arranged at the head of the waiting queue, to the computing node for processing in the following manner, which specifically includes:
sending a task to be scheduled to a corresponding computing node through a communication connection between a virtual node with the minimum load rate and the corresponding computing node by a task sender; deleting the task to be scheduled from the waiting queue of the virtual node with the minimum load rate; and inserting the task to be scheduled into the tail of the work queue of the virtual node with the minimum load rate.
After the task to be scheduled is sent to the computing node with the minimum load rate in the above mode, the computing node receives and processes the task to be scheduled, and returns a processing result corresponding to the task to be scheduled after the processing is completed.
The client receives a processing result corresponding to the task to be scheduled, which is returned by the computing node corresponding to the virtual node with the minimum load rate; and deleting the task to be scheduled from the work queue of the virtual node with the minimum load rate.
Due to the fact that the problem that data packets are lost possibly exists in the process of sending the tasks to be scheduled, the computing nodes cannot finish processing the tasks to be scheduled, the client cannot receive the processing result corresponding to the tasks to be scheduled, which is returned by the computing nodes, of the computing nodes, and therefore the tasks to be scheduled are cached in the work queue all the time, and dead tasks are formed. In order to avoid dead tasks in the work queue, the embodiment of the invention times the tasks cached in the work queue and deletes overtime tasks. The specific operation process of the timing mechanism comprises the following steps:
setting a timer in the created virtual node; respectively recording the processing duration of each task in the work queue of the virtual node through a timer; and when detecting the task with the processing time length exceeding the preset time length, deleting the task with the processing time length exceeding the preset time length from the work queue, and displaying the overtime error prompt message.
The timer mechanism can avoid the problem of task loss caused by packet loss of network transmission data and the problem that computing resources cannot be fully utilized due to the fact that dead tasks exist in a work queue.
in the embodiment of the invention, the load rate comprehensively considers the number of tasks currently allocated to the computing nodes and the computing capacity of the computing nodes, the problem of uniform scheduling of matching the task quantity with the computing capacity is effectively solved, and meanwhile, the load state of the computing nodes can be timely sensed through a processing result feedback mechanism of the computing nodes, so that the computing resources are fully utilized. According to the scheme, threshold limitation can be set on the load rate of the computing node, when the load rate of the computing node exceeds the threshold, tasks are not dispatched to the computing node any more, and the problem of insufficient memory caused by overlarge buffer amount of the distributed tasks is avoided.
The distributed scheduling system provided by the embodiment of the invention has high timeliness of task scheduling, can quickly return the processing result corresponding to the task after the task is issued, and is also suitable for interactive application with high real-time requirement.
in the embodiment of the invention, the node information of each computing node is obtained from the scheduling center, the load rate corresponding to each computing node is respectively calculated according to the number of tasks currently distributed to each computing node and the performance characteristic information of each computing node, and the task to be scheduled is distributed to the computing node with the minimum load rate according to the load rate corresponding to each computing node. The invention carries out scheduling by the client, and the task data does not pass through the scheduling center, thereby avoiding the bottleneck limit of the scheduling center. And the performance of the computing nodes is considered during load balancing, and uniform scheduling can be realized.
the method provided by the embodiment of the present invention has the same implementation principle and technical effect as the principle and technical effect of the client in the foregoing system embodiment, and for brief description, reference may be made to the corresponding contents in the foregoing system embodiment for the part of the method embodiment that is not mentioned. It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process of the system described above may also refer to the corresponding process in the foregoing method embodiment, and is not described herein again.
In the embodiments provided by the present invention, it should be understood that the disclosed system and method can be implemented in other ways. The above-described system and method embodiments are merely exemplary. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus once an item is defined in one figure, it need not be further defined and explained in subsequent figures, and moreover, the terms "first", "second", "third", etc. are used merely to distinguish one description from another and are not to be construed as indicating or implying relative importance.
finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present invention, which are used for illustrating the technical solutions of the present invention and not for limiting the same, and the protection scope of the present invention is not limited thereto, although the present invention is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the present invention in its spirit and scope. Are intended to be covered by the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (12)
1. A distributed scheduling system for big data processing, the system comprising: the system comprises a client, a plurality of dispatching centers and a plurality of computing nodes;
The dispatching center is used for receiving and storing node information sent by each computing node, wherein the node information comprises node position information and performance characteristic information;
the client is used for acquiring node information of each computing node from the scheduling center, respectively calculating the load rate corresponding to each computing node according to the number of tasks currently allocated to each computing node and the performance characteristic information of each computing node, and allocating the tasks to be scheduled to the computing nodes with the minimum load rate according to the load rate corresponding to each computing node;
the computing node is used for sending node information to each scheduling center and receiving and processing the task to be scheduled sent by the client;
The client is used for respectively creating corresponding virtual nodes for each computing node through a Software Development Kit (SDK); storing node information of a computing node corresponding to the virtual node in the created virtual node, and constructing a waiting queue and a work queue; establishing communication connection between the created virtual node and the corresponding computing node through an interface provided by the SDK;
The client is further used for sequencing the virtual nodes corresponding to each computing node according to a preset sequence according to the load rate corresponding to each computing node; selecting a virtual node with the minimum load rate from the sorted virtual nodes; and inserting the task to be scheduled into the tail of the waiting queue of the virtual node with the minimum load rate through a scheduler.
2. The system according to claim 1, wherein the client is configured to establish a communication connection with the dispatch center through an interface provided by a software development kit SDK, and obtain node information of each computing node from the dispatch center through the established communication connection.
3. The system of claim 1,
The client is used for counting the number of tasks in a waiting queue and a work queue corresponding to a first computing node, determining the counted number of tasks as the number of tasks currently allocated to the first computing node, and the first computing node is any one of the computing nodes; acquiring the number of CPU cores and dominant frequency of a central processing unit of the first computing node from the performance characteristic information of the first computing node; calculating the calculation capacity value of the first calculation node according to the CPU core number and the main frequency; and calculating the load rate corresponding to the first calculation node according to the number of tasks currently distributed to the first calculation node and the calculation capacity value.
4. The system of claim 1,
the client is used for sending the task to be scheduled to the corresponding computing node through a task sender through the communication connection between the virtual node with the minimum load rate and the corresponding computing node; deleting the task to be scheduled from the waiting queue of the virtual node with the minimum load rate; and inserting the task to be scheduled into the tail of the work queue of the virtual node with the minimum load rate.
5. the system according to claim 4, wherein the client is configured to receive a processing result corresponding to the task to be scheduled, which is returned by the computing node with the smallest load rate; and deleting the task to be scheduled from the work queue of the virtual node with the minimum load rate.
6. the system of claim 1, wherein the client is configured to set a timer in the created virtual node; respectively recording the processing duration of each task in the work queue of the virtual node through the timer; and when detecting the task with the processing time length exceeding the preset time length, deleting the task with the processing time length exceeding the preset time length from the work queue, and displaying overtime error prompt information.
7. a distributed scheduling method for big data processing, the method comprising:
acquiring node information of each computing node from a scheduling center, wherein the node information comprises node position information and performance characteristic information;
Respectively creating corresponding virtual nodes for each computing node through a Software Development Kit (SDK); storing node information of a computing node corresponding to the virtual node in the created virtual node, and constructing a waiting queue and a work queue; establishing communication connection between the created virtual node and the corresponding computing node through an interface provided by the SDK;
Respectively calculating the load rate corresponding to each computing node according to the number of tasks currently distributed to each computing node and the performance characteristic information of each computing node;
According to the load rate corresponding to each computing node and the load rate corresponding to each computing node, sequencing the virtual nodes corresponding to each computing node according to a preset sequence; selecting a virtual node with the minimum load rate from the sorted virtual nodes; and inserting the task to be scheduled into the tail of the waiting queue of the virtual node with the minimum load rate through the scheduler.
8. The method according to claim 7, wherein the calculating the load rate corresponding to each computing node according to the number of tasks currently allocated to each computing node and the performance characteristic information of each computing node respectively comprises:
Counting the number of tasks in a waiting queue and a work queue corresponding to a first computing node, and determining the counted number of tasks as the number of tasks currently allocated to the first computing node, wherein the first computing node is any one of the computing nodes;
Acquiring the number of CPU cores and dominant frequency of a central processing unit of the first computing node from the performance characteristic information of the first computing node;
Calculating the calculation capacity value of the first calculation node according to the CPU core number and the main frequency;
and calculating the load rate corresponding to the first calculation node according to the number of tasks currently distributed to the first calculation node and the calculation capacity value.
9. the method of claim 7, wherein the obtaining node information for each computing node from a dispatch center comprises:
Establishing communication connection with a dispatching center through an interface provided by a Software Development Kit (SDK);
And acquiring the node information of each computing node from the dispatching center through the established communication connection.
10. The method of claim 7, further comprising:
Sending the task to be scheduled to the corresponding computing node through the communication connection between the virtual node with the minimum load rate and the corresponding computing node by a task sender;
Deleting the task to be scheduled from a waiting queue of the virtual node with the minimum load rate;
and inserting the task to be scheduled into the tail of the work queue of the virtual node with the minimum load rate.
11. the method of claim 10, further comprising:
Receiving a processing result corresponding to the task to be scheduled, which is returned by the computing node corresponding to the virtual node with the minimum load rate;
and deleting the task to be scheduled from the work queue of the virtual node with the minimum load rate.
12. The method of claim 7, further comprising:
Setting a timer in the created virtual node;
respectively recording the processing duration of each task in the work queue of the virtual node through the timer;
and when detecting the task with the processing time length exceeding the preset time length, deleting the task with the processing time length exceeding the preset time length from the work queue, and displaying overtime error prompt information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610404437.0A CN106095572B (en) | 2016-06-08 | 2016-06-08 | distributed scheduling system and method for big data processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610404437.0A CN106095572B (en) | 2016-06-08 | 2016-06-08 | distributed scheduling system and method for big data processing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106095572A CN106095572A (en) | 2016-11-09 |
| CN106095572B true CN106095572B (en) | 2019-12-06 |
Family
ID=57228432
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610404437.0A Active CN106095572B (en) | 2016-06-08 | 2016-06-08 | distributed scheduling system and method for big data processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106095572B (en) |
Families Citing this family (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108319495B (en) * | 2017-01-16 | 2023-01-31 | 阿里巴巴集团控股有限公司 | Task processing method and device |
| CN106874112B (en) * | 2017-01-17 | 2020-04-28 | 华南理工大学 | Workflow backfilling method combined with load balancing |
| CN106933662A (en) * | 2017-03-03 | 2017-07-07 | 广东神马搜索科技有限公司 | Distributed system and its dispatching method and dispatching device |
| CN106937092A (en) * | 2017-04-11 | 2017-07-07 | 北京邮电大学 | Video data moving method and device in a kind of Distributed Computing Platform |
| CN110569252B (en) * | 2018-05-16 | 2023-04-07 | 杭州海康威视数字技术股份有限公司 | Data processing system and method |
| CN109254835A (en) * | 2018-06-27 | 2019-01-22 | 阿里巴巴集团控股有限公司 | Processing method, device, server and the readable storage medium storing program for executing of batch processing task |
| CN109359930A (en) * | 2018-12-20 | 2019-02-19 | 上海德启信息科技有限公司 | Logistics operation resource monitoring method, device and computer equipment |
| CN111381956B (en) * | 2018-12-28 | 2024-02-27 | 杭州海康威视数字技术股份有限公司 | Task processing method and device and cloud analysis system |
| CN110554923B (en) * | 2019-09-09 | 2020-05-22 | 北京人民在线网络有限公司 | Optimization method and system for distributed chained computing resources for cloud computing |
| CN110708576A (en) * | 2019-09-25 | 2020-01-17 | 李多 | Viewing data processing method, device and storage medium |
| CN110737521B (en) * | 2019-10-14 | 2021-03-05 | 中国人民解放军32039部队 | Disaster recovery method and device based on task scheduling center |
| CN112887345A (en) * | 2019-11-29 | 2021-06-01 | 上海交通大学 | Node load balancing scheduling method for edge computing environment |
| CN111522662B (en) * | 2020-04-23 | 2020-11-27 | 柴懿晖 | Node system for financial analysis and implementation method thereof |
| CN114063888A (en) * | 2020-07-31 | 2022-02-18 | 中移(苏州)软件技术有限公司 | Data storage system, data processing method, terminal and storage medium |
| CN111813565B (en) * | 2020-09-15 | 2021-08-10 | 北京东方通科技股份有限公司 | Method and system for balancing workload in a grid computing environment |
| CN112769907A (en) * | 2020-12-29 | 2021-05-07 | 苏宁消费金融有限公司 | Internal data exchange system of distributed system based on internet financial industry |
| CN113483479B (en) * | 2021-05-31 | 2022-07-26 | 国网浙江省电力有限公司 | Auxiliary service method and system combining variable frequency air conditioner and energy storage battery |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2004061663A2 (en) * | 2002-12-31 | 2004-07-22 | Globespanvirata Incorporated | System and method for providing hardware-assisted task scheduling |
| CN101604264A (en) * | 2009-07-08 | 2009-12-16 | 深圳先进技术研究院 | The method for scheduling task of supercomputer and system |
| CN102033777A (en) * | 2010-09-17 | 2011-04-27 | 中国资源卫星应用中心 | Distributed type job scheduling engine based on ICE (internet communication engine) |
| CN102685237A (en) * | 2012-05-16 | 2012-09-19 | 东南大学 | Method for requesting session maintaining and dispatching in cluster environment |
| CN102868733A (en) * | 2012-08-29 | 2013-01-09 | 北京邮电大学 | Method for remapping virtual network resources |
| CN103763373A (en) * | 2014-01-23 | 2014-04-30 | 浪潮(北京)电子信息产业有限公司 | Method for dispatching based on cloud computing and dispatcher |
| CN104253850A (en) * | 2014-01-07 | 2014-12-31 | 深圳市华傲数据技术有限公司 | Distributed task scheduling method and system |
| CN104317650A (en) * | 2014-10-10 | 2015-01-28 | 北京工业大学 | Map/Reduce type mass data processing platform-orientated job scheduling method |
| CN104778080A (en) * | 2014-01-14 | 2015-07-15 | 中兴通讯股份有限公司 | Job scheduling processing method and device based on coprocessor |
| CN104852934A (en) * | 2014-02-13 | 2015-08-19 | 阿里巴巴集团控股有限公司 | Method for realizing flow distribution based on front-end scheduling, device and system thereof |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7475108B2 (en) * | 2003-06-26 | 2009-01-06 | International Business Machines Corporation | Slow-dynamic load balancing method |
| CN103246546B (en) * | 2013-05-07 | 2016-01-20 | 山东大学 | Based on open electric system Numeral Emulation System and the emulation mode thereof of cloud |
| CN103617086B (en) * | 2013-11-20 | 2017-02-08 | 东软集团股份有限公司 | Parallel computation method and system |
| CN104317658B (en) * | 2014-10-17 | 2018-06-12 | 华中科技大学 | A kind of loaded self-adaptive method for scheduling task based on MapReduce |
-
2016
- 2016-06-08 CN CN201610404437.0A patent/CN106095572B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2004061663A2 (en) * | 2002-12-31 | 2004-07-22 | Globespanvirata Incorporated | System and method for providing hardware-assisted task scheduling |
| CN101604264A (en) * | 2009-07-08 | 2009-12-16 | 深圳先进技术研究院 | The method for scheduling task of supercomputer and system |
| CN102033777A (en) * | 2010-09-17 | 2011-04-27 | 中国资源卫星应用中心 | Distributed type job scheduling engine based on ICE (internet communication engine) |
| CN102685237A (en) * | 2012-05-16 | 2012-09-19 | 东南大学 | Method for requesting session maintaining and dispatching in cluster environment |
| CN102868733A (en) * | 2012-08-29 | 2013-01-09 | 北京邮电大学 | Method for remapping virtual network resources |
| CN104253850A (en) * | 2014-01-07 | 2014-12-31 | 深圳市华傲数据技术有限公司 | Distributed task scheduling method and system |
| CN104778080A (en) * | 2014-01-14 | 2015-07-15 | 中兴通讯股份有限公司 | Job scheduling processing method and device based on coprocessor |
| CN103763373A (en) * | 2014-01-23 | 2014-04-30 | 浪潮(北京)电子信息产业有限公司 | Method for dispatching based on cloud computing and dispatcher |
| CN104852934A (en) * | 2014-02-13 | 2015-08-19 | 阿里巴巴集团控股有限公司 | Method for realizing flow distribution based on front-end scheduling, device and system thereof |
| CN104317650A (en) * | 2014-10-10 | 2015-01-28 | 北京工业大学 | Map/Reduce type mass data processing platform-orientated job scheduling method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106095572A (en) | 2016-11-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106095572B (en) | distributed scheduling system and method for big data processing | |
| US9888048B1 (en) | Supporting millions of parallel light weight data streams in a distributed system | |
| US10999201B2 (en) | Dynamic advertisement routing | |
| Kliazovich et al. | CA-DAG: Modeling communication-aware applications for scheduling in cloud computing | |
| CN109076026B (en) | System and method for latency-based queuing | |
| US10341196B2 (en) | Reliably updating a messaging system | |
| CN111338773B (en) | Distributed timing task scheduling method, scheduling system and server cluster | |
| US20180034832A1 (en) | Systems and Methods for Real-time Configurable Load Determination | |
| CN108933829A (en) | A kind of load-balancing method and device | |
| CN109729106B (en) | Method, system and computer program product for processing computing tasks | |
| US11159443B2 (en) | Queue management in a forwarder | |
| CN109510878B (en) | Long connection session keeping method and device | |
| US10944683B1 (en) | Hybrid queue system for request throttling | |
| US9736235B2 (en) | Computer system, computer, and load balancing method | |
| Kliazovich et al. | CA-DAG: Communication-aware directed acyclic graphs for modeling cloud computing applications | |
| CN111080126B (en) | Task allocation method and device | |
| CN105681426B (en) | Heterogeneous system | |
| US20070233450A1 (en) | Simulation of connected devices | |
| CN111708637A (en) | Data processing method and device and computer readable medium | |
| CN109254854A (en) | Asynchronous invoking method, computer installation and storage medium | |
| JP2005310120A (en) | Computer system, and task assigning method | |
| Hu et al. | Job scheduling without prior information in big data processing systems | |
| CN115334082A (en) | Load balancing method, device, computer equipment, storage medium and product | |
| CN111431730A (en) | Service processing method, system, computer equipment and readable medium | |
| US8924481B2 (en) | Apparatus for routing requests |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| PP01 | Preservation of patent right |
Effective date of registration: 20220726 Granted publication date: 20191206 |
|
| PP01 | Preservation of patent right |