CN110012062A - A kind of multimachine room method for scheduling task, device and storage medium - Google Patents
A kind of multimachine room method for scheduling task, device and storage medium Download PDFInfo
- Publication number
- CN110012062A CN110012062A CN201910134018.3A CN201910134018A CN110012062A CN 110012062 A CN110012062 A CN 110012062A CN 201910134018 A CN201910134018 A CN 201910134018A CN 110012062 A CN110012062 A CN 110012062A
- Authority
- CN
- China
- Prior art keywords
- task
- cluster
- pending
- execution
- pending task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
- H04L67/1012—Server selection for load balancing based on compliance of requirements or conditions with available server resources
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Debugging And Monitoring (AREA)
Abstract
The present invention provides a kind of multimachine room method for scheduling task, device and storage mediums, and applied to the system for including multiple clusters, scheduler and memory, this method is implemented in scheduler, this method comprises: obtaining the data mode of previous task in memory；According to the data production status of previous task, it is determined whether to enable pending tasks；In the case where determining the pending task of unlatching, the execution duration for executing cluster and executing pending required by task for executing pending task is determined, which is the idle cluster in multiple clusters；According to execution cluster and execute duration, utilize preset implementation strategy, complete pending task, the implementation strategy is used for when the cluster for executing pending task is multiple, different idle clusters is determined as to execute cluster, it is carried out with corresponding across the scheduling of multiple cluster resources and distribution in computer room, and then promotes the robustness of big data system and the ability of lifting system safe operation, optimized allocation of resources.
Description
Technical field
The invention belongs to data processing fields, are situated between more particularly to a kind of multimachine room method for scheduling task, device and storage
Matter.
Background technique
In the prior art, the scheduling that cluster resource is carried out in single computer room is the suitable cluster resource of distribution to hold
The pending task of row, and in the computer room not idle cluster resource when, pending task is placed in and waits in line state, directly
To available free and suitable cluster resource.And will lead to the cluster resource of not executable task if computer room breaks down,
So that it cannot continue to execute pending task, therefore, the reliability of single computer room is low and does not have redundancy ability.
Currently, can use the pending task of cluster resource execution for being distributed in multimachine room in big data system, but each
Due to the limitation of communication function/permission between computer room, and it can not know resource and task status in other computer rooms, it is difficult to realize
The flexible allotment of cluster resource is carried out between multimachine room, also just can not overcome single machine using the cluster resource in multiple computer rooms
Reliability present in room is low and does not have the problem of redundancy ability, and then is difficult to ensure the robustness and disaster tolerance of big data system
Property.
Summary of the invention
In view of this, the present invention provides a kind of multimachine room method for scheduling task, device and storage medium, it is existing to solve
The problem of can not realizing the rational allocation of resource in technology in the case where multimachine room.
According to the present invention in a first aspect, provide a kind of multimachine room method for scheduling task, be applied to including multiple clusters,
The system of scheduler and memory, the multiple clustered deploy(ment) are carried out with the memory respectively in different computer rooms, each cluster
Communication, the memory are communicated with the scheduler, and the method is implemented in the scheduler, which comprises
Obtain the data mode of previous task in the memory；
According to the data production status of the previous task, it is determined whether to enable pending tasks；
In the case where determining the unlatching pending task, the execution cluster for executing the pending task is determined
And the execution duration of the pending required by task is executed, the cluster that executes is the idle cluster in the multiple cluster；
The pending task is completed using preset implementation strategy according to the execution cluster and the execution duration,
The implementation strategy is used to be determined as different idle clusters described when the cluster for executing the pending task is multiple
Execute cluster.
Second aspect according to the present invention provides a kind of multimachine room task scheduling apparatus, be applied to including multiple clusters,
The system of scheduler and memory, the multiple clustered deploy(ment) are carried out with the memory respectively in different computer rooms, each cluster
Communication, the memory are communicated with the scheduler, and described device is set to the scheduler, and described device includes:
Module is obtained, the data mode of previous task in the memory is obtained；
Detection module, for the data production status according to the previous task, it is determined whether to enable described pending
Business；
Cluster determining module, for determine open the pending task in the case where, determine for execute it is described to
The execution duration for executing cluster and executing the pending required by task of execution task, the execution cluster is the multiple
Idle cluster in cluster；
Task execution module is used for according to the execution cluster and the execution duration, complete using preset implementation strategy
At the pending task, the implementation strategy is used for when the cluster for executing the pending task is multiple, will be different
Idle cluster is determined as the execution cluster.
The third aspect according to the present invention provides a kind of computer readable storage medium, the computer-readable storage
Computer program is stored on medium, realizes that multimachine room as described in relation to the first aspect is appointed when the computer program is executed by processor
The step of business dispatching method.
For first technology, the present invention has following advantage:
Using scheduler according to the data mode of previous task, it is determined whether open pending task, be somebody's turn to do determining to open
Execution duration for cluster idle in its distribution system and is correspondingly arranged according to resource idle condition after pending task, passes through tune
The data interaction between device and multiple computer rooms is spent, multiple cluster states of multiple computer rooms are inquired, so as to corresponding multiple
Cluster resource between computer room is scheduled and distributes, and avoids based on single machine room failure, the caused execution for being unable to complete task,
Simultaneously in the case where executing the cluster of the pending task is multiple clusters, the multiple cluster is determined as to different skies
Not busy cluster, and then can dispatch other when single cluster failure, mission failure, cluster are crowded and be suitble to resource complete
It is optimized allocation of resources at pending task with the ability that lifting system is safely operated.
The above description is only an overview of the technical scheme of the present invention, in order to better understand the technical means of the present invention,
And it can be implemented in accordance with the contents of the specification, and in order to allow above and other objects of the present invention, feature and advantage can
It is clearer and more comprehensible, the followings are specific embodiments of the present invention.
Detailed description of the invention
By reading the following detailed description of the preferred embodiment, various other advantages and benefits are common for this field
Technical staff will become clear.The drawings are only for the purpose of illustrating a preferred embodiment, and is not considered as to the present invention
Limitation.And throughout the drawings, the same reference numbers will be used to refer to the same parts.In the accompanying drawings:
Fig. 1 is a kind of step flow chart of multimachine room method for scheduling task provided in an embodiment of the present invention；
Fig. 2 is a kind of specific steps flow chart of multimachine room method for scheduling task provided in an embodiment of the present invention；
Fig. 3 is a kind of specific steps flow chart of multimachine room method for scheduling task provided in an embodiment of the present invention；
Fig. 4 is a kind of specific steps flow chart of multimachine room method for scheduling task provided in an embodiment of the present invention；
Fig. 5 is a kind of block diagram of multimachine room task scheduling apparatus provided in an embodiment of the present invention.
Specific embodiment
The exemplary embodiment that the present invention will be described in more detail below with reference to accompanying drawings.Although showing the present invention in attached drawing
Exemplary embodiment, it being understood, however, that may be realized in various forms the present invention without should be by embodiments set forth here
It is limited.It is to be able to thoroughly understand the present invention on the contrary, providing these embodiments, and can be by the scope of the present invention
It is fully disclosed to those skilled in the art.
Fig. 1 is a kind of step flow chart of multimachine room method for scheduling task provided in an embodiment of the present invention, as shown in Figure 1,
Applied to the system for including multiple clusters, scheduler and memory, multiple clustered deploy(ment)s in different computer rooms, each cluster respectively with
Memory is communicated, and memory is communicated with scheduler, and this method is implemented in scheduler, may include:
Step 101, the data mode of previous task in memory is obtained.
In a particular application, (such as cheating data cleansing or report refer to some destination item based on big data system
Mark statistics), when carrying out a large amount of data processing, generally comprises several and execute step (task), and (appoint in these steps
Business) in, each operation (exploitation, submission and attended operation) may be under the jurisdiction of different team, since different team are corresponding with not
With cluster resource, and limited with different permission, A team can not/lack of competence be directly viewable B team task execution feelings
Condition, for execute the corresponding operation of A team cluster resource can not between the cluster resource for executing the corresponding operation of B team into
Row data interaction, therefore need to be scheduled and coordinate wherein using scheduler in big data system, it is obtained by scheduler
The completion status of partner i.e. previous step (task) is known, so that start the i.e. pending task of our task executes operation.
It should be noted that scheduler can use the data-interface of each computer room, the cluster in each computer room is got
Resource status and performed task status in each computer room.Illustratively, by HDFS (Chinese: Hadoop distributed document
System；English: Hadoop Distributed File System) it is used as memory, it is used to store done file, this document
To identify the completion status of task.After the completion of each task execution, one done file of corresponding generation, which is stored in, to be deposited
In reservoir, to show that the data production status of the task as completion status, such as can be set scheduler and periodically arrive HDFS
Whether the done file of middle inquiry previous task generates the data mode of previous task (obtain), when detecting that previous task given birth to
When at corresponding done file, scheduler, which then determines, opens pending task；Otherwise wait for the next period is inquired again, with
Determine whether previous task generates done file, and then it is determined whether to enable pending tasks.
It should be noted that computer room involved in the present invention refers to the hardware device institute that enterprise stores and calculates for data
On ground, for example, certain degree computer room, the self-built computer room in all parts of the country of computer room and other enterprises in certain, the number for the same enterprise
National different computer room can be distributed according to computing resource；And there is corresponding business team in each enterprise, is according to enterprise
The personnel that inside is divided according to function, the usual same big data analysis task, it may be necessary to multiple team are participated in, and one
Team provides a part of result and uses to other team, can be blacklist data and is also possible to abstract some statistical natures
Deng, it is therefore desirable to scheduler according to the upstream-downstream relationship between different machine room resource utilization rates and task to the resource of system into
Row scheduling.
Step 102, according to the data production status of previous task, it is determined whether to enable pending tasks.
Wherein, pending task is the task based on previous task data produced to execute, and pending task is to work as
Before the data production task that needs to be implemented.This includes the system of multiple clusters, scheduler and memory, by scheduler from depositing
The status information of cluster and the data production status of task are obtained on reservoir, and then realize the resource overall situation allotment in system,
It should be noted that the scheduler can be that by the functional module of technical solution proposed by the present invention, it is integrated in hardware
Virtual bench in equipment；Or technical solution proposed by the present invention can be can be realized with a hardware device with entity
Method and step, the present invention is with no restrictions.
Step 103, in the case where determining the pending task of unlatching, the execution cluster for executing pending task is determined
And execute the execution duration of pending required by task.
Wherein, executing cluster is the idle cluster in multiple clusters.
Under concrete application scene, when carrying out rational management to resource by scheduler, such as actual current each
The executive condition of a data production task determines its suitable execution cluster and corresponding execution duration, and then pending
This company-data production task failure of task, when being lined up or running slow, scheduler schedules other to can be used for executing data raw
The cluster of production task, so that the data production of pending task can continue to execute.Wherein, cluster refers to that distributed computing provides
Source can be distributed in different computer rooms, therefore in the scheduling process to pending task, the corresponding reasonable tune for carrying out executing cluster
Match, to optimize allocation of resources.
Step 104, pending task is completed using preset implementation strategy according to execution cluster and execution duration.
Wherein, implementation strategy is used to determine different idle clusters when the cluster for executing pending task is multiple
To execute cluster.
In a particular application, the execution cluster according to determined by step 103 and execution duration, will grow interior utilization when being executed
Execute that cluster carries out pending task executes operation, to complete the pending task.If long interior completion is pending when being executed
Task, such as corresponding data mode data, e.g. Done file can be generated in HDFS, to indicate that subsequent task can
To continue to execute；Otherwise, the idle cluster that other will be used different from predetermined execution cluster, continues pending
Business.For example, when using cluster A as execution cluster, when in preset execution duration without completing pending task, it may be possible to
As cluster A break down and caused by, and then scheduler is when distributing new execution cluster for pending task, even if cluster
A is idle state, does not also use cluster A, avoids making using cluster A (reusing the cluster resource to break down) again
Interior the case where completing can not be grown when being executed at pending task, to ensure the completion rate of pending task.
It should be noted that the scheduler that technical solution proposed by the present invention is related to, is that setting is (single in multiple computer rooms itself
One computer room) scheduling system on scheduler, United Dispatching after capable of being integrated to the cluster resource in different computer rooms,
So that in entire big data system realize resource global assignment, overcome stability existing for single computer room and disaster tolerance difference
Problem.
In conclusion multimachine room provided by the invention method for scheduling task, obtains the number of previous task in memory
According to state；According to the data production status of previous task, it is determined whether to enable pending tasks；Pending task is opened determining
In the case where, determine the execution duration for executing cluster and executing pending required by task for executing pending task, it should
Executing cluster is the idle cluster in multiple clusters；According to executing cluster and executing duration, using preset implementation strategy, complete
Pending task, the implementation strategy are used to determine different idle clusters when the cluster for executing pending task is multiple
To execute cluster, carried out with corresponding across the scheduling of multiple cluster resources and distribution in computer room, and then promote the Shandong of big data system
Stick and the ability of lifting system safe operation, optimize allocation of resources.
Optionally, Fig. 2 is a kind of specific steps flow chart of multimachine room method for scheduling task provided in an embodiment of the present invention,
It, can be with it is determined whether to enable pending task as shown in Fig. 2, according to the data production status of previous task described in step 102
Include:
Step 1021, according to the execution status data of previous task in systems, the data production shape of previous task is obtained
State.
Wherein, the current execution state that status data is used to indicate previous task is executed, illustratively, executing status data can
To be the data got by executing status file or execution state parameter.
In a particular application, the mode that can use Done file described in step 101, as execute status file,
Details are not described herein again.Further, it is also possible to determine preposition using the execution state parameter of each task stored in memory
The current execution state of business.For example, being inserted into a state in MySQL database (memory) after the completion of the step of working as A team
Data, such as be set as status (value be 0 or 1), as status=1, then it represents that the execution state of the task has been
At state；And as status=0, then it represents that the execution state of the task is unfinished state.Scheduler can be periodic
It is inquired in MySQL database, whether confirmation status is 1, and then judges whether to open and execute pending task.
In a particular application, exist between the partner due to previous task as pending task, with pending task
The data call relationship of upstream and downstream, therefore be only completion status in the data production status of the previous task of pending task
Situation can open the operation that executes of pending task, that is, carry out the operation of step 1022；Otherwise, in the number of previous task
In the case where being unfinished state according to production status, sleep (SLEEP) mechanism can be introduced, is carried out again after waiting for a period of time
The inquiry of the data production status of previous task, until it meets completion condition and then that opens pending task executes behaviour
Make.
Step 1022, in the case where the data production status of previous task is completion status, pending task is opened.
That is, can open after previous task has completed data production and execute pending business, it will be wait hold
Row task is put into data production task pond, such as pending business can be put into message team by the way of message queue
In column, after being assigned with being executed for task to idle cluster in step below, idle cluster is directly from message team
The corresponding task definition of the task is got in column (such as by included for obtaining the ground of task definition in message queue
Location pointer), and then execute corresponding task.
Step 1023, in the case where the data production status of previous task is unfinished state, pause is opened pending
Task.
Illustratively, in the case where the data production status of previous task is unfinished state, illustrate pending task
The data of partner's (previous task) are simultaneously unripe, need that previous task is waited to continue to execute, until data production status is
Completion status is then turned on pending task, and pending task will be placed in sleep state, and period in waiting process by scheduler
Property inquiry previous task data production status.
Optionally, Fig. 3 is a kind of specific steps flow chart of multimachine room method for scheduling task provided in an embodiment of the present invention,
As shown in figure 3, being determined in the case where determining the pending task of unlatching for executing pending task described in step 103
It executes cluster and executes the execution duration of pending required by task, may include:
Step 1031, the resource idle condition according to the status information of cluster in system, in acquisition system.
Illustratively, when carrying out multimachine room task schedule, scheduler needs the monitoring of each cluster state of real-time perfoming, respectively
A cluster reports to its state (whether idle) in the memory of system, before scheduler can carry out within a preset time interval
The inquiry of the status information of task is set, and then determines the state of cluster resource and the number of each task in real-time multimachine room
According to information such as production statuses, cluster is executed in order to determine for pending task, while the longest that pending task is arranged executes
Duration is as duration is executed, to start to execute pending task.
Step 1032, according to resource idle condition and be the preset alternative cluster of pending task, filter out alternative cluster
In an idle cluster, using as executing cluster.
In a particular application, it due to the machine that cluster is a kind of offer distributed computing resource, can deposit in systems sometimes
In dedicated cluster, the dedicated cluster of task e.g. on execution line, also some are test clusters, dedicated for executing corresponding survey
Trial business.In order to reduce scheduling failure rate, when system is initially set up, Initialize installation, choosing are carried out to the resource distribution of task
Several clusters that can support task execution are selected, alternately cluster, scheduler according to above-mentioned alternative cluster and can work as later
The preceding alternative cluster whether be idle state information, carry out cluster resource scheduling.That is, by one in alternative cluster
Idle cluster is as cluster is executed, to execute pending task.
Step 1033, according to pending task, setting executes duration.
Illustratively, the setting of the execution duration can be determined based on data volume size included by pending task, also
Adjustment can be corresponded in conjunction with the computing resource size in above-mentioned steps being alternative cluster set by pending task.The execution
Maximum needed for duration can be used as execution cluster executes duration, after maximum, which executes duration, to be terminated, then will not be further continued for benefit
The execution of pending task is carried out with the execution cluster, executes same pending to avoid the cluster to break down is reused
Business.
Optionally, Fig. 4 is a kind of specific steps flow chart of multimachine room method for scheduling task provided in an embodiment of the present invention,
As shown in figure 4, using preset implementation strategy, completing pending according to executing cluster and executing duration described in step 104
It is engaged in, may include:
Step 1041, in growing when being executed, start to execute pending task using cluster is executed.
Step 1042, if executing completions in pending task is long when being executed, pending task is completed in determination.
Step 1043, if pending task when being executed grow in be not carried out completion, the resource space in reacquisition system
Not busy situation, to determine new execution cluster and new execution duration according to resource idle condition and pending task.
Step 1044, pending task is continued to execute in new execution duration using new execution cluster, until completing
Pending task.
In a particular application, idle cluster is utilized to execute pending task and do not reuse according to length when being executed is interior
The principle of same asset, and then judged whether after execution duration (maximum executes duration) determined by step 1033
At pending business, if completing this finishing scheduling for illustrating pending task；Otherwise pending when the execution duration terminates
Task does not complete also, then scheduler is by the resource idle condition in inquiry system again, to determine new idle cluster, i.e. step
1043, primary pending task is submitted again in new (same task is not submitted repeatedly in same cluster) idle cluster, together
When be correspondingly arranged a new execution duration, and then continued to execute in new execution duration using new execution cluster pending
Task, until pending task is completed.
It should be noted that be different to execute cluster determined by pending task, and executing duration can be identical
It can also be different, it is true according to the resource idle condition and pending task in actual system for executing cluster and executing duration
Fixed.For example, when scheduler is when opening pending task, it is corresponding to determine that cluster 1 (one of alternative cluster) is used as execution cluster,
And according to the actual task content of pending task (the data volume size in task), when execution of setting a length of 300s.?
After 300s, pending task is remained unfulfilled, then the resource idle condition in reacquisition system, and discovery cluster 1 is still sky
Not busy state, but since same task is not submitted repeatedly in same cluster, and the cluster 2 in the alternative cluster of pending task is at this time
For idle state, then cluster 2 is selected to execute cluster, and the remaining task definition of pending task is former task definition
70%, a length of 500s when execution can be set, start with cluster 2 and continue to execute in the task of residue 70% of pending task
Hold.Therefore, it can be broken down in pending task in single cluster by above-mentioned source scheduling mode, mission failure, cluster
When crowded, when will deploy other suitable idling-resources by scheduler to pending task and readjust execution
It is long, pending task is continued to execute, to complete the pending task.
Fig. 5 is a kind of block diagram of multimachine room task scheduling apparatus provided in an embodiment of the present invention, as shown in figure 5, being applied to
System including multiple clusters, scheduler and memory, multiple clustered deploy(ment)s in different computer rooms, each cluster respectively with memory
It is communicated, memory is communicated with scheduler, and device 500 is set to scheduler, which includes:
Module 510 is obtained, the data mode of previous task in memory is obtained.
Detection module 520, for the data production status according to previous task, it is determined whether to enable pending tasks.
Cluster determining module 530, for determining for executing pending in the case where determining the pending task of unlatching
The execution duration for executing cluster and executing pending required by task of business, executing cluster is the idle cluster in multiple clusters.
Task execution module 540, for according to executing cluster and executing duration, using preset implementation strategy, complete to
Execution task, implementation strategy are used to be determined as holding by different idle clusters when the cluster for executing pending task is multiple
Row cluster.
Optionally, which includes:
State acquisition submodule obtains previous task for the execution status data according to previous task in systems
Data production status executes the current execution state that status data is used to indicate previous task.
Task status determines submodule, for opening in the case where the data production status of previous task is completion status
Open pending task；Alternatively, suspending pending in the case where the data production status of previous task is unfinished state
Business.
Optionally, the cluster determining module 530, comprising:
Resource acquisition submodule, the resource idle condition for the status information according to cluster in system, in acquisition system；
Cluster screen submodule, for according to resource idle condition and be the preset alternative cluster of pending task, screening
One in alternative cluster idle cluster out, using as executing cluster；
Submodule is arranged in duration, and for the data volume according to pending task, setting executes duration.
Optionally, the task execution module 540, comprising:
Task execution submodule starts to execute pending task in growing when being executed using cluster is executed；
It completes to determine submodule, if determining for executing completion in pending task when being executed length and completing pending
Business.
Optionally, task execution module, further includes:
Resource acquisition submodule reacquires system if being not carried out completion in growing when being executed for pending task
Interior resource idle condition, with according to resource idle condition and pending task, determine it is new execute cluster and new execution when
It is long；
Data implementation sub-module, for continuing to execute pending in new execution duration using new execution cluster
Business, until completing pending task.
In addition, the embodiment of the present invention also provides a kind of terminal, including processor, memory, storage is on a memory and can
The computer program run in processing, the computer program realize above-mentioned multimachine room method for scheduling task when being executed by processor
Each process of embodiment, and identical technical effect can be reached, to avoid repeating, which is not described herein again.
The embodiment of the present invention also provides a kind of computer readable storage medium, and meter is stored on computer readable storage medium
Calculation machine program, the computer program realize each mistake of above-mentioned multimachine room method for scheduling task embodiment when being executed by processor
Journey, and identical technical effect can be reached, to avoid repeating, which is not described herein again.Wherein, the computer-readable storage medium
Matter can be read-only memory (Read-Only Memory, abbreviation ROM), random access memory (Random Access
Memory, abbreviation RAM), magnetic or disk etc..
For above-mentioned apparatus embodiment, since it is basically similar to the method embodiment, so be described relatively simple,
The relevent part can refer to the partial explaination of embodiments of method.
All the embodiments in this specification are described in a progressive manner, the highlights of each of the examples are with
The difference of other embodiments, the same or similar parts between the embodiments can be referred to each other.
It would have readily occurred to a person skilled in the art that: any combination application of above-mentioned each embodiment is all feasible, therefore
Any combination between above-mentioned each embodiment is all embodiment of the present invention, but this specification exists as space is limited,
This is not just detailed one by one.
Multimachine room method for scheduling task is not solid with any certain computer, virtual system or other equipment provided herein
There is correlation.Various general-purpose systems can also be used together with teachings based herein.As described above, construction has this hair
Structure required by the system of bright scheme is obvious.In addition, the present invention is also not directed to any particular programming language.It should
Understand, can use various programming languages and realize summary of the invention described herein, and language-specific is done above
Description is in order to disclose the best mode of carrying out the invention.
In the instructions provided here, numerous specific details are set forth.It is to be appreciated, however, that implementation of the invention
Example can be practiced without these specific details.In some instances, well known method, structure is not been shown in detail
And technology, so as not to obscure the understanding of this specification.
Similarly, it should be understood that in order to simplify the present invention and help to understand one or more of the various inventive aspects,
Above in the description of exemplary embodiment of the present invention, each feature of the invention is grouped together into single implementation sometimes
In example, figure or descriptions thereof.However, the disclosed method should not be interpreted as reflecting the following intention: i.e. required to protect
Shield the present invention claims features more more than feature expressly recited in each claim.More precisely, such as right
As claim reflects, inventive aspect is all features less than single embodiment disclosed above.Therefore, it then follows tool
Thus claims of body embodiment are expressly incorporated in the specific embodiment, wherein each claim conduct itself
Separate embodiments of the invention.
Those skilled in the art will understand that can be carried out adaptively to the module in the equipment in embodiment
Change and they are arranged in one or more devices different from this embodiment.It can be the module or list in embodiment
Member or component are combined into a module or unit or component, and furthermore they can be divided into multiple submodule or subelement or
Sub-component.Other than such feature and/or at least some of process or unit exclude each other, it can use any
Combination is to all features disclosed in this specification (including adjoint claim, abstract and attached drawing) and so disclosed
All process or units of what method or apparatus are combined.Unless expressly stated otherwise, this specification is (including adjoint power
Benefit require, abstract and attached drawing) disclosed in each feature can carry out generation with an alternative feature that provides the same, equivalent, or similar purpose
It replaces.
In addition, it will be appreciated by those of skill in the art that although some embodiments described herein include other embodiments
In included certain features rather than other feature, but the combination of the feature of different embodiments mean it is of the invention
Within the scope of and form different embodiments.For example, in detail in the claims, embodiment claimed it is one of any
Can in any combination mode come using.
Various component embodiments of the invention can be implemented in hardware, or to run on one or more processors
Software module realize, or be implemented in a combination thereof.It will be understood by those of skill in the art that can be used in practice
Microprocessor or digital signal processor (DSP) are realized in multimachine room according to an embodiment of the present invention method for scheduling task
The some or all functions of some or all components.The present invention is also implemented as executing method as described herein
Some or all device or device programs (for example, computer program and computer program product).Such reality
Existing program of the invention can store on a computer-readable medium, or may be in the form of one or more signals.
Such signal can be downloaded from an internet website to obtain, and perhaps be provided on the carrier signal or in any other forms
It provides.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and ability
Field technique personnel can be designed alternative embodiment without departing from the scope of the appended claims.In the claims,
Any reference symbol between parentheses should not be configured to limitations on claims.Word "comprising" does not exclude the presence of not
Element or step listed in the claims.Word "a" or "an" located in front of the element does not exclude the presence of multiple such
Element.The present invention can be by means of including the hardware of several different elements and being come by means of properly programmed computer real
It is existing.In the unit claims listing several devices, several in these devices can be through the same hardware branch
To embody.The use of word first, second, and third does not indicate any sequence.These words can be explained and be run after fame
Claim.
Claims (11)
1. a kind of multimachine room method for scheduling task, which is characterized in that applied to including multiple clusters, scheduler and memory
System, the multiple clustered deploy(ment) are communicated with the memory respectively in different computer rooms, each cluster, the memory and institute
It states scheduler to be communicated, the method is implemented in the scheduler, which comprises
Obtain the data mode of previous task in the memory；
According to the data production status of the previous task, it is determined whether to enable pending tasks；
Determine open the pending task in the case where, determine for execute the pending task execution cluster and
The execution duration of the pending required by task is executed, the cluster that executes is the idle cluster in the multiple cluster；
The pending task is completed using preset implementation strategy according to the execution cluster and the execution duration, it is described
Implementation strategy is used to that different idle clusters to be determined as the execution when the cluster for executing the pending task is multiple
Cluster.
2. the method according to claim 1, wherein the data production status according to previous task, determines
Whether the pending task is opened, comprising:
According to the execution status data of the previous task in the system, the data production shape of the previous task is obtained
State；
In the case where the data production status of the previous task is completion status, the pending task is opened；Alternatively,
In the case that the data production status of the previous task is unfinished state, suspend the pending task.
3. the method according to claim 1, wherein described determining the case where opening the pending task
Under, determine the execution duration for executing cluster and executing the pending required by task for executing the pending task,
Include:
According to the status information of cluster described in the system, the resource idle condition in the system is obtained；
According to the resource idle condition and it is the pending preset alternative cluster of task, filters out in the alternative cluster
An idle cluster, using as the execution cluster；
According to the data volume of the pending task, the execution duration is set.
4. according to the method described in claim 3, utilizing preset execution it is characterized in that, described according to the execution cluster
Strategy completes the pending task, comprising:
In the execution duration, start to execute the pending task using the execution cluster；
If the pending task executes completion in the execution duration, determines and complete the pending task.
5. according to the method described in claim 4, it is characterized in that, the method also includes:
If the pending task is not carried out completion in the execution duration, the resource reacquired in the system is idle
Situation, to determine new execution cluster and new execution duration according to the resource idle condition and the pending task；
The pending task is continued to execute in the new execution duration using the new execution cluster, until completing institute
State pending task.
6. a kind of multimachine room task scheduling apparatus, which is characterized in that applied to including multiple clusters, scheduler and memory
System, the multiple clustered deploy(ment) are communicated with the memory respectively in different computer rooms, each cluster, the memory and institute
It states scheduler to be communicated, described device is set to the scheduler, and described device includes:
Module is obtained, the data mode of previous task in the memory is obtained；
Detection module, for the data production status according to the previous task, it is determined whether to enable the pending tasks；
Cluster determining module, in the case where determining the unlatching pending task, determination to be described pending for executing
The execution duration for executing cluster and executing the pending required by task of task, the execution cluster is the multiple cluster
In idle cluster；
Task execution module, for completing institute using preset implementation strategy according to the execution cluster and the execution duration
Pending task is stated, the implementation strategy is used for when the cluster for executing the pending task is multiple, by the different free time
Cluster is determined as the execution cluster.
7. device according to claim 6, which is characterized in that the detection module, comprising:
State acquisition submodule, for the execution status data according to the previous task in the system, before acquisition is described
Set the data production status of task；
Task status determines submodule, for opening in the case where the data production status of the previous task is completion status
Open the pending task；Alternatively, suspending institute in the case where the data production status of the previous task is unfinished state
State pending task.
8. device according to claim 6, which is characterized in that the cluster determining module, comprising:
Resource acquisition submodule obtains the resource in the system for the status information of the cluster according to the system
Idle condition；
Cluster screen submodule, for according to the resource idle condition and be the pending preset alternative cluster of task,
One in the alternative cluster idle cluster is filtered out, using as the execution cluster；
Duration is arranged submodule and the execution duration is arranged for the data volume according to the pending task.
9. device according to claim 8, which is characterized in that the task execution module, comprising:
Task execution submodule, for starting to execute described pending using the execution cluster in the execution duration
Business；
It completes to determine submodule, if executing completion in the execution duration for the pending task, determine described in completing
Pending task.
10. device according to claim 9, which is characterized in that the task execution module, further includes:
Resource acquisition submodule reacquires if being not carried out completion in the execution duration for the pending task
Resource idle condition in the system, to determine new execution according to the resource idle condition and the pending task
Cluster and new execution duration；
Data implementation sub-module, for described in being continued to execute in the new execution duration using new the executions cluster to
Execution task, until completing the pending task.
11. a kind of computer readable storage medium, which is characterized in that store computer journey on the computer readable storage medium
Sequence realizes the multimachine room task schedule as described in any one of claims 1 to 5 when the computer program is executed by processor
The step of method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910134018.3A CN110012062B (en) | 2019-02-22 | 2019-02-22 | Multi-computer-room task scheduling method and device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910134018.3A CN110012062B (en) | 2019-02-22 | 2019-02-22 | Multi-computer-room task scheduling method and device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110012062A true CN110012062A (en) | 2019-07-12 |
| CN110012062B CN110012062B (en) | 2022-02-08 |
Family
ID=67165890
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910134018.3A Active CN110012062B (en) | 2019-02-22 | 2019-02-22 | Multi-computer-room task scheduling method and device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110012062B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110519381A (en) * | 2019-08-29 | 2019-11-29 | 深圳前海微众银行股份有限公司 | Processing method, device, terminal device and the storage medium of distributed task scheduling |
| CN111818159A (en) * | 2020-07-08 | 2020-10-23 | 腾讯科技（深圳）有限公司 | Data processing node management method, device, equipment and storage medium |
| CN112114950A (en) * | 2020-09-21 | 2020-12-22 | 中国建设银行股份有限公司 | Task scheduling method and device and cluster management system |
| CN112181607A (en) * | 2020-11-06 | 2021-01-05 | 北京凌云雀科技有限公司 | Component processing method, device, equipment and storage medium |
| CN112817713A (en) * | 2021-01-27 | 2021-05-18 | 广州虎牙科技有限公司 | Job scheduling method and device and electronic equipment |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080046500A1 (en) * | 2004-06-15 | 2008-02-21 | Hiroki Kato | Process Management Apparatus, Computer Systems, Distributed Processing Method, and Computer Program |
| CN104243617A (en) * | 2014-10-14 | 2014-12-24 | 中国科学院信息工程研究所 | Task scheduling method and system facing mixed load in heterogeneous cluster |
| CN105487930A (en) * | 2015-12-01 | 2016-04-13 | 中国电子科技集团公司第二十八研究所 | Task optimization scheduling method based on Hadoop |
| CN106453497A (en) * | 2016-09-08 | 2017-02-22 | 天津海量信息技术股份有限公司 | Distributed cluster system and resource allocation method thereof |
| CN108255595A (en) * | 2018-01-16 | 2018-07-06 | 北京中关村科金技术有限公司 | A kind of dispatching method of data task, device, equipment and readable storage medium storing program for executing |
| CN108287756A (en) * | 2018-01-25 | 2018-07-17 | 联动优势科技有限公司 | A kind of method and device of processing task |
| CN108762896A (en) * | 2018-03-26 | 2018-11-06 | 福建星瑞格软件有限公司 | One kind being based on Hadoop cluster tasks dispatching method and computer equipment |
| CN109144699A (en) * | 2018-08-31 | 2019-01-04 | 阿里巴巴集团控股有限公司 | Distributed task dispatching method, apparatus and system |
| CN109634748A (en) * | 2018-12-12 | 2019-04-16 | 深圳前海微众银行股份有限公司 | Cluster resource dispatching method, device, equipment and computer readable storage medium |
-
2019
- 2019-02-22 CN CN201910134018.3A patent/CN110012062B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080046500A1 (en) * | 2004-06-15 | 2008-02-21 | Hiroki Kato | Process Management Apparatus, Computer Systems, Distributed Processing Method, and Computer Program |
| CN104243617A (en) * | 2014-10-14 | 2014-12-24 | 中国科学院信息工程研究所 | Task scheduling method and system facing mixed load in heterogeneous cluster |
| CN105487930A (en) * | 2015-12-01 | 2016-04-13 | 中国电子科技集团公司第二十八研究所 | Task optimization scheduling method based on Hadoop |
| CN106453497A (en) * | 2016-09-08 | 2017-02-22 | 天津海量信息技术股份有限公司 | Distributed cluster system and resource allocation method thereof |
| CN108255595A (en) * | 2018-01-16 | 2018-07-06 | 北京中关村科金技术有限公司 | A kind of dispatching method of data task, device, equipment and readable storage medium storing program for executing |
| CN108287756A (en) * | 2018-01-25 | 2018-07-17 | 联动优势科技有限公司 | A kind of method and device of processing task |
| CN108762896A (en) * | 2018-03-26 | 2018-11-06 | 福建星瑞格软件有限公司 | One kind being based on Hadoop cluster tasks dispatching method and computer equipment |
| CN109144699A (en) * | 2018-08-31 | 2019-01-04 | 阿里巴巴集团控股有限公司 | Distributed task dispatching method, apparatus and system |
| CN109634748A (en) * | 2018-12-12 | 2019-04-16 | 深圳前海微众银行股份有限公司 | Cluster resource dispatching method, device, equipment and computer readable storage medium |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110519381A (en) * | 2019-08-29 | 2019-11-29 | 深圳前海微众银行股份有限公司 | Processing method, device, terminal device and the storage medium of distributed task scheduling |
| CN110519381B (en) * | 2019-08-29 | 2023-08-25 | 深圳前海微众银行股份有限公司 | Distributed task processing method and device, terminal equipment and storage medium |
| CN111818159A (en) * | 2020-07-08 | 2020-10-23 | 腾讯科技（深圳）有限公司 | Data processing node management method, device, equipment and storage medium |
| CN111818159B (en) * | 2020-07-08 | 2024-04-05 | 腾讯科技（深圳）有限公司 | Management method, device, equipment and storage medium of data processing node |
| CN112114950A (en) * | 2020-09-21 | 2020-12-22 | 中国建设银行股份有限公司 | Task scheduling method and device and cluster management system |
| CN112181607A (en) * | 2020-11-06 | 2021-01-05 | 北京凌云雀科技有限公司 | Component processing method, device, equipment and storage medium |
| CN112817713A (en) * | 2021-01-27 | 2021-05-18 | 广州虎牙科技有限公司 | Job scheduling method and device and electronic equipment |
| CN112817713B (en) * | 2021-01-27 | 2023-10-13 | 广州虎牙科技有限公司 | Job scheduling method and device and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110012062B (en) | 2022-02-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110012062A (en) | A kind of multimachine room method for scheduling task, device and storage medium | |
| CN106802826B (en) | Service processing method and device based on thread pool | |
| CN112379995B (en) | DAG-based unitized distributed scheduling system and method | |
| CN104094235B (en) | Multithreading calculates | |
| US20060268967A1 (en) | Supplying instruction to operational stations | |
| CN109656782A (en) | Visual scheduling monitoring method, device and server | |
| WO2010067377A2 (en) | Method for reorganizing tasks for optimization of resources | |
| US20130227528A1 (en) | Paradigm for concurrency testcase generation | |
| CN112114973B (en) | Data processing method and device | |
| CN102541642B (en) | Task management method for enhancing real-time performance | |
| CN106062716B (en) | The method, apparatus and single task system of multitask are realized in single task system | |
| CN106502767A (en) | A kind of process management method and its device | |
| US20160170734A1 (en) | Revision Safe Upgrade in a Hybrid Cloud Landscape | |
| CN110611707A (en) | Task scheduling method and device | |
| CN103024791B (en) | Multi-service processing model and multi-service processing method in communication test system | |
| US20090119668A1 (en) | Dynamic feasibility analysis for event based programming | |
| CN102508717A (en) | Memory scheduling method and memory scheduling device for multi-core processor | |
| CN107832130A (en) | A kind of job stream scheduling of banking system performs method, apparatus and electronic equipment | |
| CN102662636B (en) | Two-layer virtual machines-based concurrent job scheduling method | |
| CN105474173B (en) | For reducing the parallel network application scheduling of power consumption | |
| CN107220114A (en) | Distributed resource scheduling method based on resource United Dispatching | |
| EP2128761A2 (en) | Scheduling for computing systems with multiple levels of determinism | |
| CN106445634A (en) | Container monitoring method and device | |
| CN109829005A (en) | A kind of big data processing method and processing device | |
| CN105843735B (en) | A kind of consumption method and device of terminal memory |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |