CN107807815A - The method and apparatus of distributed treatment task - Google Patents
The method and apparatus of distributed treatment task Download PDFInfo
- Publication number
- CN107807815A CN107807815A CN201710976299.8A CN201710976299A CN107807815A CN 107807815 A CN107807815 A CN 107807815A CN 201710976299 A CN201710976299 A CN 201710976299A CN 107807815 A CN107807815 A CN 107807815A
- Authority
- CN
- China
- Prior art keywords
- task
- deployment node
- compiling
- pending
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/40—Transformation of program code
- G06F8/41—Compilation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multi Processors (AREA)
- Stored Programmes (AREA)
Abstract
The present invention provides a kind of method and apparatus of distributed treatment task, wherein, this method includes：Receive the task requests of pending task；According to the task requests, selection performs the task compiling deployment node of the pending task, and the pending task is sent to task compiling deployment node and performed.The method and apparatus of distributed treatment task provided in an embodiment of the present invention, task compiling deployment node set by establishing the task compiling deployment node comprising maximum concurrent quantity handles multiple tasks and asks and can create task compiling deployment node in real time according to the task quantity of processing to meet the process demand to extensive request task, efficiently solve individual task compiling deployment node in face of handling extensive task when bottleneck problem.
Description
Technical field
The present invention relates to field of computer technology, more particularly to a kind of method and apparatus of distributed treatment task.
Background technology
Continuous integrating instrument can be used for the work that monitoring continues to repeat, including：Lasting software version issue/test item
The work that mesh and monitoring external call perform.Each task compiling deployment node can be used for realizing big rule in continuous integrating instrument
The processing of the deployment compiler task of mould.In the distributed task scheduling compiling deployment system of continuous integrating tools build, task compiling
Deployment node is easily trapped into when handling large scale deployment compiler task on performance bottleneck or individual task compiling deployment node
The problem of role pressure is excessive.
By taking Jenkins as an example, Jenkins is a kind of continuous integrating instrument based on Java exploitations.Jenkins distribution
The a whole set of environment of structure needs a central server (Jenkins Master nodes) and some test machine (Slave nodes)
Composition, wherein, Jenkins Master nodes are responsible for the scheduling of task and the preservation of metadata and scheduling result, Jenkins
Slave nodes are responsible for the specific execution of task, and return result to Jenkins Master nodes, same set of so as to allow
Code or project are compiled in different environment, disposed etc..
On Jenkins framework, following three kinds of forms in the prior art be present：
(1) Jenkins single cpu modes
As shown in figure 1, the pattern refers to that the Jenkins servers of deployment are not only Jenkins Master nodes and are also
Jenkins Slave nodes, Jenkins servers will perform these when receiving structure or deployment request in the machine
Business.
(2) single Jenkins Master and static multiple Jenkins Slave patterns are combined
As shown in Fig. 2 the pattern refers to dispose a Jenkins Master node and multiple Jenkins Slave sections
Point, these Jenkins Slave nodes are to create in advance, and are registered on Jenkins Master nodes and supply Jenkins
Master scheduling uses.When Jenkins Master receive request, Jenkins Slave can be selected, and task scheduling is arrived
Performed on Jenkins Slave, after the completion of execution, the metadata and result of task are stored on Jenkins Master.
(3) single Jenkins Master and the multiple Jenkins Slave patterns of dynamic are combined
As shown in figure 3, the pattern refers to dispose a Jenkins Master node and multiple DYNAMIC J enkins Slave
Node, Jenkins are created when Slave nodes are execution tasks, and are dynamically registered to Jenkins Master nodes
Upper dispatched for Jenkins Master uses.When Jenkins Master receive request, it is suitable to be created by K8s plug-in units
Jenkins Slave nodes, and will be performed in task scheduling to Jenkins Slave nodes, after the completion of execution, Jenkins
Slave nodes can be destroyed, and the metadata and result of task are stored on Jenkins Master.
However, have the following disadvantages in the prior art：
(1) on Jenkins single cpu modes
It is all because Jenkins single cpu modes are only suitable for handling small-scale task, therefore when task quantity increase
Waiting task will perform on a machine, thereby increase machine loading, ultimately result in server crash.
(2) on combining single Master and static multiple Jenkins Slave patterns
Although static Slave patterns can avoid what the considerable task in foregoing single cpu mode performed in same machine
Problem, but what the number of Slave nodes was still to determine, and increase Slave nodes are required for manual registration every time, therefore
Operation is very troublesome, and when task exceedes certain scale, the pressure on Master nodes can be very big.
(3) on gathering single Master and the multiple Jenkins Slave patterns of dynamic
Although dynamic creation Jenkins Slave patterns can save the trouble of manual registration, and Jenkins Slave
Node without existing all the time, only need to when there is task requests just needs, but this mode still can not solve to handle on a large scale
The problem of task, i.e., when task scale exceeds disposal ability (for example, more than 10000), now single Jenkins Master
Performance can be excessively poor, can not be worked so as to directly result in Jenkins Master nodes.
The content of the invention
In view of this, the embodiment of the present invention provides a kind of method and apparatus of distributed treatment task, to solve existing skill
Bottleneck problem when individual task compiling deployment node is in face of handling extensive task in art.
To achieve the above object, one side according to embodiments of the present invention, there is provided a kind of distributed treatment task
Method, this method include：
Receive the task requests of pending task；
According to the task requests, selection performs the task compiling deployment node of the pending task, and is treated described
Execution task sends to task compiling deployment node and performed.
Alternatively, this method also includes：Before the task requests of pending task are received, the task of predetermined number is created
Compiling deployment node, wherein, the predetermined number refers to the quantity of the maximum number of concurrent not less than the pending task.
Alternatively, in the step of creating the task compiling deployment node of predetermined number, including：Pass through cluster management system
Create the task compiling deployment node set of the task compiling deployment node comprising predetermined number.
Alternatively, this method also includes：After the task compiling deployment node of the predetermined number is created, described it will appoint
The information of business compiling deployment node is preserved into database, wherein, described information includes at least one of following content：Each task
IP address, running status, the task quantity for having performed completion task of the server of compiling deployment node, wherein the operation
State includes idle condition and working condition.
Alternatively, according to the task requests, selection performs the task compiling deployment node of the pending task, and will
The pending task, which is sent, to be compiled deployment node to the task and carries out execution and include：
After the task requests that pending task is compiler task are received, the task requests of the compiler task are believed
Breath preserves and selects to meet default bar from the database into the database, and according to the task requests of the compiler task
The task compiling deployment node of part, then sends the task requests of the compiler task to the meeting preparatory condition of the task
Compiling deployment node is performed.
Alternatively, according to the task requests, selection performs the task compiling deployment node of the pending task, and will
The pending task, which sends to the task to compile deployment node and performed, also to be included：
Before the task requests that pending task is compiler task are received, according to the task of the establishment task received
Request, the task request message of the establishment task is preserved into the database.
Alternatively, the meeting preparatory condition of the task is selected from the database according to the task requests of the compiler task
Compiling deployment node includes：
Obtain the list of the running status that is preserved in the database for the task compiling deployment node of idle condition, and from
The task compiling for having performed the task minimum number of completion task is selected to dispose node in the list；And
If there is no the task compiling deployment node that running status is idle condition in the database, pass through the collection
Cluster management system creates a new task compiling deployment node as the task compiling deployment node for performing the compiler task.
Alternatively, this method also includes：, will after selection performs the task compiling deployment node of the pending task
The running status of the task compiling deployment node is arranged to working condition；And
After the pending tasks carrying finishes, the object information of the pending task is preserved, and described will appoint
The running status of business compiling deployment node is arranged to idle condition.
The one side of the embodiment of the present invention, there is provided a kind of device of distributed treatment task, the device include：
Receiving module, for receiving the task requests of pending task；
Execution module, for according to the task requests, selection to perform the task compiling deployment section of the pending task
Point, and the pending task is sent to task compiling deployment node and performed.
Alternatively, the device also includes：Creation module, for before the task requests of pending task are received, creating
The task compiling deployment node of predetermined number, wherein, the predetermined number refer to not less than the pending task it is maximum simultaneously
Send out the quantity of number.
Alternatively, the creation module is additionally operable to：The task for including predetermined number is created by cluster management system
The task compiling deployment section set of compiling deployment node.
Alternatively, the device also includes：Memory module, for creating the task compiling deployment node of the predetermined number
Afterwards, the information that the task is compiled to deployment node is preserved into database, wherein, described information includes at least one of following
Content：IP address, running status, having performed completion task of the task of the server of each task compiling deployment node
Quantity, wherein the running status includes idle condition and working condition.
Alternatively, the execution module is additionally operable to：The task request message of the compiler task is preserved to the data
In storehouse, and the meeting preparatory condition of the task is selected to compile deployment from the database according to the task requests of the compiler task
Node, then the task requests of the compiler task are sent to the task compiling deployment node for meeting preparatory condition and carried out
Perform.
Alternatively, the execution module is additionally operable to：Before the task requests that pending task is compiler task are received,
According to the task requests of the establishment task received, the task request message of the establishment task is preserved to the database
In.
Alternatively, the execution module is additionally operable to：It is idle condition to obtain the running status preserved in the database
The list of task compiling deployment node, and select from the list to have performed the task of the task minimum number of completion task
Compiling deployment node；And
If there is no the task compiling deployment node that running status is idle condition in the database, pass through the collection
Cluster management system creates a new task compiling deployment node as the task compiling deployment node for performing the compiler task.
Alternatively, the device also includes：Setup module, for performing the task compiling portion of the pending task in selection
After affixing one's name to node, the running status that the task is compiled to deployment node is arranged to working condition；And at described pending
After business is finished, the object information of the pending task is preserved, and the task is compiled to the operation shape of deployment node
State is arranged to idle condition.
The embodiment of the present invention also provides a kind of terminal device of distributed treatment task, including：One or more processors；
Storage device, for storing one or more programs, when one or more of programs are held by one or more of processors
OK so that the method that one or more of processors realize the distributed treatment task such as the embodiment of the present invention.
The embodiment of the present invention also provides a kind of computer-readable medium, is stored thereon with computer program, described program quilt
The method that the distributed treatment task such as the embodiment of the present invention is realized during computing device.
The method and apparatus of distributed treatment task provided in an embodiment of the present invention, individual task compiling portion is can solve the problem that
Affix one's name to node in face of handle extensive task when bottleneck problem；Deployment is compiled by creating the task comprising maximum concurrent quantity
The task compiling deployment node set of node, so that the distributed system can have multiple tasks compiling deployment section
Point is asked in order to handle multiple tasks；By the way that the information of the task compiling deployment node of establishment is preserved into database, from
And it can handle and be easy to selection task compiling deployment node to perform waiting task according to the process demand of pending task；Pass through
Select the task compiling for meeting preparatory condition to dispose node from database according to the task requests of the compiler task, then will
The task requests of compiler task send to the task compiling deployment node for meeting preparatory condition and performed, full so as to select
The task compiling deployment node of sufficient condition performs corresponding pending task；Preserved by the information that task is compiled to deployment node
Into database, so as to realize the data persistence that deployment nodal information is compiled to task, to facilitate during subsequent treatment
Acquisition data and/or check data at any time；The running status preserved by obtaining in database is compiled for the task of idle condition
The list of deployment node is translated, and selects the task compiling deployment for having performed the task minimum number of completion task to save from list
Point；And if there is no the task compiling deployment node that running status is idle condition in database, then pass through cluster management system
System creates task compiling deployment node of the new task compiling deployment node as the execution compiler task, so as to
Select and meet that the task compiling deployment node of preparatory condition performs waiting task.
Further effect adds hereinafter in conjunction with embodiment possessed by above-mentioned non-usual optional mode
With explanation.
Brief description of the drawings
Technical scheme in order to illustrate the embodiments of the present invention more clearly, make required in being described below to embodiment
Accompanying drawing is briefly described, it should be apparent that, drawings in the following description are only some embodiments of the present invention, for
For those of ordinary skill in the art, on the premise of not paying creative work, other can also be obtained according to these accompanying drawings
Accompanying drawing.In the accompanying drawings：
Fig. 1 is the schematic diagram on the Jenkins single cpu modes in Jenkins framework in the prior art；
Fig. 2 is in the prior art on the single Master of combination in Jenkins framework and static multiple Slave patterns
Schematic diagram；
Fig. 3 is in the prior art on the single Master of combination in Jenkins framework and the multiple Slave patterns of dynamic
Schematic diagram；
Fig. 4 is the schematic diagram of the main flow of the method for distributed treatment task according to embodiments of the present invention；
Fig. 5 is the schematic diagram of the main modular of the device of distributed treatment task according to embodiments of the present invention；
Fig. 6 is adapted for the structural representation of the computer system of the terminal device for realizing the embodiment of the present invention.
Embodiment
The one exemplary embodiment of the present invention is explained below in conjunction with accompanying drawing, including the various of the embodiment of the present invention
Details should think them only exemplary to help understanding.Therefore, those of ordinary skill in the art should recognize
Arrive, various changes and modifications can be made to the embodiments described herein, without departing from scope and spirit of the present invention.Together
Sample, for clarity and conciseness, the description to known function and structure is eliminated in following description.
Herein, it is to be understood that in involved term：
Jenkins：It is an open source software project, is a kind of continuous integrating instrument based on Java exploitations, for monitoring
Continue the work repeated, it is desirable to provide the easy-to-use software platform of an opening, the continuous integrating of software is become possibility.
Kubernetes：Referred to as k8s, it is the Open Source Platform for automating container operation, is that one kind is used to realize containerization
The system of application cluster management, these operations include deployment, extended between scheduling and node cluster.Kubernetes is not only supported
Docker, Rocket is also supported, this is another container technique.
MySQL：It is a Relational DBMS, is developed by MySQL AB companies of Sweden, belonged at present
Product under Oracle.MySQL is one of most popular Relational DBMS, and in terms of WEB application, MySQL is
Best RDBMS (Relational Database Management System, relational database management system) applications are soft
Part.
Illustrative methods
As shown in Figure 4, there is provided a kind of method of distributed treatment task, this method include：
Step S401：Receive the task requests of pending task；
Step S402：According to the task requests, selection performs the task compiling deployment node of the pending task
Jenkins Master, and the pending task is sent to task compiling deployment node and performed.It is of the invention real
Apply in example, task compiling deployment node can be, but not limited to be the Jenkins Master in Jenkins continuous integrating instruments.With
Under by taking Jenkins Master as an example, carry out task compiling deployment node establishment and task processing explanation.
Alternatively, this method also includes：Before the task requests of pending task are received, predetermined number is created
Jenkins Master, wherein, the predetermined number refers to the quantity of the maximum number of concurrent not less than the pending task.
Alternatively, in the step of creating the Jenkins Master of predetermined number, including：
The Jenkins Master of predetermined number Jenkins is included by Kubernetes system creations
Master gathers.
Alternatively, this method also includes：After the Jenkins Master of the predetermined number are created, by described in
Jenkins Master information is preserved into database, wherein, described information includes at least one of following content：Each
IP address, running status, the task quantity for having performed completion task of Jenkins Master server, wherein the fortune
Row state includes idle condition and working condition.
Alternatively, according to the task requests, selection performs the Jenkins Master of the pending task, and by institute
State pending task and send and carry out execution to the Jenkins Master and include：It is compiler task receiving pending task
Task requests after, the task request message of the compiler task is preserved into the database, and according to the compiling
The task requests of task select the Jenkins Master for meeting preparatory condition from the database, then appoint the compiling
The task requests of business send to the Jenkins Master for meeting preparatory condition and performed.
Alternatively, according to the task requests, selection performs the Jenkins Master of the pending task, and by institute
Stating pending task and sending to the Jenkins Master to be performed also includes：
Before the task requests that pending task is compiler task are received, according to the task of the establishment task received
Request, the task request message of the establishment task is preserved into the database.
Alternatively, selected to meet preparatory condition from the database according to the task requests of the compiler task
Jenkins Master include：
Obtain the list of the running status that is preserved in the database for the Jenkins Master of idle condition, and from institute
State the Jenkins Master that the task minimum number for having performed completion task is selected in list；And
If there is no the Jenkins Master that running status is idle condition in the database, by described
The new Jenkins Master of Kubernetes system creations one are as the Jenkins Master for performing the compiler task.
Alternatively, this method also includes：After the Jenkins Master that selection performs the pending task, by institute
The running status for stating Jenkins Master is arranged to working condition；And
After the pending tasks carrying finishes, the object information of the pending task is preserved, and by described in
Jenkins Master running status is arranged to idle condition.
Embodiment
The present invention is specifically described with reference to a specific embodiment, however, it should be noted that the specific implementation
Example merely to preferably description the present invention, do not form inappropriate limitation of the present invention.
First, according to the process demand of task, the Jenkins of predetermined number is included by Kubernetes system creations
Master Jenkins Master set (Jenkins Master Pool), it is every during the Jenkins Master are gathered
Individual Jenkins Master information is saved in the Server tables of Mysql databases, wherein, the information recorded in Server tables
Comprise at least：The IP address of each Master server, the state of the Master whether be the idle and Master
Perform one or more of number of completion task.
Then, after application programming interfaces (API) receive task requests, the request type of the task requests is judged.
If the task requests are the task requests of " establishment task ", one is inserted in the Job tables into Mysql databases
Bar records.Specifically, because task can be stored in local by primary Jenkins by way of text, therefore only originally
The Jenkins on ground can just be inquired about and be obtained this task, and the purpose for inserting record in embodiments of the present invention is this
Mission bit stream is preserved to database, and this can be obtained from the database when being compiled in order to future to the task
Business, then one Jenkins Master execution of selection should from Jenkins Master set (Jenkins Master Pool)
Task；
If the task requests are the request of " compiler task ", the Build tables insertion one first into Mysql databases
The record of bar structure.Specifically, structure is recorded to the purpose for being saved in Mysql databases to be to look into after user for convenience
These records are seen, even if the Jenkins Master for performing this structure record at that time are destroyed, we still can be from
Check that this is recorded in Mysql databases.Then, from then from Server tables one Master for meeting preparatory condition of selection comes
Manage the task.Specifically, the preparatory condition is as follows：
First, the Master lists of free time are filtered out from Server tables；
Second, in the Jenkins Master lists of free time, being sorted according to the number for having performed completion task, from
It is middle to select the minimum Jenkins Master of the number for having performed completion task as the Jenkins for meeting preparatory condition
Master；
3rd, if there is no the Jenkins Master of free time in Server tables, pass through Kubernetes systems dynamic
A new Jenkins Master is created, the new Jenkins Master are handled into the task, and in Server tables
Insert this new Jenkins Master record.
That is, when the task quantity that the task requests received include the pending task is more than described preset
During quantity, then by the Kubernetes systems in the continuous establishment task quantity of the Jenkins Master aggregate relays
With the Jenkins Master of the difference amount of the predetermined number.
Finally, wait after finding the Jenkins Master for meeting preparatory condition, the Master in Server tables is recorded
State in information is arranged to " working condition ", and after treating that the tasks carrying finishes, the object information of the task is saved in
In Build tables, then the state of the Jenkins Master is arranged to " idle condition ", used for follow-up.
Can be by large-scale task by single by the method for distributed treatment task provided in an embodiment of the present invention
Jenkins Master nodes are preposition into database, and these tasks are uniformly then dispatched into each Jenkins Master
On node, the form of the single extensive task of Jenkins Master node processings is converted into multiple Jenkins Master and saved
Point handles small-scale task simultaneously, efficiently solves the bottleneck run into during single Jenkins Master node processing multitasks
Problem.
In addition, in " compiling " or " deployment " task of execution, the distributed system only accordingly please of receiving of needs processing
Ask, without tackling and judging the source of proposition waiting task, so as to which waiting task and Jenkins services be decoupled.
As shown in figure 5, it is the schematic diagram of the main modular of the device of distributed treatment task according to embodiments of the present invention；
Wherein, the device includes：
Receiving module 501, for receiving the task requests of pending task；
Execution module 502, for performing the Jenkins of the pending task according to the task requests, selection
Master, and the pending task is sent to the Jenkins Master and performed.
Alternatively, the device also includes：Creation module, for before the task requests of pending task are received, creating
The Jenkins Master of predetermined number, wherein, the predetermined number refers to concurrent not less than the maximum of the pending task
Several quantity.
Alternatively, the creation module is additionally operable to：Included by Kubernetes system creations described in predetermined number
Jenkins Master Jenkins Master set.
Alternatively, the device also includes：Memory module, for create the predetermined number Jenkins Master it
Afterwards, the information of the Jenkins Master is preserved into database, wherein, described information is included at least one of following
Hold：IP address, running status, the number of tasks for having performed completion task of each Jenkins Master server
Amount, wherein the running status includes idle condition and working condition.
Alternatively, the execution module is additionally operable to：The task request message of the compiler task is preserved to the data
In storehouse, and the Jenkins for selecting to meet preparatory condition from the database according to the task requests of the compiler task
Master, then the task requests of the compiler task are sent to the Jenkins Master for meeting preparatory condition and carried out
Perform.
Alternatively, the execution module is additionally operable to：Before the task requests that pending task is compiler task are received,
According to the task requests of the establishment task received, the task request message of the establishment task is preserved to the database
In.
Alternatively, the execution module is additionally operable to：It is idle condition to obtain the running status preserved in the database
Jenkins Master list, and select from the list to have performed the task minimum number of completion task
Jenkins Master；And
If there is no the Jenkins Master that running status is idle condition in the database, by described
The new Jenkins Master of Kubernetes system creations one are as the Jenkins Master for performing the compiler task.
Alternatively, the device also includes：Setup module, for performing the Jenkins of the pending task in selection
After Master, the running status of the Jenkins Master is arranged to working condition；And in the pending task
After being finished, the object information of the pending task is preserved, and the running status of the Jenkins Master is set
It is set to idle condition.
Because the device of distributed treatment task provided by the invention is device corresponding to the above method, therefore do not go to live in the household of one's in-laws on getting married herein
State.
Below with reference to Fig. 6, it illustrates suitable for for realizing the computer system 600 of the terminal device of the embodiment of the present invention
Structural representation.Terminal device shown in Fig. 6 is only an example, to the function of the embodiment of the present invention and should not use model
Shroud carrys out any restrictions.
As shown in fig. 6, computer system 600 includes CPU (CPU) 601, it can be read-only according to being stored in
Program in memory (ROM) 602 or be loaded into program in random access storage device (RAM) 603 from storage part 608 and
Perform various appropriate actions and processing.In RAM 603, also it is stored with system 600 and operates required various programs and data.
CPU 601, ROM 602 and RAM 603 are connected with each other by bus 604.Input/output (I/O) interface 605 is also connected to always
Line 604.
I/O interfaces 605 are connected to lower component：Importation 606 including keyboard, mouse etc.；Penetrated including such as negative electrode
The output par, c 607 of spool (CRT), liquid crystal display (LCD) etc. and loudspeaker etc.；Storage part 608 including hard disk etc.；
And the communications portion 609 of the NIC including LAN card, modem etc..Communications portion 609 via such as because
The network of spy's net performs communication process.Driver 610 is also according to needing to be connected to I/O interfaces 605.Detachable media 611, such as
Disk, CD, magneto-optic disk, semiconductor memory etc., it is arranged on as needed on driver 610, in order to read from it
Computer program be mounted into as needed storage part 608.
Especially, according to embodiment disclosed by the invention, may be implemented as counting above with reference to the process of flow chart description
Calculation machine software program.For example, embodiment disclosed by the invention includes a kind of computer program product, it includes being carried on computer
Computer program on computer-readable recording medium, the computer program include the program code for being used for the method shown in execution flow chart.
In such embodiment, the computer program can be downloaded and installed by communications portion 609 from network, and/or from can
Medium 611 is dismantled to be mounted.When the computer program is performed by CPU (CPU) 601, system of the invention is performed
The above-mentioned function of middle restriction.
It should be noted that the computer-readable medium shown in the present invention can be computer-readable signal media or meter
Calculation machine readable storage medium storing program for executing either the two any combination.Computer-readable recording medium for example can be --- but not
Be limited to --- electricity, magnetic, optical, electromagnetic, system, device or the device of infrared ray or semiconductor, or it is any more than combination.Meter
The more specifically example of calculation machine readable storage medium storing program for executing can include but is not limited to：Electrical connection with one or more wires, just
Take formula computer disk, hard disk, random access storage device (RAM), read-only storage (ROM), erasable type and may be programmed read-only storage
Device (EPROM or flash memory), optical fiber, portable compact disc read-only storage (CD-ROM), light storage device, magnetic memory device,
Or above-mentioned any appropriate combination.In the present invention, computer-readable recording medium can any include or store journey
The tangible medium of sequence, the program can be commanded the either device use or in connection of execution system, device.And at this
In invention, computer-readable signal media can include in a base band or as carrier wave a part propagation data-signal,
Wherein carry computer-readable program code.The data-signal of this propagation can take various forms, including but unlimited
In electromagnetic signal, optical signal or above-mentioned any appropriate combination.Computer-readable signal media can also be that computer can
Any computer-readable medium beyond storage medium is read, the computer-readable medium, which can send, propagates or transmit, to be used for
By instruction execution system, device either device use or program in connection.Included on computer-readable medium
Program code can be transmitted with any appropriate medium, be included but is not limited to：Wirelessly, electric wire, optical cable, RF etc., or it is above-mentioned
Any appropriate combination.
Flow chart and block diagram in accompanying drawing, it is illustrated that according to the system of various embodiments of the invention, method and computer journey
Architectural framework in the cards, function and the operation of sequence product.At this point, each square frame in flow chart or block diagram can generation
The part of one module of table, program segment or code, a part for above-mentioned module, program segment or code include one or more
For realizing the executable instruction of defined logic function.It should also be noted that some as replace realization in, institute in square frame
The function of mark can also be with different from the order marked in accompanying drawing generation.For example, two square frames succeedingly represented are actual
On can perform substantially in parallel, they can also be performed in the opposite order sometimes, and this is depending on involved function.Also
It is noted that the combination of each square frame and block diagram in block diagram or flow chart or the square frame in flow chart, can use and perform rule
Fixed function or the special hardware based system of operation are realized, or can use the group of specialized hardware and computer instruction
Close to realize.
Being described in module involved in the embodiment of the present invention can be realized by way of software, can also be by hard
The mode of part is realized.Described module can also be set within a processor, for example, can be described as：A kind of processor bag
Include receiving module 501, execution module 502.Wherein, the title of these modules is not formed to the module in itself under certain conditions
Restriction.
As on the other hand, present invention also offers a kind of computer-readable medium, the computer-readable medium can be
Included in equipment described in above-described embodiment；Can also be individualism, and without be incorporated the equipment in.Above-mentioned calculating
Machine computer-readable recording medium carries one or more program, when said one or multiple programs are performed by the equipment, makes
Obtaining the equipment includes：
Step S401：Receive the task requests of pending task；
Step S402：According to the task requests, selection performs the Jenkins Master of pending task, and will wait to hold
Row task sends to Jenkins Master and performed.
The method and apparatus of distributed treatment task provided in an embodiment of the present invention, individual task compiling portion is can solve the problem that
Affix one's name to node in face of handle extensive task when bottleneck problem；Deployment is compiled by creating the task comprising maximum concurrent quantity
The task compiling deployment node set of node, so that the distributed system can have multiple tasks compiling deployment section
Point is asked in order to handle multiple tasks；By the way that the information of the task compiling deployment node of establishment is preserved into database, from
And it can handle and be easy to selection task compiling deployment node to perform waiting task according to the process demand of pending task；Pass through
Select the task compiling for meeting preparatory condition to dispose node from database according to the task requests of the compiler task, then will
The task requests of compiler task send to the task compiling deployment node for meeting preparatory condition and performed, full so as to select
The task compiling deployment node of sufficient condition performs corresponding pending task；Preserved by the information that task is compiled to deployment node
Into database, so as to realize the data persistence that deployment nodal information is compiled to task, to facilitate during subsequent treatment
Acquisition data and/or check data at any time；The running status preserved by obtaining in database is compiled for the task of idle condition
The list of deployment node is translated, and selects the task compiling deployment for having performed the task minimum number of completion task to save from list
Point；And if there is no the task compiling deployment node that running status is idle condition in database, then pass through cluster management system
System creates task compiling deployment node of the new task compiling deployment node as the execution compiler task, so as to
Select and meet that the task compiling deployment node of preparatory condition performs waiting task.
Above-mentioned embodiment, does not form limiting the scope of the invention.Those skilled in the art should be bright
It is white, depending on design requirement and other factors, various modifications, combination, sub-portfolio and replacement can occur.It is any
Modifications, equivalent substitutions and improvements made within the spirit and principles in the present invention etc., should be included in the scope of the present invention
Within.
Claims (18)
- A kind of 1. method of distributed treatment task, it is characterised in that including：Receive the task requests of pending task；According to the task requests, selection performs the task compiling deployment node of the pending task, and will be described pending Task sends to task compiling deployment node and performed.
- 2. according to the method for claim 1, it is characterised in that also include：Before the task requests of pending task are received, the task compiling deployment node of predetermined number is created, wherein, it is described pre- If quantity refers to the quantity of the maximum number of concurrent not less than the pending task.
- 3. according to the method for claim 2, it is characterised in that in the step for the task compiling deployment node for creating predetermined number In rapid, including：The task compiling deployment node set of the task compiling deployment node comprising predetermined number is created by cluster management system.
- 4. according to the method for claim 3, it is characterised in that also include：After the task compiling deployment node of the predetermined number is created, the information that the task is compiled to deployment node preserves Into database, wherein, described information includes at least one of following content：The IP of the server of each task compiling deployment node Address, running status, the task quantity for having performed completion task, wherein the running status includes idle condition and work shape State.
- 5. according to the method described in claim 3, it is characterised in that according to the task requests, wait to hold described in selection execution The task compiling deployment node of row task, and the pending task is sent to task compiling deployment node and performed Including：After the task requests that pending task is compiler task are received, the task request message of the compiler task is protected Deposit into the database, and select to meet preparatory condition from the database according to the task requests of the compiler task Task compiling deployment node, then the task requests of the compiler task are sent to the meeting preparatory condition of the task compile Deployment node is performed.
- 6. according to the method for claim 5, it is characterised in that according to the task requests, selection performs described pending The task compiling deployment node of task, and the pending task is sent to task compiling deployment node and perform also Including：, please according to the task of the establishment task received before the task requests that pending task is compiler task are received Ask, the task request message of the establishment task is preserved into the database.
- 7. according to the method described in claim 5, it is characterised in that according to the task requests of the compiler task from the number Meeting the task compiling deployment node of preparatory condition according to selection in storehouse includes：Obtain the list of the running status that is preserved in the database for the task compiling deployment node of idle condition, and from described The task compiling for having performed the task minimum number of completion task is selected to dispose node in list；AndIf there is no the task compiling deployment node that running status is idle condition in the database, by described The new task compiling deployment node of Kubernetes system creations is as the task compiling deployment node for performing the compiler task.
- 8. according to method according to any one of claims 1 to 7, it is characterised in that also include：After selection performs the task compiling deployment node of the pending task, the task is compiled to the fortune of deployment node Row state is arranged to working condition；AndAfter the pending tasks carrying finishes, the object information of the pending task is preserved, and the task is compiled The running status for translating deployment node is arranged to idle condition.
- A kind of 9. device of distributed treatment task, it is characterised in that including：Receiving module, for receiving the task requests of pending task；Execution module, for disposing node according to the task compiling of the task requests, the selection execution pending task, and The pending task is sent to task compiling deployment node and performed.
- 10. device according to claim 9, it is characterised in that also include：Creation module, for before the task requests of pending task are received, creating the task compiling deployment section of predetermined number Point, wherein, the predetermined number refers to the quantity of the maximum number of concurrent not less than the pending task.
- 11. device according to claim 10, it is characterised in that the creation module is additionally operable to：The task compiling deployment node set of the task compiling deployment node comprising predetermined number is created by cluster management system.
- 12. device according to claim 11, it is characterised in that also include：Memory module, for after the task compiling deployment node of the predetermined number being created, the task being compiled and disposed The information of node is preserved into database, wherein, described information includes at least one of following content：Each task compiling portion IP address, running status, the task quantity for having performed completion task of the server of node are affixed one's name to, wherein the running status bag Include idle condition and working condition.
- 13. according to the device described in claim 11, it is characterised in that the execution module is additionally operable to：The task request message of the compiler task is preserved into the database, and please according to the task of the compiler task Ask and the task compiling deployment node for meeting preparatory condition is selected from the database, then please by the task of the compiler task Ask to send to the task compiling deployment node for meeting preparatory condition and performed.
- 14. according to the device described in claim 13, it is characterised in that the execution module is additionally operable to：, please according to the task of the establishment task received before the task requests that pending task is compiler task are received Ask, the task request message of the establishment task is preserved into the database.
- 15. according to the device described in claim 11, it is characterised in that the execution module is additionally operable to：Obtain the list of the running status that is preserved in the database for the task compiling deployment node of idle condition, and from described The task compiling for having performed the task minimum number of completion task is selected to dispose node in list；AndIf there is no the task compiling deployment node that running status is idle condition in the database, pass through the cluster pipe The new task compiling deployment node of system creation is managed as the task compiling deployment node for performing the compiler task.
- 16. the device according to any one of claim 9~15, it is characterised in that also include：Setup module, after node is disposed in the task compiling for performing the pending task in selection, the task is compiled The running status for translating deployment node is arranged to working condition；AndAfter the pending tasks carrying finishes, the object information of the pending task is preserved, and the task is compiled The running status for translating deployment node is arranged to idle condition.
- A kind of 17. terminal device of distributed treatment task, it is characterised in that including：One or more processors；Storage device, for storing one or more programs,When one or more of programs are by one or more of computing devices so that one or more of processors are real The now method as described in any in claim 1-8.
- 18. a kind of computer-readable medium, is stored thereon with computer program, it is characterised in that described program is held by processor The method as described in any in claim 1-8 is realized during row.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710976299.8A CN107807815B (en) | 2017-10-19 | 2017-10-19 | Method and device for processing tasks in distributed mode |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710976299.8A CN107807815B (en) | 2017-10-19 | 2017-10-19 | Method and device for processing tasks in distributed mode |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107807815A true CN107807815A (en) | 2018-03-16 |
| CN107807815B CN107807815B (en) | 2020-11-24 |
Family
ID=61592812
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710976299.8A Active CN107807815B (en) | 2017-10-19 | 2017-10-19 | Method and device for processing tasks in distributed mode |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107807815B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108681777A (en) * | 2018-05-07 | 2018-10-19 | 北京京东尚科信息技术有限公司 | A kind of method and apparatus of the machine learning program operation based on distributed system |
| CN109032784A (en) * | 2018-08-07 | 2018-12-18 | 郑州云海信息技术有限公司 | A kind of multi-task parallel construction method and device |
| CN109388405A (en) * | 2018-10-25 | 2019-02-26 | 北京大米未来科技有限公司 | A kind of task processing method, device, electronic equipment and medium |
| CN109947452A (en) * | 2019-03-26 | 2019-06-28 | 南京联创信息科技有限公司 | A kind of Kubernetes container platform application update method |
| CN111736810A (en) * | 2020-08-18 | 2020-10-02 | 四川新网银行股份有限公司 | Method for dynamically creating node of docker container to execute task based on jenkins real-time task |
| CN112019581A (en) * | 2019-05-30 | 2020-12-01 | 华为技术有限公司 | Method and device for scheduling task processing entities |
| CN112183053A (en) * | 2020-10-10 | 2021-01-05 | 湖南快乐阳光互动娱乐传媒有限公司 | Data processing method and device |
| CN112559022A (en) * | 2020-12-16 | 2021-03-26 | 同盾控股有限公司 | Jenkins high-availability system and method |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110271263A1 (en) * | 2010-04-29 | 2011-11-03 | International Business Machines Corporation | Compiling Software For A Hierarchical Distributed Processing System |
| CN104079630A (en) * | 2014-06-09 | 2014-10-01 | 中国建设银行股份有限公司 | Business server side load balancing method, client side, server side and system |
| CN104794095A (en) * | 2014-01-16 | 2015-07-22 | 华为技术有限公司 | Distributed computation processing method and device |
| CN105760459A (en) * | 2016-02-04 | 2016-07-13 | 四川嘉宝资产管理集团股份有限公司 | Distributed data processing system and method |
| CN106909451A (en) * | 2017-02-28 | 2017-06-30 | 郑州云海信息技术有限公司 | A kind of distributed task dispatching system and method |
| CN107026762A (en) * | 2017-05-24 | 2017-08-08 | 郑州云海信息技术有限公司 | A kind of disaster tolerance system and method based on distributed type assemblies |
| US20170286144A1 (en) * | 2016-03-31 | 2017-10-05 | International Business Machines Corporation | Software-defined storage cluster unified frontend |
-
2017
- 2017-10-19 CN CN201710976299.8A patent/CN107807815B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110271263A1 (en) * | 2010-04-29 | 2011-11-03 | International Business Machines Corporation | Compiling Software For A Hierarchical Distributed Processing System |
| CN104794095A (en) * | 2014-01-16 | 2015-07-22 | 华为技术有限公司 | Distributed computation processing method and device |
| CN104079630A (en) * | 2014-06-09 | 2014-10-01 | 中国建设银行股份有限公司 | Business server side load balancing method, client side, server side and system |
| CN105760459A (en) * | 2016-02-04 | 2016-07-13 | 四川嘉宝资产管理集团股份有限公司 | Distributed data processing system and method |
| US20170286144A1 (en) * | 2016-03-31 | 2017-10-05 | International Business Machines Corporation | Software-defined storage cluster unified frontend |
| CN106909451A (en) * | 2017-02-28 | 2017-06-30 | 郑州云海信息技术有限公司 | A kind of distributed task dispatching system and method |
| CN107026762A (en) * | 2017-05-24 | 2017-08-08 | 郑州云海信息技术有限公司 | A kind of disaster tolerance system and method based on distributed type assemblies |
Non-Patent Citations (2)
| Title |
|---|
| JUYONG PARK: "Distribution of node characteristics in complex networks", 《THE NATIONAL ACADEMY OF SCIENCES OF THE USA》 * |
| LANSEYOUHU: "IncrediBuild分布式编译环境搭建", 《HTTPS://BLOG.CSDN.NET/LANSEYOUHU/ARTICLE/DETAILS/12356503》 * |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108681777A (en) * | 2018-05-07 | 2018-10-19 | 北京京东尚科信息技术有限公司 | A kind of method and apparatus of the machine learning program operation based on distributed system |
| CN108681777B (en) * | 2018-05-07 | 2021-07-20 | 北京京东尚科信息技术有限公司 | Method and device for running machine learning program based on distributed system |
| CN109032784A (en) * | 2018-08-07 | 2018-12-18 | 郑州云海信息技术有限公司 | A kind of multi-task parallel construction method and device |
| CN109388405A (en) * | 2018-10-25 | 2019-02-26 | 北京大米未来科技有限公司 | A kind of task processing method, device, electronic equipment and medium |
| CN109947452A (en) * | 2019-03-26 | 2019-06-28 | 南京联创信息科技有限公司 | A kind of Kubernetes container platform application update method |
| CN112019581A (en) * | 2019-05-30 | 2020-12-01 | 华为技术有限公司 | Method and device for scheduling task processing entities |
| CN112019581B (en) * | 2019-05-30 | 2022-02-25 | 华为技术有限公司 | Method and device for scheduling task processing entities |
| CN111736810A (en) * | 2020-08-18 | 2020-10-02 | 四川新网银行股份有限公司 | Method for dynamically creating node of docker container to execute task based on jenkins real-time task |
| CN112183053A (en) * | 2020-10-10 | 2021-01-05 | 湖南快乐阳光互动娱乐传媒有限公司 | Data processing method and device |
| CN112559022A (en) * | 2020-12-16 | 2021-03-26 | 同盾控股有限公司 | Jenkins high-availability system and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107807815B (en) | 2020-11-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107807815A (en) | The method and apparatus of distributed treatment task | |
| CN102693183B (en) | Method and system for realizing automatic software testing | |
| CN102880503B (en) | Data analysis system and data analysis method | |
| CN102576354B (en) | Support that difference disposes the extensible framework of framework | |
| CN102891873B (en) | Method for storing log data and log data storage system | |
| CN104572122A (en) | Software application data generating device and method | |
| US20080222634A1 (en) | Parallel processing for etl processes | |
| CN107203424A (en) | A kind of method and apparatus that deep learning operation is dispatched in distributed type assemblies | |
| CN109034396A (en) | Method and apparatus for handling the deep learning operation in distributed type assemblies | |
| CN107943577A (en) | Method and apparatus for scheduler task | |
| CN106874189B (en) | Method for realizing automatic test system of power grid real-time database system | |
| CN105338045A (en) | Cloud computing resource processing device, method and cloud computing system | |
| CN107301179A (en) | The method and apparatus of data base read-write separation | |
| CN106874109A (en) | A kind of distributed job distribution processing method and system | |
| JP2004529412A (en) | Method and system for incorporating legacy applications into a distributed data processing environment | |
| CN107133175A (en) | A kind of software test platform based on cloud computing | |
| CN109800081A (en) | A kind of management method and relevant device of big data task | |
| CN100407663C (en) | Universal testing system and method for telecommunication intelligent service | |
| CN109814840A (en) | A kind of customizable collaborative design and development platform for supporting resource quickly to apply and method | |
| CN113379177A (en) | Task scheduling system and method | |
| Brown et al. | Gridnexus: A grid services scientific workflow system | |
| Ivanov et al. | A hot decomposition procedure: Operational monolith system to microservices | |
| CN107766497A (en) | The method and terminal of Data Collection based on container | |
| CN104796465B (en) | Cloud platform method for processing business and system | |
| CN108959309B (en) | Method and device for data analysis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |