CN107480041A - The task automation method of testing and system of a kind of big data - Google Patents
The task automation method of testing and system of a kind of big data Download PDFInfo
- Publication number
- CN107480041A CN107480041A CN201610404001.1A CN201610404001A CN107480041A CN 107480041 A CN107480041 A CN 107480041A CN 201610404001 A CN201610404001 A CN 201610404001A CN 107480041 A CN107480041 A CN 107480041A
- Authority
- CN
- China
- Prior art keywords
- task
- cluster
- scheduling queue
- execution
- iterative
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3688—Test management for test execution, e.g. scheduling of test suites
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses the task automation method of testing and system of a kind of big data, the scheduled date, which can be obtained, to be needed to be submitted to the task of cluster execution, be added into cluster treats that scheduling queue waits scheduling to perform, relative in the prior art, tester submits the mode of task one by one manually, the automation submission of the realizing the scheduled date of the task of the present embodiment and automated execution, effectively reduce test assignment amount and the cumbersome degree of task, simplify manual operation, the cost of labor needed for test is saved, improve testing efficiency, the present invention can be when considerable task needs to submit, effectively reduce artificial workload, ensure testing efficiency, the big data that is particularly suitable for use in field.
Description
Technical field
The present invention relates to big data field, and in particular to the task automation method of testing and system of a kind of big data.
Background technology
With smart mobile phone, the popularization of tablet personal computer, also notebook computer, increasing user access network, get over
Carry out more data to produce, in internet industry, what " big data " was is such a phenomenon：Internet firm is in daily operation
Middle generation, the user network behavioral data of accumulation, the amount of these data is very huge, so that need to use P (100 T), E (100
Ten thousand T) or Z (1,000,000,000 T) weigh, horizontal with current technology, few single computers can handle so big number
According to amount, also few single storage facilities can have so Large Copacity.At this time, it is necessary to use cluster to carry out big data
Processing.Cluster is one group of computer interconnected independently of each other, by express network, and they constitute a group, and with single
The pattern of system is managed.When one client interacts with cluster, cluster shows as an independent server.Cluster with
The data mining of high speed and analysis ability are famous, and mass data can be handled.
Processing of the cluster to mass data means the processing and submission of considerable task.In the prior art, typically by testing
Personnel submit pending task into group system, and afterwards, group system can be according to the order of the submission time of task successively
Task is submitted in cluster and performed., it is necessary to manage and submit substantial amounts of task for tester, workload is heavy, institute
There can be the problem of task is cumbersome, workload is big, testing efficiency is low in a manner of manually submitting task in the prior art.
The content of the invention
The main technical problem to be solved in the present invention is to provide a kind of task automation method of testing of big data and is
System, solves that the test assignment as caused by manually submitting considerable task is cumbersome in the prior art, workload is big, testing efficiency is low asks
Topic.
In order to solve the above technical problems, the embodiment of the present invention provides a kind of task automation method of testing of big data, bag
Include：
Execution date is configured for task, obtains the task list for needing to be submitted to cluster execution in the scheduled date；
Task list addition cluster is treated into scheduling queue；
Treating in scheduling queue for task is submitted to be performed to cluster.
In order to solve the above technical problems, the embodiment of the present invention also provides a kind of task automation test system of big data,
Including：
Data management module, for configuring execution date for task, obtain needs to be submitted to cluster execution in the scheduled date
Task list；
Task management module, for task list addition cluster to be treated into scheduling queue；
Task submits execution module, for submitting treating in scheduling queue for task to be performed to cluster.
Using the task automation method of testing and system of the big data of the present invention, there can be substantial amounts of task to need to transport
When row, the acquisition scheduled date needs to be submitted to the task of cluster execution, and be added into cluster treats that scheduling queue waits tune
Degree performs, and relative in the prior art, tester submits the mode of task one by one manually, and the present embodiment can be realized largely
The automation submission of task and automated executions, test assignment amount and the cumbersome degree of task are effectively reduced, simplifies manual operation,
The cost of labor needed for test has been saved, has improved testing efficiency.
Brief description of the drawings
Fig. 1 is a kind of flow chart of the task automation method of testing for big data that the embodiment of the present invention one provides；
Fig. 2 is a kind of flow chart of the task automation method of testing for big data that the embodiment of the present invention two provides；
Fig. 3 is a kind of module diagram of the task automation test system for big data that the embodiment of the present invention three provides.
Embodiment
The present invention is described in further detail below by embodiment combination accompanying drawing.
Embodiment one：
Referring to Fig. 1, the present embodiment proposes a kind of task automation method of testing of big data, including：
S101, it is that task configures execution date, obtains the task list for needing to be submitted to cluster execution in the scheduled date；
S102, task list is added into cluster treat scheduling queue；
S103, submission treat that the task in scheduling queue performs to cluster.
It is in order to determine each task specific execution date, as rear for task configuration execution date in step S101
The continuous upper foundation of task for obtaining the scheduled date, after execution date is configured for task, task could be obtained according to the scheduled date
List.Here scheduled date can freely be set by tester, can also be submitted to appointing for cluster execution as needed by system
It can be some specific date that the actual conditions of business set the scheduled date automatically, such as on October 3rd, 2016 or one
The section time, such as March 5, to April 3,8 points to 12 points of March 5 day etc., the present embodiment is not limited in this respect.Obtain and specify day
Need to be submitted to the task list that cluster performs in phase；Actually execution date is the task list of scheduled date, this
The needing to be submitted to the scheduled date of cluster execution of the task is comprised at least in business list.
It is envisioned that in the environment of big data, cluster needs the task substantial amounts performed, and single storage is situated between
Matter, which is followed the example of, meets storage demand, so the task of the present embodiment can be stored in multiple storage mediums, is obtained when in step S101
Fetching fix the date execution task list when, it may be necessary to from multiple storage mediums obtain the scheduled date need to be submitted to cluster
The task of execution, generate task list.
Wait to adjust, it is necessary to which task list is added after the interior task list for needing to be submitted to cluster execution of fixing the date is got
The scheduling for waiting in line cluster in degree queue performs.Wherein, it is currently to wait in line the task of cluster execution to treat scheduling queue
The queue of formation.Task list is added to the scheduling queue for the treatment of of cluster, actually by appointing in task list in step S102
Scheduling queue is treated in business addition, can be specifically that the task of task list is added to the afterbody for treating scheduling queue, by specified day
The task of phase is added after scheduling queue, and system can determine whether to treat the task of scheduled date in scheduling queue, then starts execution and refers to
Fixing the date for task.Scheduling queue is treated for the cluster in step S102, it is thus understood that needs being performed by the cluster for task
The queue of formation, the task in the queue is in state to be scheduled, so can be described as treating scheduling queue, for different clusters
For, treat scheduling queue difference, so cluster treat scheduling queue be interpreted as it is corresponding with some cluster treat scheduling queue,
Task list addition cluster is treated that scheduling queue is actually to add the task of task list to perform the task in S102
Treated corresponding to cluster in scheduling queue.
In general, the task list in the scheduled date is added before treating scheduling queue, treat can all exist extremely in scheduling queue
A few task, the state of these tasks include performing and being not carried out, and state is that execution shows that the task has been filed on to cluster holding
OK, but implementation procedure does not terminate also, and state is to be not carried out showing that the task is not submitted to cluster.It is envisioned that work as task
When list and the situation when scheduling queue generation task duplication, if not changing, the task in task list is added and treated
In scheduling queue, it may appear that the situation that repeating for task is performed, this can undoubtedly waste cluster resource, lose time, and reduce collection
The effective rate of utilization of group's resource.
In order to overcome disadvantages mentioned above, the utilization rate of cluster resource is improved, it is necessary to which the ensureing to repeat of the task is only performed by cluster
Once, before S102, in addition to, task in comparison task list and treat whether the task in scheduling queue has repetition,
In the case of duplicating, deduplication operation is performed so that treat that iterative task is not present in scheduling queue in step S103.
Simultaneously, it is contemplated that the task in scheduling queue for the treatment of has two states：Perform and be not carried out, corresponding, above-mentioned duplicate removal
The implementation of operation has two kinds：
If the first, treating, the iterative task in scheduling queue is not carried out, and deletes the iterative task or task row treated in scheduling queue
Iterative task in table.
Meanwhile the information such as parameter configuration of newly obtaining for task is all execution knot newest, that newly obtaining for task obtains
Fruit is for the implementing result of the task of legacy version, hence it is evident that more for validity and practicality, in the present embodiment, specifies
The task on date is newly obtaining for task, based on discussed above, this reality for the iterative task in scheduling queue is treated
Apply example and preferentially retain iterative task in task list, i.e., before the task in task list is added treats scheduling queue, if appointing
Task in business list and treat that the task in scheduling queue has repetition, it is preferential to delete the iterative task treated in scheduling queue.
If the 2, treating, the iterative task in scheduling queue is not carried out, and deletes the iterative task or task row treated in scheduling queue
Iterative task in table.
When the iterative task in scheduling queue has been submitted to and performed in cluster, the iterative task has occupied collection
Group's resource, so needing to kill the executory iterative task of cluster, reclaims the cluster resource that the iterative task uses.The present embodiment
Cluster resource include but is not limited to internal memory and CPU core number.
After repeating for task is removed, current cluster resource service condition can be updated, appointed it is determined that killing
The cluster resource reclaimed after business, and current total idle cluster resource, are performed to carry out the scheduling of next task.
In addition, when the task in task list is not belonging to iterative task, the task belongs to for iterative task
Newly-increased task, newly-increased task is added the afterbody for treating scheduling queue.
Added by the task in task list after scheduling queue, the step of S103 can be performed, scheduling queue will be treated
Task be submitted to cluster execution.In the present embodiment, the task of scheduled date can preferentially be performed.
Preferential the submitting the scheduled date of the task performs to cluster, in one embodiment, can be specified including randomly selecting
The task on date is submitted to cluster and performed, or the task of scheduled date is submitted into cluster according to some preset rules.
It is clear, however, that these submit the mode of task not to be combined with the actual conditions of current cluster resource, especially working as
When preceding cluster resource is unsatisfactory for the execution condition for the scheduled date task submitted, system may update current cluster always
Resource, when cluster resource meets to be currently needed for the execution condition of the task of execution, cluster just starts execution task.Updating
During cluster resource, the cluster resource of some is in idle state within a certain period of time, is not effectively utilised,
It is unfavorable for the raising of the utilization rate of cluster resource, while also limit the speed of cluster resource processing task.
In order to improve the utilization rate of cluster resource, it is preferable that can be according to the actual use situation of current cluster resource
Current being submitted to cluster execution of the task of selection.Specifically, treating in scheduling queue for task is submitted to perform and can include to cluster
Following two steps：
Step 1: obtaining currently available cluster resource, plan is selected to carry from the task of scheduled date according to preset rules
It is sent to the pending task of cluster execution；
Step 2: if currently available cluster resource meets the execution condition of pending task, pending is submitted
Business to cluster performs, and otherwise, pending task is placed in and treats scheduling queue afterbody.
In above-mentioned step one, obtain currently available cluster resource and actually determine the interior of currently idle
Deposit, the cluster information such as CPU core number, make afterwards, it is necessary to choose the intending being submitted to cluster execution of the task from the task of scheduled date
For pending task, the mode of selection can be chosen according to preset rules, and the preset rules of the present embodiment can basis
Actual conditions are configured, and the preset rules include but is not limited to, and task conduct is randomly selected in the task of scheduled date and is treated
Execution task；Or the selection of pending task is carried out according to the time sequencing of the task of scheduled date, or, according to specifying
The importance of the task on date carries out the selection of pending task, or, according to the scheduled date pre-set task it is excellent
First level carries out the selection of pending task, or, can be from the downward order of the top task of scheduled date task from specified day
The selection of pending task is carried out in the task of phase, top task here can be understood as in the scheduled date for treating scheduling queue
Task in, come the task of foremost.
After it have chosen pending task, judge whether current cluster resource meets the execution bar of pending task
Part, submit pending task to cluster to perform the pending task being satisfied, the pending task not being satisfied is placed in and treated
The afterbody of scheduling queue, when scheduling next time performs.It is understood that when carrying out above-mentioned judgement, participate in what is judged
The execution state of the task of scheduled date is all not carried out.
Wherein, the execution condition of task can be understood as performing the resources such as internal memory and the CPU core number of task needs, will hold
Compared with the resource such as the row condition internal memory i.e. idle with current cluster resource and CPU core number, it is current that the task can be obtained
Whether executable result.The pending task being satisfied is submitted to after cluster performs above-mentioned, can be by the shape of the task
State is from being not carried out being changed into performing, and when selecting pending task next time, the task of the scheduled date in the state of execution is not joined
With selection.
During the execution of the task of scheduled date, the execution state of the task of all scheduled dates can be monitored, etc.
The tasks carrying on date to be specified finishes, and after the tasks carrying of scheduled date, then performs and treats other in scheduling queue
Task.
Method is performed using the task automation of the present embodiment, can be obtained when having substantial amounts of task to need operation
Fetching, which is fixed the date, needs being submitted to cluster execution of the task, and be added into cluster treats that scheduling queue waits scheduling to perform, relatively
In in the prior art, tester submits the mode of task one by one manually, and the present embodiment can realize the automatic of considerable task
Change submission and automated execution, reduce the human cost that test needs, improve testing efficiency.
Further, select the task of scheduled date to be submitted to cluster according to the service condition of cluster resource to perform, can be with
Effectively improve the utilization rate of cluster resource.
Further, the iterative task treated in scheduling queue is removed, cluster resource can be saved first, secondly, can be really
It is newest task to protect the task that cluster performs, it is ensured that the validity of obtained implementing result.
Embodiment two：
Referring to Fig. 3, a kind of task automation method of testing of big data is present embodiments provided, will can be needed the scheduled date
Being performed for task is added to cluster and treats preferentially to perform in scheduling queue, is submitted manually relative to tester in the prior art
For task, the automated testing method of the present embodiment has saved human resources, while also improves the efficiency of test.In this reality
Apply in example, each task has the date that cluster execution is submitted to corresponding to it, can be from substantial amounts of task according to the date
Determine the task of scheduled date.The cluster of the present embodiment can use Spark clusters to realize.
The task automation method of testing of the big data of the present embodiment includes：
S201, configuration scheduled date, obtain the task list for needing to be submitted to cluster execution in the scheduled date.
In step s 201, the scheduled date can be specified by tester, can also be determined by system according to actual conditions,
It can be specific date or certain a period of time.The present embodiment is not limited in this respect.
S202, compare the task in the task list of scheduled date and treat whether the task in scheduling queue has repetition, if
There is task duplication, then into S203, if without repetition, into S205.
S203, when treating that the iterative task in scheduling queue is not submitted to cluster and performs, then delete and treat weight in scheduling queue
Multiple task；Cluster execution is had been filed on to the iterative task in row when waiting to dispatch, kills the executory iterative task of cluster, and return
Accept the cluster resource that row iterative task uses.
In step S203, when when when the iterative task in scheduling queue be not submitted to cluster perform when or delete
Except the iterative task in the task list of scheduled date, it is contemplated that the iterative task of scheduled date is relative to treating scheduling queue
In task for, the information updating such as configuration parameter, perform the result that the iterative task of scheduled date obtains and have more validity, protect
Stay the iterative task in task list more excellent.
S204, renewal cluster resource service condition；Cluster resource service condition includes the utilization feelings of internal memory and CPU core number
Condition.
S205, the task in task list is appended to and treats scheduling queue afterbody, wait scheduling.
In the present embodiment, step S204 and S205 can be exchanged, and the present embodiment is not limited in this respect.
S206, obtain currently available cluster resource.
S207, judge holding for the top task whether currently available cluster resource meets to treat the scheduled date in scheduling queue
Row requires, if not satisfied, into S208, if satisfied, then entering S209.
For the top task in step S207, it is interpreted as in the multiple tasks of scheduled date of scheduling queue are treated, arranges
Task up front.
For step S207, it can also be and pending task is selected from the task of scheduled date according to preset rules, sentence
Whether disconnected cluster resource meets the execution condition of pending task.The explanation of the preset rules of the present embodiment may be referred to embodiment
To the associated description of preset rules in one.
S208, current top task is put into and treats scheduling queue afterbody, using the next task of top task as top
Portion's task, continue above-mentioned deterministic process.
If step S207 is that pending task is selected from the task of scheduled date according to preset rules, judge that cluster provides
Whether source meets the execution condition of pending task, and above-mentioned steps S208 can be placed in pending task to treat scheduling queue tail
Portion, judgement is re-started into S207.
S209, Spark clusters are submitted to perform the top task being satisfied.
S210, the task of acquisition scheduled date execution state, tasks carrying is waited to terminate.
S211, judge whether the task of scheduled date is finished, be, then it is no into S212, then return to S204.
S212, execution treat other tasks in scheduling queue.
, can be in the premise that considerable task needs perform using the task automation method of testing of the big data of the present embodiment
Under, preferential the performing the scheduled date of the task, and executable tasks carrying can be selected according to current cluster resource, be advantageous to carry
The effective rate of utilization of high cluster resource and the efficiency of automatic test, the workload of tester is reduced, save human resources.
Embodiment three：
Referring to Fig. 3, the present embodiment provides a kind of task automation test system of big data, including：
Data management module 31, for configuring execution date for task, obtain to need to be submitted to cluster in the scheduled date and hold
Capable task list；
Task management module 32, for task list addition cluster to be treated into scheduling queue；
Task submits execution module 33, for submitting treating in scheduling queue for task to be performed to cluster.
In the present embodiment, the execution date of each task is equipped with corresponding execution date, and data management module 31 can
According to the scheduled date, to obtain the task list for needing to be submitted to cluster execution in the scheduled date.
Wherein, the scheduled date can freely be set by tester, can also be submitted to cluster execution as needed by system
It can be some specific date that the actual conditions of task set the scheduled date automatically, for example, on October 3rd, 2016 or
For a period of time, such as March 5 is to April 3,8 points to 12 points of March 5 day etc., and the present embodiment is not limited in this respect.
It is envisioned that in the environment of big data, cluster needs the task substantial amounts performed, and single storage is situated between
Matter, which is followed the example of, meets storage demand, so task may be stored in multiple storage mediums, data management module 31, which obtains, specifies
During the task list that the date performs, it may be necessary to which the acquisition scheduled date needs to be submitted to cluster execution from multiple storage mediums
Task, generate task list.
After the interior task list for needing to be submitted to cluster execution of fixing the date is got, task management module 32 needs to appoint
Business list, which adds, treats that the scheduling for waiting in line cluster in scheduling queue performs.Wherein, treat that scheduling queue can be understood as currently existing
Wait in line the queue that the task of cluster execution is formed.Task list is added and treated in scheduling queue by task management module 32, real
It is to add the task in task list to treat scheduling queue on border, can is specifically to add the task of task list to wait to dispatch
The afterbody of queue.
Due to before scheduling queue is treated in the addition of the task list of scheduled date, treating can typically there is at least one in scheduling queue
Individual task, the state of these tasks include performing and being not carried out, and state is that execution shows that the task has been filed on cluster execution, but
Implementation procedure does not terminate also, and state is to be not carried out showing that the task is not submitted to cluster.It is envisioned that when task list and
When the situation of task duplication occurs for scheduling queue, if not changing, task list is added and treated in scheduling queue, repeated
Task can be performed at least twice, this can undoubtedly waste cluster resource, lose time, and reduce effective utilization of cluster resource
Rate.
Further, the task management module 32 of the present embodiment is additionally operable to the task in comparison task list and team to be dispatched
Whether the task in row has repetition, and when task is duplicated in scheduling queue, deduplication operation is performed, so as to task list
Task add after, treat that iterative task is not present in scheduling queue.
Present embodiments provide the mode of two kinds of removal iterative tasks, Yi Zhongshi, the iterative task in scheduling queue is treated
When being not carried out, task management module 32 deletes the iterative task in the iterative task or task list treated in scheduling queue；It is another
Kind is that it is executory that task management module 32 kills cluster when the iterative task in scheduling queue has been filed on to cluster performing
Iterative task, the cluster resource that recovery iterative task uses.Wherein, cluster resource includes but is not limited to internal memory and CPU core number.
The information such as the parameter configuration of newly obtaining for task are all newest, and its implementing result is appointed relative to the repetition of legacy version
For the implementing result of business, hence it is evident that more for validity and practicality, in the present embodiment, the task of scheduled date is relative to treating
It is newly obtaining for task for iterative task in scheduling queue.Based on discussed above, add by the task in task list
Enter before treating scheduling queue, if the task in task list and treating that the task in scheduling queue has repetition, it is preferred that task management mould
Block 32 deletes the iterative task treated in scheduling queue.
In addition, when the task in task list is not belonging to iterative task, the task belongs to for iterative task
Newly-increased task, task management module 32 can add newly-increased task the afterbody for treating scheduling queue.
Task management module 32 in the present embodiment adds task list after scheduling queue, and task submits execution module
33 can start to submit the task of scheduled date.
In view of in the task of the scheduled date of execution, it is understood that there may be cluster resource is unsatisfactory for the execution condition of task
Situation, now, system may update current cluster resource always, after waiting cluster resource to meet the execution condition of the task
The task is performed again, and thus, a part of cluster resource can leave unused within a certain period of time, reduce the effective rate of utilization of cluster resource.
Based on discussed above, it is preferred that the task of the present embodiment submits execution module 33 to be used to obtain currently available cluster resource,
Select to intend being submitted to the pending task that cluster performs from the task of scheduled date according to preset rules；It is if currently available
Cluster resource meets the execution condition of pending task, then submits pending task to cluster to perform, otherwise, by pending task
It is placed in and treats scheduling queue afterbody.
Wherein, it is actually to determine the current free time that task, which submits execution module 33 to obtain currently available cluster resource,
Internal memory, the cluster information such as CPU core number.Afterwards, task submits execution module 33 to need to choose plan from the task of scheduled date
Being submitted to cluster execution of the task as pending task, the mode of selection can be chosen according to preset rules, this reality
Applying the preset rules of example can be configured according to actual conditions, and the preset rules include but is not limited to, in appointing for scheduled date
Task is randomly selected in business as pending task；Or carry out pending according to the time sequencing of the task of scheduled date
The selection of business, or, the selection of pending task is carried out according to the importance of the task of scheduled date, or, according to setting in advance
The priority of the task for the scheduled date put carries out the selection of pending task, or, can be from the top of scheduled date task
The downward order of task carries out the selection of pending task from the task of scheduled date, and top task here can be understood as
In the task of scheduled date of scheduling queue is treated, the task of foremost is come.
After it have chosen pending task, task submits execution module 33 to may determine that whether current cluster resource is full
The execution condition of the pending task of foot, submit pending task to cluster to perform the pending task being satisfied, will not be expired
The pending task of foot is placed in the afterbody for treating scheduling queue, when scheduling next time performs.It is understood that carry out above-mentioned
Judgement when, the execution state of task for the scheduled date for participating in judging all is not carried out.
In order to determine the Task Progress of scheduled date, the system of the present embodiment can also include monitoring module 34, for
During the execution of the task of scheduled date, the execution state of the task of all scheduled dates, and appointing in the scheduled date are monitored
After business is finished, notice task submits execution module 33 to submit other tasks to the cluster treated in scheduling queue to perform.
Using the task automation test system of the big data of the present embodiment, can be obtained using data management module specified
Date needs to be submitted to the task list of cluster execution, adds task to be tested manually relative to tester in the prior art
Mode, the present embodiment has the advantages of automation submission task, and task management module and task submit execution module to coordinate can
To realize that considerable task is needed on the premise of performing, the purpose of preferential the performing the scheduled date of the task, while can be according to cluster
The tasks carrying for the scheduled date that the service condition selection of resource can perform, is effectively improved utilization rate and the survey of cluster resource
Try efficiency.
Obviously, those skilled in the art should be understood that each module of the invention described above or each step can be with general
Computing device realizes that they can be concentrated on single computing device, or be distributed in what multiple computing devices were formed
On network, alternatively, they can be realized with the program code that computing device can perform, it is thus possible to be stored in
Performed in storage medium (ROM/RAM, magnetic disc, CD) by computing device, and in some cases, can be with different from this
The order at place performs shown or described step, either they are fabricated to respectively each integrated circuit modules or by it
In multiple modules or step be fabricated to single integrated circuit module to realize.So the present invention be not restricted to it is any specific
Hardware and software combine.
Above content is to combine specific embodiment further description made for the present invention, it is impossible to assert this hair
Bright specific implementation is confined to these explanations.For general technical staff of the technical field of the invention, do not taking off
On the premise of from present inventive concept, some simple deduction or replace can also be made, should all be considered as belonging to the protection of the present invention
Scope.
Claims (10)
1. a kind of task automation method of testing of big data, including：
Execution date is configured for task, obtains the task list for needing to be submitted to cluster execution in the scheduled date；
The task list addition cluster is treated into scheduling queue；
Treat that the task in scheduling queue performs to the cluster described in submission.
2. the task automation method of testing of big data as claimed in claim 1, it is characterised in that wait to adjust described in the submission
Task in degree queue performs to the cluster to be included：
Currently available cluster resource is obtained, selects from the task of the scheduled date plan to be submitted to according to preset rules described
The pending task that cluster performs；
If the currently available cluster resource meets the execution condition of the pending task, the pending task is submitted
Performed to the cluster, otherwise, the pending task is placed in and described treats scheduling queue afterbody.
3. the task automation method of testing of big data as claimed in claim 1 or 2, it is characterised in that arranged in the task
Table add the cluster treat scheduling queue before, in addition to：
Compare whether the task in the task list and treating in scheduling queue for the task have repetition, and wait to dispatch described
When task is duplicated in queue, deduplication operation is performed.
4. the task automation method of testing of big data as claimed in claim 3, it is characterised in that the deduplication operation bag
Include：
If the iterative task treated in scheduling queue is not carried out, the iterative task in scheduling queue or described are treated described in deletion
The iterative task being engaged in list.
5. the task automation method of testing of big data as claimed in claim 3, it is characterised in that the deduplication operation bag
Include：
If the iterative task treated in scheduling queue has been filed on to the cluster performing, it is executory described to kill the cluster
Iterative task, reclaim the cluster resource that the iterative task uses.
A kind of 6. task automation test system of big data, it is characterised in that including：
Data management module, for configuring execution date for task, obtain needs to be submitted to appointing for cluster execution in the scheduled date
Business list；
Task management module, for the task list addition cluster to be treated into scheduling queue；
Task submits execution module, described treats that the task in scheduling queue performs to the cluster for submitting.
7. the task automation test system of big data as claimed in claim 6, it is characterised in that the task, which is submitted, to be performed
Module, for taking currently available cluster resource, plan is selected to be submitted to from the task of the scheduled date according to preset rules
The pending task that the cluster performs；If the currently available cluster resource meets the execution bar of the pending task
Part, submit the pending task to the cluster to perform, otherwise, by the pending task be placed in described in treat scheduling queue tail
Portion.
8. the task automation test system of big data as claimed in claims 6 or 7, it is characterised in that the task management
Module is additionally operable to, the task list add the cluster treat scheduling queue before, the task in the task list
Whether there is repetition with treating in scheduling queue for the task, and described when task is duplicated in scheduling queue, execution is gone
Operate again.
9. the task automation test system of big data as claimed in claim 8, it is characterised in that the task management module
For：
If the iterative task treated in scheduling queue is not carried out, the iterative task in scheduling queue or described are treated described in deletion
The iterative task being engaged in list.
10. the task automation test system of big data as claimed in claim 8, it is characterised in that the task management mould
Block is used for：
If the iterative task treated in scheduling queue has been filed on to the cluster performing, it is executory described to kill the cluster
Iterative task, reclaim the cluster resource that the iterative task uses.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610404001.1A CN107480041A (en) | 2016-06-07 | 2016-06-07 | The task automation method of testing and system of a kind of big data |
| PCT/CN2016/103969 WO2017211042A1 (en) | 2016-06-07 | 2016-10-31 | Task automation testing method and system for big data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610404001.1A CN107480041A (en) | 2016-06-07 | 2016-06-07 | The task automation method of testing and system of a kind of big data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107480041A true CN107480041A (en) | 2017-12-15 |
Family
ID=60577629
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610404001.1A Pending CN107480041A (en) | 2016-06-07 | 2016-06-07 | The task automation method of testing and system of a kind of big data |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN107480041A (en) |
| WO (1) | WO2017211042A1 (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110134520A (en) * | 2019-05-27 | 2019-08-16 | 眸芯科技（上海）有限公司 | The application method and system of integrated circuit scarce resource based on queuing |
| CN110851363A (en) * | 2019-11-12 | 2020-02-28 | 广东电网有限责任公司 | Cloud testing system and method |
| WO2020186781A1 (en) * | 2019-03-18 | 2020-09-24 | 平安普惠企业管理有限公司 | Test code handover control method and apparatus, electronic device, and computer non-volatile readable storage medium |
| CN112817808A (en) * | 2019-11-18 | 2021-05-18 | 赵伟 | Computer cluster maintenance task management method and system |
| CN113190335A (en) * | 2021-05-07 | 2021-07-30 | 安徽南瑞中天电力电子有限公司 | Multi-task scheduling and collecting method of power collecting terminal and power collecting system |
| CN113535560A (en) * | 2021-07-14 | 2021-10-22 | 杭州网易云音乐科技有限公司 | Test execution method and device, storage medium and computing equipment |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111291893B (en) * | 2020-01-22 | 2023-06-02 | 合肥本源量子计算科技有限责任公司 | Scheduling method, scheduling system, storage medium and electronic device |
| CN111416839B (en) * | 2020-02-26 | 2022-09-23 | 平安科技（深圳）有限公司 | Cluster environment timing task processing method, system, device and storage medium |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9021486B2 (en) * | 2010-08-19 | 2015-04-28 | International Business Machines Corporation | Selective constant complexity dismissal in task scheduling |
| CN103870348A (en) * | 2012-12-14 | 2014-06-18 | 中国电信股份有限公司 | Test method and system for concurrent user access |
| CN103678133A (en) * | 2013-12-18 | 2014-03-26 | 中国科学院深圳先进技术研究院 | Task scheduling system for application software cloud testing |
| CN103678132A (en) * | 2013-12-18 | 2014-03-26 | 中国科学院深圳先进技术研究院 | Application software cloud testing system |
| CN103645909A (en) * | 2013-12-30 | 2014-03-19 | 中国烟草总公司湖南省公司 | Handling method and device for timed task |
| CN105045710B (en) * | 2015-06-30 | 2017-11-10 | 吉林大学 | A kind of automatic test data creation method under cloud computing environment |
-
2016
- 2016-06-07 CN CN201610404001.1A patent/CN107480041A/en active Pending
- 2016-10-31 WO PCT/CN2016/103969 patent/WO2017211042A1/en active Application Filing
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020186781A1 (en) * | 2019-03-18 | 2020-09-24 | 平安普惠企业管理有限公司 | Test code handover control method and apparatus, electronic device, and computer non-volatile readable storage medium |
| CN110134520A (en) * | 2019-05-27 | 2019-08-16 | 眸芯科技（上海）有限公司 | The application method and system of integrated circuit scarce resource based on queuing |
| CN110851363A (en) * | 2019-11-12 | 2020-02-28 | 广东电网有限责任公司 | Cloud testing system and method |
| CN112817808A (en) * | 2019-11-18 | 2021-05-18 | 赵伟 | Computer cluster maintenance task management method and system |
| CN112817808B (en) * | 2019-11-18 | 2024-04-19 | 赵伟 | Computer cluster maintenance task management method and system |
| CN113190335A (en) * | 2021-05-07 | 2021-07-30 | 安徽南瑞中天电力电子有限公司 | Multi-task scheduling and collecting method of power collecting terminal and power collecting system |
| CN113535560A (en) * | 2021-07-14 | 2021-10-22 | 杭州网易云音乐科技有限公司 | Test execution method and device, storage medium and computing equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2017211042A1 (en) | 2017-12-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107480041A (en) | The task automation method of testing and system of a kind of big data | |
| CN105049268B (en) | Distributed computing resource distribution system and task processing method | |
| Zhang et al. | Energy-aware virtual machine allocation for cloud with resource reservation | |
| Han et al. | Benchmarking big data systems: A review | |
| Fernández-Cerero et al. | SCORE: Simulator for cloud optimization of resources and energy consumption | |
| CN109582433A (en) | A kind of resource regulating method, device, cloud computing system and storage medium | |
| CN103593323A (en) | Machine learning method for Map Reduce task resource allocation parameters | |
| Turk et al. | Temporal workload-aware replicated partitioning for social networks | |
| WO2015021931A1 (en) | Task-based modeling for parallel data integration | |
| Hu et al. | QoS promotion in energy-efficient datacenters through peak load scheduling | |
| CN107370796A (en) | A kind of intelligent learning system based on Hyper TF | |
| CN106101213A (en) | Information-distribution type storage method | |
| CN113886034A (en) | Task scheduling method, system, electronic device and storage medium | |
| CN106407395A (en) | A processing method and device for data query | |
| CN105159750A (en) | Virtual machine creation method and apparatus | |
| Machovec et al. | Utility-based resource management in an oversubscribed energy-constrained heterogeneous environment executing parallel applications | |
| Ali et al. | Petri Net based modeling and analysis for improved resource utilization in cloud computing | |
| CN102201922A (en) | Data charging method and relevant apparatus | |
| CN106254452A (en) | The big data access method of medical treatment under cloud platform | |
| Gabaldon et al. | Multi-criteria genetic algorithm applied to scheduling in multi-cluster environments | |
| de Oliveira et al. | Acosched: A scheduling algorithm in a federated cloud infrastructure for bioinformatics applications | |
| Posey et al. | Addressing the challenges of executing a massive computational cluster in the cloud | |
| CN107197013A (en) | One kind enhancing cloud computing environment energy conserving system | |
| Mansouri | An effective weighted data replication strategy for data grid | |
| CN105930202A (en) | Migration policy for virtual machine with three thresholds |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20171215 |