WO2017211042A1 - Task automation testing method and system for big data - Google Patents

Task automation testing method and system for big data Download PDF

Info

Publication number
WO2017211042A1
WO2017211042A1 PCT/CN2016/103969 CN2016103969W WO2017211042A1 WO 2017211042 A1 WO2017211042 A1 WO 2017211042A1 CN 2016103969 W CN2016103969 W CN 2016103969W WO 2017211042 A1 WO2017211042 A1 WO 2017211042A1
Authority
WO
WIPO (PCT)
Prior art keywords
task
cluster
execution
scheduled
queue
Prior art date
Application number
PCT/CN2016/103969
Other languages
French (fr)
Chinese (zh)
Inventor
赵士文
陈艳山
李勇
Original Assignee
中兴通讯股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中兴通讯股份有限公司 filed Critical 中兴通讯股份有限公司
Publication of WO2017211042A1 publication Critical patent/WO2017211042A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/36—Preventing errors by testing or debugging software
    • G06F11/3668—Software testing
    • G06F11/3672—Test management
    • G06F11/3688—Test management for test execution, e.g. scheduling of test suites
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/36—Preventing errors by testing or debugging software

Definitions

  • the present disclosure relates to the field of big data, such as a task automation test method and system involving a big data.
  • a cluster is a group of independent computers interconnected by a high-speed network that form a group and are managed in a single system. When a customer interacts with the cluster, the cluster appears as a separate server. Clusters are known for their high-speed data mining and analysis capabilities, and can process large amounts of data.
  • the processing of massive data by clusters means the processing and submission of a large number of tasks.
  • the task to be processed is generally submitted by the tester to the cluster system. After that, the cluster system submits the task to the cluster according to the submission time of the task.
  • the workload is heavy. Therefore, the manual submission of tasks in related technologies may have problems such as cumbersome tasks, large workload, and low test efficiency.
  • the main technical problem to be solved by the present disclosure is to provide a task automatic test method and system for big data, which solves the problems of cumbersome test tasks, large workload, and low test efficiency caused by manually submitting a large number of tasks in the related art.
  • Embodiments of the present disclosure provide a task automation test method for big data, including:
  • the embodiment of the present disclosure further provides a task automation test system for big data, including:
  • the data management module is configured to configure an execution date for the task, and obtain a task list that needs to be submitted to the cluster for execution on the specified date;
  • a task management module configured to add a task list to a queue to be scheduled of the cluster
  • the task submission execution module is set to submit the tasks in the queue to be scheduled to the cluster execution.
  • a non-transitory storage medium storing computer-executable instructions configured to perform the task automation test method of the above-described big data is provided.
  • a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions, when the program instructions When executed by a computer, the computer is caused to perform the above-described task automation test method for big data.
  • an electronic device comprising at least one processor and a memory communicatively coupled to the at least one processor, the memory for storing instructions executable by the at least one processor And causing the at least one when the instruction is executed by the at least one processor
  • the processor executes the above-described task automation test method for big data.
  • the task automation test method and system of the big data of the present disclosure can acquire a task that needs to be submitted to the cluster for execution on a specified date when a large number of tasks need to be run, and join the cluster to be scheduled to be scheduled for execution, as opposed to
  • the tester manually submits a task by itself, and the automatic submission and automatic execution of the task of the embodiment can effectively reduce the test task amount and task cumbersomeness, simplify the manual operation, and save the test requirement. Labor costs increase test efficiency.
  • FIG. 1 is a flowchart of a method for automatically testing a task of big data according to Embodiment 1 of the present disclosure
  • FIG. 2 is a flowchart of a task automatic test method for big data according to Embodiment 2 of the present disclosure
  • FIG. 3 is a schematic block diagram of a task automatic test system for big data according to Embodiment 3 of the present disclosure
  • FIG. 4 is a structural block diagram of an electronic device according to an embodiment of the present disclosure.
  • Embodiment 1 is a diagrammatic representation of Embodiment 1:
  • this embodiment proposes a task automatic test method for big data, including:
  • the execution date is configured for the task to determine the specific execution date of each task, and is used as the basis for the subsequent acquisition of the specified date.
  • the task list can be obtained according to the specified date.
  • the specified date here can be freely set by the tester, or can be automatically set by the system according to the actual situation of the task submitted to the cluster according to the need to specify the date can be a specific date, such as October 3, 2016, or a period of time, For example, March 5 to April 3, March 5, 8:00 to 12:00, etc., this embodiment is not limited thereto.
  • the task list includes at least one task that needs to be submitted to the specified date of cluster execution.
  • the task of this embodiment may be stored in multiple storage media, and the task list executed on the specified date is acquired in step S110.
  • you need to obtain the tasks that need to be submitted to the cluster for the specified date from multiple storage media you can generate a task list.
  • the task list can be added to the queue to be scheduled to wait for the scheduling execution of the cluster.
  • the queue to be scheduled is a queue formed by tasks currently waiting in the queue for execution.
  • the task list is added to the to-be-scheduled queue of the cluster, and the task in the task list is added to the queue to be scheduled.
  • the task may be added to the tail of the queue to be scheduled, and the task of the specified date is added to the task.
  • the system determines the task for the specified date in the queue to be scheduled, and then starts the task for the specified date.
  • the queue to be scheduled in the cluster in step S120 it can be understood as a queue formed by tasks executed by the cluster, and the tasks in the queue are in a state to be scheduled, so it can be called a queue to be scheduled, for different clusters.
  • the queue to be scheduled is different, so the queue to be scheduled of the cluster can be understood as a queue to be scheduled corresponding to a cluster.
  • the queue to be scheduled to be added to the cluster in S120 can be the task of adding the task list to the cluster performing the task. In the queue to be scheduled.
  • the task list of the specified date is added to the queue to be scheduled.
  • the status of these tasks includes execution and non-execution.
  • the status is execution indicating that the task has been submitted to the cluster for execution, but the execution is performed.
  • the process has not ended and the status is not executed indicating that the task was not submitted to the cluster. If the tasks in the task list and the queue to be scheduled are duplicated, if no tasks are modified, the tasks in the task list are added to the queue to be scheduled. Repeated tasks are executed, which wastes cluster resources, wastes time, and reduces Effective utilization of cluster resources.
  • the method may further include: comparing whether the tasks in the task list and the tasks in the queue to be scheduled have duplicates, In the case of repetition, the deduplication operation is performed, so that there is no duplicate task in the queue to be scheduled in step S130.
  • the above-mentioned deduplication operation can be implemented in two ways:
  • the task of the specified date is a newly acquired task with respect to the repetitive task in the queue to be scheduled.
  • the embodiment may preferentially retain the repetitive task in the task list, that is, the task is If a task in the list is added to the queue to be scheduled, if the task in the task list and the task in the queue to be scheduled are duplicated, the duplicate task in the queue to be scheduled is deleted first.
  • the cluster resources of this embodiment include, but are not limited to, memory and CPU cores.
  • the current cluster resource usage can be updated, and the cluster resources recovered after killing the task and the current total idle cluster resources are determined for scheduling execution of the next task.
  • the task in the task list does not belong to the repetitive task
  • the task belongs to the new task relative to the repetitive task, and the newly added task is added to the tail of the queue to be scheduled.
  • the step of S103 may be performed, and the task of the queue to be scheduled is submitted to the cluster for execution.
  • the task of the specified date can be preferentially executed.
  • the task of the specified date is preferentially submitted to the cluster for execution.
  • the task of randomly selecting the specified date may be submitted to the cluster for execution, or the task of the specified date may be submitted to the cluster according to a preset rule.
  • the system may always update the current cluster resources.
  • the cluster does not begin execution until the cluster resource meets the execution conditions of the task that needs to be executed.
  • some of the cluster resources are idle for a certain period of time, and are not effectively utilized, which is not conducive to the utilization of cluster resources, and also limits the rate of cluster resource processing tasks.
  • the tasks currently submitted to the cluster can be selected according to the actual usage of the current cluster resources. Specifically, submitting the task in the queue to be scheduled to the cluster execution may include the following two steps:
  • Step 1 Obtain the currently available cluster resources, and select the tasks to be executed to be submitted to the cluster execution from the tasks of the specified date according to the preset rules.
  • Step 2 If the currently available cluster resource meets the execution condition of the task to be executed, the task to be executed is submitted to the cluster for execution. If the currently available cluster resource does not meet the execution condition of the task to be executed, the task to be executed is placed in the queue to be scheduled. Tail.
  • obtaining the currently available cluster resources is actually determining the cluster information such as the current free memory and the number of CPU cores.
  • the tasks to be submitted to the cluster are selected from the tasks of the specified date as pending execution.
  • the task may be selected according to a preset rule.
  • the preset rule in this embodiment may be set according to an actual situation.
  • the preset rule includes, but is not limited to, randomly selecting a task as a task to be executed in a task of a specified date. Or, the selection of the task to be executed is performed in the chronological order of the tasks of the specified date, or the selection of the task to be executed is performed according to the importance of the task of the specified date, or the priority of the task of the specified date is set in advance. Performing the selection of the task, or selecting the task to be executed from the task of the specified date in descending order from the top task of the specified date task, where the top task can be understood as the task on the specified date of the queue to be scheduled, The top task.
  • the task to be executed After the task to be executed is selected, it is determined whether the current cluster resource satisfies the execution condition of the task to be executed, and the to-be-executed task to be executed is submitted to the cluster for execution, and the unexecuted task to be executed is placed in the queue to be scheduled. The tail is waiting for the next scheduled execution.
  • the execution state of the task of the specified date participating in the judgment is not executed.
  • the execution condition of the task can be understood as the resources such as the memory and the number of CPU cores required to execute the task, and the execution condition is compared with the current cluster resources, that is, the free memory and the number of CPU cores, and the task can be obtained currently. The result of the execution. After the above-mentioned task to be executed is submitted to the cluster for execution, the status of the task can be changed from execution to execution, and the next time the candidate is selected to be executed. The task of the specified date in the execution state does not participate in the selection.
  • the tasks that need to be submitted to the cluster for the specified date are acquired, and are added to the cluster to be scheduled to be scheduled for execution, compared with related technologies.
  • the manner in which the tester manually submits one task is the automatic submission and automatic execution of a large number of tasks in the embodiment, which reduces the labor cost required for the test and improves the test efficiency.
  • the task of selecting the specified date is submitted to the cluster according to the usage of the cluster resource, which can effectively improve the utilization rate of the cluster resource.
  • the repetitive tasks in the queue to be scheduled are removed, firstly, the cluster resources can be saved, and secondly, the tasks performed by the cluster are ensured to be the latest tasks, and the effectiveness of the obtained execution results is ensured.
  • Embodiment 2 is a diagrammatic representation of Embodiment 1:
  • this embodiment provides a task automatic test method for big data, which can be performed by adding a task to be executed on a specified date to a queue to be scheduled, which is compared with the manual submission of a task by a tester in the related art.
  • the automated test method of the embodiment saves human resources and also improves the efficiency of the test.
  • each task has its corresponding date submitted to the cluster for execution, and according to the date, the task of the specified date can be determined from a large number of tasks.
  • the cluster in this embodiment can be implemented by using a Spark cluster.
  • S201 Configure a specified date, and obtain a task list that needs to be submitted to the cluster for execution on the specified date.
  • the specified date may be specified by the tester, or may be determined by the system according to the actual situation.
  • the situation is determined, it can be a specific date, or it can be a certain period of time. This embodiment is not limited thereto.
  • step S203 when the repetitive task in the queue to be scheduled is not submitted to the cluster for execution, it may also be to delete the repetitive task in the task list of the specified date, but considering the repetitive task of the specified date relative to the task in the queue to be scheduled. In other words, information such as configuration parameters is updated, and the results of performing repeated tasks on a specified date are more effective, and the repetitive tasks in the task list are retained.
  • S204 Update the usage of the cluster resource; the usage of the cluster resource includes the utilization of the memory and the number of CPU cores.
  • the steps S204 and S205 can be exchanged, and the embodiment does not limit this.
  • S207 Determine whether the currently available cluster resource meets the execution requirement of the top task on the specified date in the queue to be scheduled. If the currently available cluster resource meets the execution requirement of the top task on the specified date in the queue to be scheduled, the process proceeds to S208, if currently available. If the cluster resource does not meet the execution requirement of the top task on the specified date in the queue to be scheduled, the process proceeds to S209.
  • the top task in step S207 it is understood that among the plurality of tasks on the specified date of the queue to be scheduled, the top task is ranked.
  • the task to be executed is selected from the tasks of the specified date according to the preset rule, and whether the cluster resource meets the execution condition of the task to be executed is determined.
  • Solution of the preset rule of this embodiment For a reference, refer to the related description of the preset rule in the first embodiment.
  • step S207 is to select a task to be executed from the task of the specified date according to the preset rule, and determine whether the cluster resource meets the execution condition of the task to be executed
  • the step S208 may be that the task to be executed is placed at the end of the queue to be scheduled, and the process proceeds to S207. Make a judgment.
  • the task automation test method of the big data of the embodiment can preferentially execute the task of the specified date under the premise that a large number of tasks need to be executed, and can select an executable task execution according to the current cluster resource, which is beneficial to improving the cluster resource. Effective utilization and efficiency of automated testing, reducing the workload of testers and saving human resources.
  • Embodiment 3 is a diagrammatic representation of Embodiment 3
  • the embodiment provides a task automatic test system for big data, including:
  • the data management module 31 is configured to configure an execution date for the task, and obtain a task list that needs to be submitted to the cluster for execution on the specified date;
  • the task management module 32 is configured to add the task list to the queue to be scheduled of the cluster;
  • the task submission execution module 33 is configured to submit the tasks in the queue to be scheduled to the cluster execution.
  • the execution date of each task is configured with a corresponding execution date, and the data management module 31 can obtain a task list that needs to be submitted to the cluster for execution on the specified date according to the specified date.
  • the specified date can be set freely by the tester, or can be automatically set by the system according to the actual situation of the task submitted to the cluster according to need.
  • the specified date can be a specific date, such as October 3, 2016, or a period of time. For example, March 5 to April 3, March 5, 8:00 to 12:00, etc., this embodiment is not limited thereto.
  • a task list is generated by obtaining tasks from a plurality of storage media that need to be submitted to the cluster for the specified date.
  • the task management module 32 may add the task list to the scheduled queue to wait for the scheduling execution of the cluster.
  • the queue to be scheduled may be a queue formed by tasks currently waiting to be executed by the cluster.
  • the task management module 32 adds the task list to the queue to be scheduled, and may add the task in the task list to the queue to be scheduled. Specifically, the task of the task list may be added to the tail of the queue to be scheduled.
  • the task list on the specified date is added to the queue to be scheduled, there are generally at least one task in the queue to be scheduled.
  • the status of these tasks includes execution and non-execution. The status indicates that the task has been submitted to the cluster for execution, but the execution process is still performed. Not completed, the status is not executed indicating that the task was not submitted to the cluster.
  • the task list and the queue to be scheduled are duplicated, if no modification is made, the task list is added to the queue to be scheduled, and the repeated tasks are executed at least twice, which will waste the cluster resources, waste time, and reduce the cluster. Effective utilization of resources.
  • the task management module 32 of the embodiment is further configured to compare whether the tasks in the task list and the tasks in the queue to be scheduled have duplicates, and perform a deduplication operation when the repeated tasks occur in the queue to be scheduled, so that the tasks are performed. After the task of the list is added, there are no duplicate tasks in the queue to be scheduled.
  • This embodiment provides two ways to remove duplicate tasks, one is the weight in the queue to be scheduled.
  • the task management module 32 deletes the repeated task in the queue to be scheduled or the repeated task in the task list; the other is that when the repeated task in the queue to be scheduled has been submitted to the cluster for execution, the task management module 32 kills The repetitive tasks in the cluster execution are removed, and the cluster resources used by the repetitive tasks are collected.
  • cluster resources include but are not limited to memory and CPU cores.
  • the information such as the parameter configuration of the newly acquired task is up-to-date, and the execution result is obviously more effective and practical than the execution result of the repetitive task of the old version.
  • the task of the specified date is relatively In the case of a repetitive task in the queue to be scheduled, it is a newly acquired task. Based on the above considerations, before the tasks in the task list are added to the queue to be scheduled, if the tasks in the task list and the tasks in the queue to be scheduled are duplicated, the task management module 32 deletes the repeated tasks in the queue to be scheduled. .
  • the task management module 32 can add the newly added task to the tail of the queue to be scheduled.
  • the task submission execution module 33 can start submitting the task of the specified date.
  • the task submission execution module 33 of the present embodiment is configured to acquire the currently available cluster resources, and select a task to be executed to be submitted to the cluster execution from the tasks of the specified date according to a preset rule; If the available cluster resources meet the execution conditions of the task to be executed, the task to be executed is submitted to the cluster for execution. If the currently available cluster resources do not meet the execution conditions of the task to be executed, the task to be executed is placed at the end of the queue to be scheduled.
  • the task submission execution module 33 obtains the currently available cluster resources, which is actually determining the current Cluster information such as free memory and CPU core count. Then, the task submission execution module 33 needs to select a task to be submitted to the cluster to be executed as a task to be executed from the task of the specified date, and the selected method may be selected according to a preset rule.
  • the preset rule in this embodiment may be based on actual conditions.
  • the selection of the task to be executed, the top task can be understood as the top task in the task of the specified date of the queue to be scheduled.
  • the task submission execution module 33 may determine whether the current cluster resource meets the execution condition of the task to be executed, and submit the to-be-executed task to the cluster execution, and the unsatisfied to be executed. The task is placed at the end of the queue to be scheduled and waits until the next scheduled execution. It can be understood that, when the above determination is made, the execution state of the task of the specified date participating in the judgment is not executed.
  • the system of the present embodiment may further include a listening module 34 configured to listen to the execution status of all tasks of the specified date during execution of the task of the specified date, and execute the task on the specified date. Afterwards, the notification task submission execution module 33 submits other tasks in the queue to be scheduled to the cluster execution.
  • the data management module can obtain the task list that needs to be submitted to the cluster for execution on the specified date, and the method has the method of manually adding the task to be tested by the tester in the related art.
  • the advantages of automating the submission of tasks, the task management module and the task submission execution module can achieve the purpose of performing tasks with a specified date on the premise that a large number of tasks need to be executed, and can select an executable designated date according to the usage of the cluster resources.
  • the task execution of the period effectively improves the utilization of cluster resources and the efficiency of testing.
  • the present disclosure also provides a non-transitory storage medium storing computer executable instructions arranged to perform a task automation test method for big data of the above-described embodiments.
  • the present disclosure also provides a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions, when the program instructions are executed by a computer, The computer is caused to execute the task automation test method of the big data of the above embodiment.
  • FIG. 4 is a structural block diagram of an electronic device according to an embodiment of the present disclosure.
  • the electronic device may include a processor 51 and a memory 53, and may further include a communication interface 52 and a bus 54.
  • the processor 51, the communication interface 52, and the memory 53 can complete communication with each other through the bus 54.
  • Communication interface 52 can be used for information transmission.
  • the processor 51 can call the logic instructions in the memory 53 to perform the task automation test method of the big data of the above embodiment.
  • the logic instructions in the memory 53 described above may be implemented in the form of a software functional unit and sold or used as a stand-alone product, and may be stored in a computer readable storage medium.
  • the technical solution of the present disclosure may be embodied in the form of a software product stored in a storage medium, including a plurality of instructions for causing a computer device (which may be a personal computer, a server, or a network) The device or the like) performs all or part of the steps of the method described in various embodiments of the present disclosure.
  • the foregoing storage medium may be a non-transitory storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
  • a medium that can store program code, or a transitory storage medium including: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
  • the task automatic test system of the big data in this embodiment can use the data management module to obtain a task list that needs to be submitted to the cluster for execution on a specified date, and the embodiment is automated in comparison with the manner in which the tester manually adds the task to be tested in the related art.
  • the advantages of submitting the task, the task management module and the task submission execution module can realize the purpose of executing the tasks of the specified date on the premise that a large number of tasks need to be executed, and at the same time, can select the executable task of the specified date according to the use of the cluster resources. , effectively improve the utilization of cluster resources and test efficiency.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • Quality & Reliability (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Debugging And Monitoring (AREA)

Abstract

Disclosed are a task automation testing method and system for big data. The method comprises: configuring an execution date for a task, and acquiring a list of tasks needing to be submitted to a cluster for execution within a specified date (S110); adding the list of tasks to a queue to be scheduled of the cluster (S120); and submitting the tasks in the queue to be scheduled to the cluster for execution (S130).

Description

大数据的任务自动化测试方法和***Big data task automation test method and system 技术领域Technical field
本公开涉及大数据领域，例如涉及一种大数据的任务自动化测试方法和***。The present disclosure relates to the field of big data, such as a task automation test method and system involving a big data.
背景技术Background technique
随着智能手机，平板电脑，还有笔记本电脑的普及，越来越多的用户接入网络，越来越多的数据产生。在互联网行业，“大数据”是这样一种现象：互联网公司在日常运营中生成、累积的用户网络行为数据，这些数据的量十分庞大，以至需要用P(100个T)、E(100万个T)、或Z(10亿个T)来衡量，以目前的技术水平，很少有单个计算机可以处理这么大的数据量，也很少有单个储存设备可以有这么大容量。这时候，就需要用到集群来对大数据进行处理。集群是一组相互独立的、通过高速网络互联的计算机，它们构成了一个组，并以单一***的模式加以管理。一个客户与集群相互作用时，集群表现为一个独立的服务器。集群以高速的数据挖掘和分析能力著称，可以对海量数据进行处理。With the popularity of smartphones, tablets, and laptops, more and more users are accessing the network, and more and more data is being generated. In the Internet industry, “Big Data” is a phenomenon in which Internet companies generate and accumulate user network behavior data in daily operations. The amount of such data is so large that P (100 T) and E (1 million) are needed. T), or Z (1 billion T), at the current state of the art, few single computers can handle such a large amount of data, and few single storage devices can have such a large capacity. At this time, you need to use clusters to process big data. A cluster is a group of independent computers interconnected by a high-speed network that form a group and are managed in a single system. When a customer interacts with the cluster, the cluster appears as a separate server. Clusters are known for their high-speed data mining and analysis capabilities, and can process large amounts of data.
集群对海量数据的处理意味着大量任务的处理和提交。相关技术中，一般由测试人员提交待处理的任务到集群***中，之后，集群***会根据任务的提交时间的顺序依次将任务提交到集群中执行。对测试人员而言，需要管理和提交大量的任务，工作量繁重，所以相关技术中人工提交任务的方式会存在任务繁琐、工作量大、测试效率低下的问题。 The processing of massive data by clusters means the processing and submission of a large number of tasks. In the related art, the task to be processed is generally submitted by the tester to the cluster system. After that, the cluster system submits the task to the cluster according to the submission time of the task. For testers, there is a need to manage and submit a large number of tasks, and the workload is heavy. Therefore, the manual submission of tasks in related technologies may have problems such as cumbersome tasks, large workload, and low test efficiency.
发明内容Summary of the invention
本公开要解决的主要技术问题是，提供一种大数据的任务自动化测试方法和***，解决相关技术中由人工提交大量任务引起的测试任务繁琐、工作量大、测试效率低的问题。The main technical problem to be solved by the present disclosure is to provide a task automatic test method and system for big data, which solves the problems of cumbersome test tasks, large workload, and low test efficiency caused by manually submitting a large number of tasks in the related art.
本公开实施例提供一种大数据的任务自动化测试方法，包括：Embodiments of the present disclosure provide a task automation test method for big data, including:
为任务配置执行日期，获取指定日期内需要提交到集群执行的任务列表；Configure the execution date for the task to obtain a list of tasks that need to be submitted to the cluster for execution on the specified date;
将任务列表加入集群的待调度队列；以及Add the task list to the queue to be scheduled for the cluster;
提交待调度队列中的任务到集群执行。Submit the tasks in the queue to be scheduled to the cluster for execution.
本公开实施例还提供一种大数据的任务自动化测试***，包括：The embodiment of the present disclosure further provides a task automation test system for big data, including:
数据管理模块，设置为为任务配置执行日期，获取指定日期内需要提交到集群执行的任务列表；The data management module is configured to configure an execution date for the task, and obtain a task list that needs to be submitted to the cluster for execution on the specified date;
任务管理模块，设置为将任务列表加入集群的待调度队列；a task management module, configured to add a task list to a queue to be scheduled of the cluster;
任务提交执行模块，设置为提交待调度队列中的任务到集群执行。The task submission execution module is set to submit the tasks in the queue to be scheduled to the cluster execution.
根据本公开的又一方面，提供了一种非暂态存储介质，存储有计算机可执行指令，所述计算机可执行指令设置为执行上述的大数据的任务自动化测试方法。According to still another aspect of the present disclosure, a non-transitory storage medium storing computer-executable instructions configured to perform the task automation test method of the above-described big data is provided.
根据本公开的又一方面，提供了一种计算机程序产品，所述计算机程序产品包括存储在非暂态计算机可读存储介质上的计算机程序，所述计算机程序包括程序指令，当所述程序指令被计算机执行时，使所述计算机执行上述的大数据的任务自动化测试方法。According to still another aspect of the present disclosure, a computer program product is provided, the computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions, when the program instructions When executed by a computer, the computer is caused to perform the above-described task automation test method for big data.
根据本公开的又一方面，提供了一种电子设备，包括至少一个处理器和与所述至少一个处理器通信连接的存储器，所述存储器用于存储可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行时，使所述至少一 个处理器执行上述的大数据的任务自动化测试方法。According to still another aspect of the present disclosure, there is provided an electronic device comprising at least one processor and a memory communicatively coupled to the at least one processor, the memory for storing instructions executable by the at least one processor And causing the at least one when the instruction is executed by the at least one processor The processor executes the above-described task automation test method for big data.
采用本公开的大数据的任务自动化测试方法和***，可以在有大量的任务需要运行的时候，获取指定日期需要提交到集群执行的任务，将其加入集群的待调度队列等待调度执行，相对于相关技术中，测试人员手动提交一个个任务的方式，本实施例的能实现大量任务的自动化提交和自动化执行，有效降低了测试任务量和任务繁琐度，简化了人工操作，节约了测试所需的人工成本，提高了测试效率。The task automation test method and system of the big data of the present disclosure can acquire a task that needs to be submitted to the cluster for execution on a specified date when a large number of tasks need to be run, and join the cluster to be scheduled to be scheduled for execution, as opposed to In the related art, the tester manually submits a task by itself, and the automatic submission and automatic execution of the task of the embodiment can effectively reduce the test task amount and task cumbersomeness, simplify the manual operation, and save the test requirement. Labor costs increase test efficiency.
附图说明DRAWINGS
一个或多个实施例通过与之对应的附图中的图片进行示例性说明，这些示例性说明并不构成对实施例的限定，附图中具有相同参考数字标号的元件表示为类似的元件，除非有特别申明，附图中的图不构成比例限制。The one or more embodiments are exemplified by the accompanying drawings in the accompanying drawings, and FIG. The figures in the drawings do not constitute a scale limitation unless otherwise stated.
图1为本公开实施例一提供的一种大数据的任务自动化测试方法的流程图；FIG. 1 is a flowchart of a method for automatically testing a task of big data according to Embodiment 1 of the present disclosure;
图2为本公开实施例二提供的一种大数据的任务自动化测试方法的流程图；2 is a flowchart of a task automatic test method for big data according to Embodiment 2 of the present disclosure;
图3为本公开实施例三提供的一种大数据的任务自动化测试***的模块示意图；3 is a schematic block diagram of a task automatic test system for big data according to Embodiment 3 of the present disclosure;
图4是本公开实施例提供的电子设备的结构框图。FIG. 4 is a structural block diagram of an electronic device according to an embodiment of the present disclosure.
具体实施方式detailed description
下面通过具体实施方式结合附图对本公开作进一步详细说明。在不冲突的情况下，一下实施例和实施例中的特征可以相互组合。The present disclosure will be further described in detail below with reference to the accompanying drawings. In the case of no conflict, the features of the following embodiments and embodiments may be combined with each other.
实施例一：Embodiment 1:
参见图1，本实施例提出了一种大数据的任务自动化测试方法，包括：Referring to FIG. 1, this embodiment proposes a task automatic test method for big data, including:
S110、为任务配置执行日期，获取指定日期内需要提交到集群执行的任务 列表；S110. Configure an execution date for the task, and obtain a task that needs to be submitted to the cluster for execution on the specified date. List
S120、将任务列表加入集群的待调度队列；S120. Add a task list to the to-be-scheduled queue of the cluster.
S130、提交待调度队列中的任务到集群执行。S130. Submit a task in the queue to be scheduled to the cluster for execution.
步骤S110中，为任务配置执行日期是为了确定每个任务具体的执行日期，作为后续获取指定日期的任务上位依据，在为任务配置执行日期后，才能根据指定日期获取任务列表。这里的指定日期可由测试人员自由设置，也可由***根据需要提交到集群执行的任务的实际情况自动设置指定日期可以是某个具体的日期，例如2016年10月3号，也可以是一段时间，例如3月5号到4月3号，3月5号8点到12点等等，本实施例对此没有限制。获取指定日期内需要提交到集群执行的任务列表；实际上就是执行日期为指定日期的任务列表，该任务列表中至少包括一个需要提交到集群执行的指定日期的任务。In step S110, the execution date is configured for the task to determine the specific execution date of each task, and is used as the basis for the subsequent acquisition of the specified date. After the execution date is configured for the task, the task list can be obtained according to the specified date. The specified date here can be freely set by the tester, or can be automatically set by the system according to the actual situation of the task submitted to the cluster according to the need to specify the date can be a specific date, such as October 3, 2016, or a period of time, For example, March 5 to April 3, March 5, 8:00 to 12:00, etc., this embodiment is not limited thereto. Gets the list of tasks that need to be submitted to the cluster for execution on the specified date; in fact, it is the task list whose execution date is the specified date. The task list includes at least one task that needs to be submitted to the specified date of cluster execution.
在大数据的环境下，集群需要执行的任务数量庞大，单个的存储介质取法满足存储需求，所以本实施例的任务可以存储在多个存储介质中，当步骤S110中获取指定日期执行的任务列表时，可能需要从多个存储介质中获取指定日期需要提交到集群执行的任务，生成任务列表。In the big data environment, the number of tasks that the cluster needs to perform is huge, and the single storage medium is used to satisfy the storage requirement. Therefore, the task of this embodiment may be stored in multiple storage media, and the task list executed on the specified date is acquired in step S110. When you need to obtain the tasks that need to be submitted to the cluster for the specified date from multiple storage media, you can generate a task list.
在获取到定日期内需要提交到集群执行的任务列表后，可以将任务列表加入待调度队列中排队等待集群的调度执行。其中，待调度队列是当前在排队等待集群执行的任务形成的队列。步骤S120中将任务列表加入集群的待调度队列，可以是将任务列表中的任务加入待调度队列，具体的可以是将任务列表的任务加入待调度队列的尾部，在将指定日期的任务加入待调度队列后，***会确定待调度队列中指定日期的任务，然后开始执行指定日期的任务。对于步骤S120中的集群的待调度队列，可理解为需要由该集群执行的任务形成的队列，该队列中的任务处于待调度的状态，所以可称为待调度队列，对于不同的集群而言， 待调度队列不同，所以集群的待调度队列可理解为与某个集群对应的待调度队列，S120中将任务列表加入集群的待调度队列可以是将任务列表的任务加入执行该任务的集群对应的待调度队列中。After obtaining the task list that needs to be submitted to the cluster within the specified date, the task list can be added to the queue to be scheduled to wait for the scheduling execution of the cluster. The queue to be scheduled is a queue formed by tasks currently waiting in the queue for execution. In the step S120, the task list is added to the to-be-scheduled queue of the cluster, and the task in the task list is added to the queue to be scheduled. The task may be added to the tail of the queue to be scheduled, and the task of the specified date is added to the task. After the queue is scheduled, the system determines the task for the specified date in the queue to be scheduled, and then starts the task for the specified date. For the queue to be scheduled in the cluster in step S120, it can be understood as a queue formed by tasks executed by the cluster, and the tasks in the queue are in a state to be scheduled, so it can be called a queue to be scheduled, for different clusters. , The queue to be scheduled is different, so the queue to be scheduled of the cluster can be understood as a queue to be scheduled corresponding to a cluster. The queue to be scheduled to be added to the cluster in S120 can be the task of adding the task list to the cluster performing the task. In the queue to be scheduled.
一般而言，在指定日期的任务列表加入待调度队列前，待调度队列中都会存在至少一个任务，这些任务的状态包括执行和未执行，状态为执行表明该任务已提交到集群执行，但执行过程还未结束，状态为未执行表明该任务未提交到集群。当任务列表和待调度队列发生任务重复的情况时，若不作任何修改，将任务列表中的任务加入待调度队列中，会出现重复的任务被执行的情况，会浪费集群资源，浪费时间，降低集群资源的有效利用率。Generally, before the task list of the specified date is added to the queue to be scheduled, at least one task exists in the queue to be scheduled. The status of these tasks includes execution and non-execution. The status is execution indicating that the task has been submitted to the cluster for execution, but the execution is performed. The process has not ended and the status is not executed indicating that the task was not submitted to the cluster. If the tasks in the task list and the queue to be scheduled are duplicated, if no tasks are modified, the tasks in the task list are added to the queue to be scheduled. Repeated tasks are executed, which wastes cluster resources, wastes time, and reduces Effective utilization of cluster resources.
为了克服上述缺点，提高集群资源的利用率，可以设置重复的任务只被集群执行一次，在S120之前，还可以包括，比较任务列表中的任务和待调度队列中的任务是否有重复，在出现重复的情况下，执行去重操作，使得步骤S130中待调度队列中不存在重复任务。In order to overcome the above disadvantages and improve the utilization of the cluster resources, it is possible to set the repeated tasks to be performed only once by the cluster. Before S120, the method may further include: comparing whether the tasks in the task list and the tasks in the queue to be scheduled have duplicates, In the case of repetition, the deduplication operation is performed, so that there is no duplicate task in the queue to be scheduled in step S130.
同时，考虑到待调度队列中的任务有两个状态：执行和未执行，对应的，上述去重操作的实现方式可以有两种：At the same time, considering that the task in the queue to be scheduled has two states: execution and non-execution, correspondingly, the above-mentioned deduplication operation can be implemented in two ways:
一、若待调度队列中的重复任务未执行，删除待调度队列中的重复任务或任务列表中的重复任务。1. If the repeated tasks in the queue to be scheduled are not executed, delete the duplicate tasks in the queue to be scheduled or the duplicate tasks in the task list.
同时，新获取的任务的参数配置等信息都是最新的，新获取的任务得到的执行结果相对于旧版本的任务的执行结果而言，更具备有效性和实用性。在本实施例中，指定日期的任务相对于待调度队列中的重复任务而言，是新获取的任务，基于上述的考虑，本实施例可优先保留任务列表中的重复任务，即在将任务列表中的任务加入待调度队列前，若任务列表中的任务和待调度队列中的任务有重复，优先删除待调度队列中的重复任务。 At the same time, the information such as the parameter configuration of the newly acquired task is the latest, and the execution result obtained by the newly acquired task is more effective and practical than the execution result of the task of the old version. In this embodiment, the task of the specified date is a newly acquired task with respect to the repetitive task in the queue to be scheduled. Based on the above considerations, the embodiment may preferentially retain the repetitive task in the task list, that is, the task is If a task in the list is added to the queue to be scheduled, if the task in the task list and the task in the queue to be scheduled are duplicated, the duplicate task in the queue to be scheduled is deleted first.
二、若待调度队列中的重复任务未执行，删除待调度队列中的重复任务或任务列表中的重复任务。2. If the recurring task in the queue to be scheduled is not executed, delete the repetitive task in the queue to be scheduled or the repetitive task in the task list.
当待调度队列中的重复任务已经提交到集群中执行时，该重复任务已经占用了集群资源，所以需要杀掉集群执行中的重复任务，回收该重复任务使用的集群资源。本实施例的集群资源包括但不限于内存和CPU核数。When the recurring task in the queue to be scheduled has been submitted to the cluster for execution, the repetitive task has already occupied the cluster resources. Therefore, it is necessary to kill the repetitive tasks in the cluster execution and recover the cluster resources used by the repetitive task. The cluster resources of this embodiment include, but are not limited to, memory and CPU cores.
在去除重复的任务之后，可以对当前的集群资源使用情况进行更新，确定杀死任务后回收的集群资源，以及当前总的空闲集群资源，以便进行下一个任务的调度执行。After the duplicate tasks are removed, the current cluster resource usage can be updated, and the cluster resources recovered after killing the task and the current total idle cluster resources are determined for scheduling execution of the next task.
此外，当任务列表中的任务不属于重复任务时，该任务相对于重复任务而言属于新增任务，将新增任务加入待调度队列的尾部。In addition, when the task in the task list does not belong to the repetitive task, the task belongs to the new task relative to the repetitive task, and the newly added task is added to the tail of the queue to be scheduled.
在将任务列表中的任务加入待调度队列后，可以执行S103的步骤，将待调度队列的任务提交到集群执行。在本实施例中，可以优先执行指定日期的任务。After the task in the task list is added to the queue to be scheduled, the step of S103 may be performed, and the task of the queue to be scheduled is submitted to the cluster for execution. In this embodiment, the task of the specified date can be preferentially executed.
优先提交指定日期的任务到集群执行，在一个实施例中，可以包括随机选取指定日期的任务提交到集群进行执行，或是按照某个预设规则将指定日期的任务提交到集群。但是，很明显这些提交任务的方式没有与当前的集群资源的实际情况相结合，尤其是在当前的集群资源不满足提交的指定日期任务的执行条件时，***可能会一直更新当前的集群资源，直到集群资源满足当前需要执行的任务的执行条件时，集群才开始执行任务。在更新集群资源的过程中，有一部分的集群资源在一定时间内处于闲置状态，没有被有效地利用，不利于集群资源的利用率的提高，同时也限制了集群资源处理任务的速率。The task of the specified date is preferentially submitted to the cluster for execution. In one embodiment, the task of randomly selecting the specified date may be submitted to the cluster for execution, or the task of the specified date may be submitted to the cluster according to a preset rule. However, it is obvious that the way to submit tasks is not combined with the actual situation of the current cluster resources, especially when the current cluster resources do not meet the execution conditions of the submitted specified date tasks, the system may always update the current cluster resources. The cluster does not begin execution until the cluster resource meets the execution conditions of the task that needs to be executed. During the process of updating the cluster resources, some of the cluster resources are idle for a certain period of time, and are not effectively utilized, which is not conducive to the utilization of cluster resources, and also limits the rate of cluster resource processing tasks.
为了提高集群资源的利用率，可选地，可以根据当前的集群资源的实际使用情况选择当前提交到集群执行的任务。具体的，提交待调度队列中的任务到集群执行可以包括以下两个步骤： In order to improve the utilization of the cluster resources, the tasks currently submitted to the cluster can be selected according to the actual usage of the current cluster resources. Specifically, submitting the task in the queue to be scheduled to the cluster execution may include the following two steps:
步骤一、获取当前可用的集群资源，按照预设规则从指定日期的任务中选择拟提交到集群执行的待执行任务；Step 1: Obtain the currently available cluster resources, and select the tasks to be executed to be submitted to the cluster execution from the tasks of the specified date according to the preset rules.
步骤二、如果当前可用的集群资源满足待执行任务的执行条件，则提交待执行任务至集群执行，如果当前可用的集群资源不满足待执行任务的执行条件，将待执行任务置于待调度队列尾部。Step 2: If the currently available cluster resource meets the execution condition of the task to be executed, the task to be executed is submitted to the cluster for execution. If the currently available cluster resource does not meet the execution condition of the task to be executed, the task to be executed is placed in the queue to be scheduled. Tail.
在上述的步骤一中，获取当前可用的集群资源实际上就是确定当前的空闲的内存、CPU核数等集群信息，之后，需要从指定日期的任务中选取拟提交到集群执行的任务作为待执行任务，选择的方式可以是按照预设规则进行选取，本实施例的预设规则可以根据实际情况进行设置，该预设规则包括但不限于，在指定日期的任务中随机选取任务作为待执行任务；或者，按照指定日期的任务的时间顺序进行待执行任务的选择，或者，按照指定日期的任务的重要性进行待执行任务的选择，或者，按照预先设置的指定日期的任务的优先级进行待执行任务的选择，或者，可以从指定日期任务的顶部任务向下的顺序从指定日期的任务中进行待执行任务的选择，这里的顶部任务可以理解为在待调度队列的指定日期的任务中，排在最前面的任务。In step 1 above, obtaining the currently available cluster resources is actually determining the cluster information such as the current free memory and the number of CPU cores. After that, the tasks to be submitted to the cluster are selected from the tasks of the specified date as pending execution. The task may be selected according to a preset rule. The preset rule in this embodiment may be set according to an actual situation. The preset rule includes, but is not limited to, randomly selecting a task as a task to be executed in a task of a specified date. Or, the selection of the task to be executed is performed in the chronological order of the tasks of the specified date, or the selection of the task to be executed is performed according to the importance of the task of the specified date, or the priority of the task of the specified date is set in advance. Performing the selection of the task, or selecting the task to be executed from the task of the specified date in descending order from the top task of the specified date task, where the top task can be understood as the task on the specified date of the queue to be scheduled, The top task.
在选取了待执行任务之后，判断当前的集群资源是否满足待执行任务的执行条件，将被满足的待执行任务提交待执行任务至集群执行，将不被满足的待执行任务置于待调度队列的尾部，等到下一次的调度执行。进行上述的判断时，参与判断的指定日期的任务的执行状态都是未执行。After the task to be executed is selected, it is determined whether the current cluster resource satisfies the execution condition of the task to be executed, and the to-be-executed task to be executed is submitted to the cluster for execution, and the unexecuted task to be executed is placed in the queue to be scheduled. The tail is waiting for the next scheduled execution. When the above judgment is made, the execution state of the task of the specified date participating in the judgment is not executed.
其中，任务的执行条件可以理解为执行该任务需要的内存和CPU核数等资源，将执行条件与当前的集群资源即空闲的内存和CPU核数等资源进行比较，可以得到该任务当前是否可执行的结果。在上述将被满足的待执行任务提交到集群执行后，可以将该任务的状态从未执行变为执行，在下一次选择待执行任 务时，处于执行状态的指定日期的任务不参与选择。The execution condition of the task can be understood as the resources such as the memory and the number of CPU cores required to execute the task, and the execution condition is compared with the current cluster resources, that is, the free memory and the number of CPU cores, and the task can be obtained currently. The result of the execution. After the above-mentioned task to be executed is submitted to the cluster for execution, the status of the task can be changed from execution to execution, and the next time the candidate is selected to be executed. The task of the specified date in the execution state does not participate in the selection.
在指定日期的任务的执行期间，可以监听所有的指定日期的任务的执行状态，等待指定日期的任务执行完毕，在指定日期的任务执行完毕后，再执行待调度队列中的其他任务。During the execution of the task on the specified date, you can listen to the execution status of all the tasks of the specified date, wait for the task of the specified date to complete, and execute the other tasks in the queue to be scheduled after the execution of the task on the specified date.
采用本实施例的任务自动化执行方法，可以在有大量的任务需要运行的时候，获取指定日期需要提交到集群执行的任务，将其加入集群的待调度队列等待调度执行，相对于相关技术中，测试人员手动提交一个个任务的方式，本实施例的能实现大量任务的自动化提交和自动化执行，降低测试需要的人力成本，提高测试效率。With the task auto-execution method of the embodiment, when a large number of tasks need to be run, the tasks that need to be submitted to the cluster for the specified date are acquired, and are added to the cluster to be scheduled to be scheduled for execution, compared with related technologies. The manner in which the tester manually submits one task is the automatic submission and automatic execution of a large number of tasks in the embodiment, which reduces the labor cost required for the test and improves the test efficiency.
可选地，根据集群资源的使用情况选择指定日期的任务提交到集群执行，可以有效地提高集群资源的使用率。Optionally, the task of selecting the specified date is submitted to the cluster according to the usage of the cluster resource, which can effectively improve the utilization rate of the cluster resource.
可选地，将待调度队列中的重复任务去除，首先可以节约集群资源，其次，能确保集群执行的任务是最新任务，确保得到的执行结果的有效性。Optionally, the repetitive tasks in the queue to be scheduled are removed, firstly, the cluster resources can be saved, and secondly, the tasks performed by the cluster are ensured to be the latest tasks, and the effectiveness of the obtained execution results is ensured.
实施例二：Embodiment 2:
参见图3，本实施例提供了一种大数据的任务自动化测试方法，可以将指定日期需要执行的任务加入到集群的待调度队列中优先执行，相对于相关技术中测试人员手动提交任务而言，本实施例的自动化测试方法节约了人力资源，同时也提高了测试的效率。在本实施例中，每个任务都有其对应的提交到集群执行的日期，根据该日期，可以从大量的任务中确定指定日期的任务。本实施例的集群可以采用Spark集群实现。Referring to FIG. 3, this embodiment provides a task automatic test method for big data, which can be performed by adding a task to be executed on a specified date to a queue to be scheduled, which is compared with the manual submission of a task by a tester in the related art. The automated test method of the embodiment saves human resources and also improves the efficiency of the test. In this embodiment, each task has its corresponding date submitted to the cluster for execution, and according to the date, the task of the specified date can be determined from a large number of tasks. The cluster in this embodiment can be implemented by using a Spark cluster.
本实施例的大数据的任务自动化测试方法包括：The task automation test method for big data of this embodiment includes:
S201、配置指定日期，获取指定日期内需要提交到集群执行的任务列表。S201: Configure a specified date, and obtain a task list that needs to be submitted to the cluster for execution on the specified date.
在步骤S201中，指定日期可以由测试人员指定，也可以由***根据实际情 况确定，可以是具体的日期，也可以是某一段时间。本实施例对此没有限制。In step S201, the specified date may be specified by the tester, or may be determined by the system according to the actual situation. The situation is determined, it can be a specific date, or it can be a certain period of time. This embodiment is not limited thereto.
S202、比较指定日期的任务列表中的任务和待调度队列中的任务是否有重复，若有任务重复，则进入S203，若无重复，进入S205。S202. Compare the task in the task list of the specified date with the task in the queue to be scheduled. If there is a task, go to S203. If there is no duplicate, go to S205.
S203、当待调度队列中的重复任务未提交到集群执行，则删除待调度队列中的重复任务；当待调度对列中的重复任务已提交到集群执行，杀掉集群执行中的重复任务，并回收执行重复任务使用的集群资源。S203. When the repeated tasks in the queue to be scheduled are not submitted to the cluster for execution, delete the repeated tasks in the queue to be scheduled; when the repeated tasks in the queue to be scheduled have been submitted to the cluster for execution, kill the repeated tasks in the cluster execution. And reclaim the cluster resources used to perform the repetitive tasks.
在步骤S203中，当当待调度队列中的重复任务未提交到集群执行时，也可以是删除指定日期的任务列表中的重复任务，但是考虑到指定日期的重复任务相对于待调度队列中的任务而言，配置参数等信息更新，执行指定日期的重复任务得到的结果更具有效性，保留任务列表中的重复任务更优。In step S203, when the repetitive task in the queue to be scheduled is not submitted to the cluster for execution, it may also be to delete the repetitive task in the task list of the specified date, but considering the repetitive task of the specified date relative to the task in the queue to be scheduled. In other words, information such as configuration parameters is updated, and the results of performing repeated tasks on a specified date are more effective, and the repetitive tasks in the task list are retained.
S204、更新集群资源使用情况；集群资源使用情况包括内存和CPU核数的利用情况。S204: Update the usage of the cluster resource; the usage of the cluster resource includes the utilization of the memory and the number of CPU cores.
S205、把任务列表中的任务追加到待调度队列尾部，等待调度。S205. Append the task in the task list to the tail of the queue to be scheduled, and wait for scheduling.
本实施例中，步骤S204和S205可以交换，本实施例对此没有限制。In this embodiment, the steps S204 and S205 can be exchanged, and the embodiment does not limit this.
S206、获取当前可用的集群资源。S206. Obtain a currently available cluster resource.
S207、判断当前可用的集群资源是否满足待调度队列中指定日期的顶部任务的执行要求，若当前可用的集群资源满足待调度队列中指定日期的顶部任务的执行要求，进入S208，若当前可用的集群资源不满足待调度队列中指定日期的顶部任务的执行要求，则进入S209。S207. Determine whether the currently available cluster resource meets the execution requirement of the top task on the specified date in the queue to be scheduled. If the currently available cluster resource meets the execution requirement of the top task on the specified date in the queue to be scheduled, the process proceeds to S208, if currently available. If the cluster resource does not meet the execution requirement of the top task on the specified date in the queue to be scheduled, the process proceeds to S209.
对于步骤S207中的顶部任务，理解为在待调度队列的指定日期的多个任务中，排在最前面的任务。For the top task in step S207, it is understood that among the plurality of tasks on the specified date of the queue to be scheduled, the top task is ranked.
对于步骤S207，还可以是按照预设规则从指定日期的任务中选择待执行任务，判断集群资源是否满足待执行任务的执行条件。本实施例的预设规则的解 释可以参考实施例一中对预设规则的相关描述。For the step S207, the task to be executed is selected from the tasks of the specified date according to the preset rule, and whether the cluster resource meets the execution condition of the task to be executed is determined. Solution of the preset rule of this embodiment For a reference, refer to the related description of the preset rule in the first embodiment.
S208、将当前的顶部任务放到待调度队列尾部，将顶部任务的下一个任务作为顶部任务，继续上述的判断过程。S208: The current top task is placed at the end of the queue to be scheduled, and the next task of the top task is used as the top task, and the foregoing determining process is continued.
如果步骤S207是按照预设规则从指定日期的任务中选择待执行任务，判断集群资源是否满足待执行任务的执行条件，上述步骤S208可以是将待执行任务置于待调度队列尾部，进入S207重新进行判断。If the step S207 is to select a task to be executed from the task of the specified date according to the preset rule, and determine whether the cluster resource meets the execution condition of the task to be executed, the step S208 may be that the task to be executed is placed at the end of the queue to be scheduled, and the process proceeds to S207. Make a judgment.
S209、把被满足的顶部任务提交Spark集群执行。S209. Submit the satisfied top task to the Spark cluster for execution.
S210、获取指定日期的任务的执行状态，等待任务执行结束。S210. Acquire an execution state of the task of the specified date, and wait for the execution of the task to end.
S211、判断指定日期的任务是否执行完毕，若指定日期的任务已执行完毕，则进入S212，若指定日期的任务未执行完毕，则返回S204。S211. Determine whether the task of the specified date is completed. If the task of the specified date has been executed, the process proceeds to S212. If the task of the specified date has not been executed, the process returns to S204.
S212、执行待调度队列中的其他任务。S212. Perform other tasks in the queue to be scheduled.
采用本实施例的大数据的任务自动化测试方法，能在大量任务需要执行的前提下，优先执行指定日期的任务，并能根据当前的集群资源选择可执行的任务执行，有利于提高集群资源的有效利用率和自动化测试的效率，降低测试人员的工作量，节约人力资源。The task automation test method of the big data of the embodiment can preferentially execute the task of the specified date under the premise that a large number of tasks need to be executed, and can select an executable task execution according to the current cluster resource, which is beneficial to improving the cluster resource. Effective utilization and efficiency of automated testing, reducing the workload of testers and saving human resources.
实施例三：Embodiment 3:
参见图3，本实施例提供一种大数据的任务自动化测试***，包括：Referring to FIG. 3, the embodiment provides a task automatic test system for big data, including:
数据管理模块31，设置为为任务配置执行日期，获取指定日期内需要提交到集群执行的任务列表；The data management module 31 is configured to configure an execution date for the task, and obtain a task list that needs to be submitted to the cluster for execution on the specified date;
任务管理模块32，设置为将任务列表加入集群的待调度队列；The task management module 32 is configured to add the task list to the queue to be scheduled of the cluster;
任务提交执行模块33，设置为提交待调度队列中的任务到集群执行。The task submission execution module 33 is configured to submit the tasks in the queue to be scheduled to the cluster execution.
本实施例中，每个任务的执行日期都配置有对应的执行日期，数据管理模块31可以根据该指定日期，获取指定日期内需要提交到集群执行的任务列表。 In this embodiment, the execution date of each task is configured with a corresponding execution date, and the data management module 31 can obtain a task list that needs to be submitted to the cluster for execution on the specified date according to the specified date.
其中，指定日期可由测试人员自由设置，也可由***根据需要提交到集群执行的任务的实际情况自动设置指定日期可以是某个具体的日期，例如2016年10月3号，也可以是一段时间，例如3月5号到4月3号，3月5号8点到12点等等，本实施例对此没有限制。The specified date can be set freely by the tester, or can be automatically set by the system according to the actual situation of the task submitted to the cluster according to need. The specified date can be a specific date, such as October 3, 2016, or a period of time. For example, March 5 to April 3, March 5, 8:00 to 12:00, etc., this embodiment is not limited thereto.
在大数据的环境下，集群需要执行的任务数量庞大，单个的存储介质取法满足存储需求，所以任务可能会存储在多个存储介质中，数据管理模块31获取指定日期执行的任务列表时，可以从多个存储介质中获取指定日期需要提交到集群执行的任务，生成任务列表。In the big data environment, the number of tasks that the cluster needs to perform is huge, and the single storage medium takes the storage requirements, so the task may be stored in multiple storage media. When the data management module 31 obtains the task list executed on the specified date, A task list is generated by obtaining tasks from a plurality of storage media that need to be submitted to the cluster for the specified date.
在获取到定日期内需要提交到集群执行的任务列表后，任务管理模块32可以将任务列表加入待调度队列中排队等待集群的调度执行。其中，待调度队列可以为当前在排队等待集群执行的任务形成的队列。任务管理模块32将任务列表加入待调度队列中，可以是将任务列表中的任务加入待调度队列，具体的可以是将任务列表的任务加入待调度队列的尾部。After obtaining the task list that needs to be submitted to the cluster within the scheduled date, the task management module 32 may add the task list to the scheduled queue to wait for the scheduling execution of the cluster. The queue to be scheduled may be a queue formed by tasks currently waiting to be executed by the cluster. The task management module 32 adds the task list to the queue to be scheduled, and may add the task in the task list to the queue to be scheduled. Specifically, the task of the task list may be added to the tail of the queue to be scheduled.
由于在指定日期的任务列表加入待调度队列前，待调度队列中一般会存在至少一个任务，这些任务的状态包括执行和未执行，状态为执行表明该任务已提交到集群执行，但执行过程还未结束，状态为未执行表明该任务未提交到集群。当任务列表和待调度队列发生任务重复的情况时，若不作任何修改，将任务列表加入待调度队列中，重复的任务会被执行至少两次，这无疑会浪费集群资源，浪费时间，降低集群资源的有效利用率。Since the task list on the specified date is added to the queue to be scheduled, there are generally at least one task in the queue to be scheduled. The status of these tasks includes execution and non-execution. The status indicates that the task has been submitted to the cluster for execution, but the execution process is still performed. Not completed, the status is not executed indicating that the task was not submitted to the cluster. When the task list and the queue to be scheduled are duplicated, if no modification is made, the task list is added to the queue to be scheduled, and the repeated tasks are executed at least twice, which will waste the cluster resources, waste time, and reduce the cluster. Effective utilization of resources.
可选地，本实施例的任务管理模块32还设置为比较任务列表中的任务和待调度队列中的任务是否有重复，并在待调度队列中出现重复任务时，执行去重操作，以便任务列表的任务加入后，待调度队列中不存在重复任务。Optionally, the task management module 32 of the embodiment is further configured to compare whether the tasks in the task list and the tasks in the queue to be scheduled have duplicates, and perform a deduplication operation when the repeated tasks occur in the queue to be scheduled, so that the tasks are performed. After the task of the list is added, there are no duplicate tasks in the queue to be scheduled.
本实施例提供了两种去除重复任务的方式，一种是，在待调度队列中的重 复任务未执行时，任务管理模块32删除待调度队列中的重复任务或任务列表中的重复任务；另一种是在待调度队列中的重复任务已提交至集群执行时，任务管理模块32杀掉集群执行中的重复任务，回收重复任务使用的集群资源。其中，集群资源包括但不限于内存和CPU核数。This embodiment provides two ways to remove duplicate tasks, one is the weight in the queue to be scheduled. When the complex task is not executed, the task management module 32 deletes the repeated task in the queue to be scheduled or the repeated task in the task list; the other is that when the repeated task in the queue to be scheduled has been submitted to the cluster for execution, the task management module 32 kills The repetitive tasks in the cluster execution are removed, and the cluster resources used by the repetitive tasks are collected. Among them, cluster resources include but are not limited to memory and CPU cores.
新获取的任务的参数配置等信息都是最新的，其执行结果相对于旧版本的重复任务的执行结果而言，明显更具备有效性和实用性，在本实施例中，指定日期的任务相对于待调度队列中的重复任务而言，是新获取的任务。基于上述的考虑，在将任务列表中的任务加入待调度队列前，若任务列表中的任务和待调度队列中的任务有重复，可选的，任务管理模块32删除待调度队列中的重复任务。The information such as the parameter configuration of the newly acquired task is up-to-date, and the execution result is obviously more effective and practical than the execution result of the repetitive task of the old version. In this embodiment, the task of the specified date is relatively In the case of a repetitive task in the queue to be scheduled, it is a newly acquired task. Based on the above considerations, before the tasks in the task list are added to the queue to be scheduled, if the tasks in the task list and the tasks in the queue to be scheduled are duplicated, the task management module 32 deletes the repeated tasks in the queue to be scheduled. .
此外，当任务列表中的任务不属于重复任务时，该任务相对于重复任务而言属于新增任务，任务管理模块32可以将新增任务加入待调度队列的尾部。In addition, when the task in the task list does not belong to the repetitive task, the task belongs to the new task relative to the repetitive task, and the task management module 32 can add the newly added task to the tail of the queue to be scheduled.
本实施例中的任务管理模块32将任务列表加入待调度队列后，任务提交执行模块33可以开始提交指定日期的任务。After the task management module 32 in this embodiment adds the task list to the queue to be scheduled, the task submission execution module 33 can start submitting the task of the specified date.
考虑到在执行的指定日期的任务时，可能存在集群资源不满足任务的执行条件的情况，此时，***可能会一直更新当前的集群资源，等待集群资源满足该任务的执行条件后再执行该任务，由此，一部分集群资源会在一定时间内闲置，降低集群资源的有效利用率。基于上述的考虑，可选的，本实施例的任务提交执行模块33设置为获取当前可用的集群资源，按照预设规则从指定日期的任务中选择拟提交到集群执行的待执行任务；如果当前可用的集群资源满足待执行任务的执行条件，则提交待执行任务至集群执行，如果当前可用的集群资源不满足待执行任务的执行条件，将待执行任务置于待调度队列尾部。When the task of the specified date is executed, there may be cases where the cluster resource does not meet the execution condition of the task. At this time, the system may update the current cluster resource all the time, waiting for the cluster resource to meet the execution condition of the task, and then executing the Tasks, as a result, some cluster resources will be idle for a certain period of time, reducing the effective utilization of cluster resources. Based on the above considerations, optionally, the task submission execution module 33 of the present embodiment is configured to acquire the currently available cluster resources, and select a task to be executed to be submitted to the cluster execution from the tasks of the specified date according to a preset rule; If the available cluster resources meet the execution conditions of the task to be executed, the task to be executed is submitted to the cluster for execution. If the currently available cluster resources do not meet the execution conditions of the task to be executed, the task to be executed is placed at the end of the queue to be scheduled.
其中，任务提交执行模块33获取当前可用的集群资源实际上就是确定当前 的空闲的内存、CPU核数等集群信息。之后，任务提交执行模块33需要从指定日期的任务中选取拟提交到集群执行的任务作为待执行任务，选择的方式可以是按照预设规则进行选取，本实施例的预设规则可以根据实际情况进行设置，该预设规则包括但不限于，在指定日期的任务中随机选取任务作为待执行任务；或者，按照指定日期的任务的时间顺序进行待执行任务的选择，或者，按照指定日期的任务的重要性进行待执行任务的选择，或者，按照预先设置的指定日期的任务的优先级进行待执行任务的选择，或者，可以从指定日期任务的顶部任务向下的顺序从指定日期的任务中进行待执行任务的选择，这里的顶部任务可以理解为在待调度队列的指定日期的任务中，排在最前面的任务。The task submission execution module 33 obtains the currently available cluster resources, which is actually determining the current Cluster information such as free memory and CPU core count. Then, the task submission execution module 33 needs to select a task to be submitted to the cluster to be executed as a task to be executed from the task of the specified date, and the selected method may be selected according to a preset rule. The preset rule in this embodiment may be based on actual conditions. Setting, including but not limited to, randomly selecting a task as a task to be executed in a task of a specified date; or selecting a task to be executed according to a time sequence of tasks of a specified date, or a task according to a specified date The importance of the selection of the task to be executed, or the selection of the task to be executed according to the priority of the task of the specified date specified in advance, or the order from the top task of the specified date task to the task of the specified date The selection of the task to be executed, the top task here can be understood as the top task in the task of the specified date of the queue to be scheduled.
在选取了待执行任务之后，任务提交执行模块33可以判断当前的集群资源是否满足待执行任务的执行条件，将被满足的待执行任务提交待执行任务至集群执行，将不被满足的待执行任务置于待调度队列的尾部，等到下一次的调度执行。可以理解的是，进行上述的判断时，参与判断的指定日期的任务的执行状态都是未执行。After the task to be executed is selected, the task submission execution module 33 may determine whether the current cluster resource meets the execution condition of the task to be executed, and submit the to-be-executed task to the cluster execution, and the unsatisfied to be executed. The task is placed at the end of the queue to be scheduled and waits until the next scheduled execution. It can be understood that, when the above determination is made, the execution state of the task of the specified date participating in the judgment is not executed.
为了确定指定日期的任务进度，本实施例的***还可以包括监听模块34，设置为在指定日期的任务的执行期间，监听所有的指定日期的任务的执行状态，并在指定日期的任务执行完毕后，通知任务提交执行模块33提交待调度队列中的其他任务至集群执行。In order to determine the progress of the task on the specified date, the system of the present embodiment may further include a listening module 34 configured to listen to the execution status of all tasks of the specified date during execution of the task of the specified date, and execute the task on the specified date. Afterwards, the notification task submission execution module 33 submits other tasks in the queue to be scheduled to the cluster execution.
采用本实施例的大数据的任务自动化测试***，利用数据管理模块可以获取指定日期需要提交到集群执行的任务列表，相对于相关技术中测试人员手动添加待测试的任务的方式，本实施例具有自动化提交任务的优点，任务管理模块和任务提交执行模块配合可以实现大量任务需要执行的前提下，优先执行指定日期的任务的目的，同时可以根据集群资源的使用情况选择可执行的指定日 期的任务执行，有效地提高了集群资源的利用率和测试效率。With the task automation test system of the big data of the embodiment, the data management module can obtain the task list that needs to be submitted to the cluster for execution on the specified date, and the method has the method of manually adding the task to be tested by the tester in the related art. The advantages of automating the submission of tasks, the task management module and the task submission execution module can achieve the purpose of performing tasks with a specified date on the premise that a large number of tasks need to be executed, and can select an executable designated date according to the usage of the cluster resources. The task execution of the period effectively improves the utilization of cluster resources and the efficiency of testing.
本公开还提供了一种非暂态存储介质，存储有计算机可执行指令，所述计算机可执行指令设置为执行上述实施例的大数据的任务自动化测试方法。The present disclosure also provides a non-transitory storage medium storing computer executable instructions arranged to perform a task automation test method for big data of the above-described embodiments.
本公开还提供了一种计算机程序产品，所述计算机程序产品包括存储在非暂态计算机可读存储介质上的计算机程序，所述计算机程序包括程序指令，当所述程序指令被计算机执行时，使所述计算机执行上述实施例的大数据的任务自动化测试方法。The present disclosure also provides a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions, when the program instructions are executed by a computer, The computer is caused to execute the task automation test method of the big data of the above embodiment.
本公开还提供了一种电子设备。图4是本公开实施例提供的电子设备的结构框图。该电子设备可以包括：处理器(processor)51和存储器(memory)53，还可以包括通信接口(Communications Interface)52和总线54。其中，处理器51、通信接口52、存储器53可以通过总线54完成相互间的通信。通信接口52可以用于信息传输。处理器51可以调用存储器53中的逻辑指令，以执行上述实施例的大数据的任务自动化测试方法。The present disclosure also provides an electronic device. FIG. 4 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device may include a processor 51 and a memory 53, and may further include a communication interface 52 and a bus 54. The processor 51, the communication interface 52, and the memory 53 can complete communication with each other through the bus 54. Communication interface 52 can be used for information transmission. The processor 51 can call the logic instructions in the memory 53 to perform the task automation test method of the big data of the above embodiment.
此外，上述的存储器53中的逻辑指令可以通过软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本公开的技术方案可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本公开各个实施例所述方法的全部或部分步骤。而前述的存储介质可以是非暂态存储介质，包括：U盘、移动硬盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质，也可以是暂态存储介质。In addition, the logic instructions in the memory 53 described above may be implemented in the form of a software functional unit and sold or used as a stand-alone product, and may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present disclosure may be embodied in the form of a software product stored in a storage medium, including a plurality of instructions for causing a computer device (which may be a personal computer, a server, or a network) The device or the like) performs all or part of the steps of the method described in various embodiments of the present disclosure. The foregoing storage medium may be a non-transitory storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. A medium that can store program code, or a transitory storage medium.
以上仅为本公开的部分实施例，并非因此限制本公开的专利范围，凡是利 用本公开说明书及附图内容所作的等效结构或等效流程变换，或直接或间接运用在其他相关的技术领域，均同理包括在本公开的专利保护范围内。The above is only some of the embodiments of the present disclosure, and thus does not limit the scope of the patent of the present disclosure. The equivalent structure or equivalent flow transformations made by the disclosure and the contents of the drawings, or directly or indirectly applied to other related technical fields, are included in the scope of patent protection of the present disclosure.
工业实用性Industrial applicability
本实施例的大数据的任务自动化测试***，利用数据管理模块可以获取指定日期需要提交到集群执行的任务列表，相对于相关技术中测试人员手动添加待测试的任务的方式，本实施例具有自动化提交任务的优点，任务管理模块和任务提交执行模块配合可以实现大量任务需要执行的前提下，优先执行指定日期的任务的目的，同时可以根据集群资源的使用情况选择可执行的指定日期的任务执行，有效地提高了集群资源的利用率和测试效率。 The task automatic test system of the big data in this embodiment can use the data management module to obtain a task list that needs to be submitted to the cluster for execution on a specified date, and the embodiment is automated in comparison with the manner in which the tester manually adds the task to be tested in the related art. The advantages of submitting the task, the task management module and the task submission execution module can realize the purpose of executing the tasks of the specified date on the premise that a large number of tasks need to be executed, and at the same time, can select the executable task of the specified date according to the use of the cluster resources. , effectively improve the utilization of cluster resources and test efficiency.

Claims (13)

  1. 一种大数据的任务自动化测试方法，包括：A task automation test method for big data, including:
    为任务配置执行日期，获取指定日期内需要提交到集群执行的任务列表；Configure the execution date for the task to obtain a list of tasks that need to be submitted to the cluster for execution on the specified date;
    将所述任务列表加入所述集群的待调度队列；以及Adding the task list to the queue to be scheduled of the cluster;
    提交所述待调度队列中的任务到所述集群执行。Submitting the tasks in the queue to be scheduled to the cluster for execution.
  2. 如权利要求1所述的大数据的任务自动化测试方法，其中，所述提交所述待调度队列中的任务到所述集群执行包括：The task automation test method for big data according to claim 1, wherein the submitting the task in the queue to be scheduled to the cluster execution comprises:
    获取当前可用的集群资源，按照预设规则从所述指定日期的任务中选择拟提交到所述集群执行的待执行任务；Obtaining a currently available cluster resource, and selecting, from a task of the specified date, a task to be executed to be executed by the cluster according to a preset rule;
    如果所述当前可用的集群资源满足所述待执行任务的执行条件，提交所述待执行任务至所述集群执行，如果所述当前可用的集群资源不满足所述待执行任务的执行条件，将所述待执行任务置于所述待调度队列尾部。If the currently available cluster resource meets the execution condition of the to-be-executed task, submit the to-be-executed task to the cluster for execution, and if the currently available cluster resource does not satisfy the execution condition of the to-be-executed task, The task to be executed is placed at the end of the queue to be scheduled.
  3. 如权利要求1或2所述的大数据的任务自动化测试方法，其中，在所述任务列表加入所述集群的待调度队列前，还包括：The task automation test method for big data according to claim 1 or 2, wherein before the task list is added to the queue to be scheduled of the cluster, the method further includes:
    比较所述任务列表中的任务和所述待调度队列中的任务是否有重复，并在所述待调度队列中出现重复任务时，执行去重操作。And comparing whether the task in the task list and the task in the to-be-scheduled queue are duplicated, and performing a deduplication operation when a recurring task occurs in the to-be-scheduled queue.
  4. 如权利要求3所述的大数据的任务自动化测试方法，其中，所述去重操作包括：The task automation test method of big data according to claim 3, wherein the deduplication operation comprises:
    若所述待调度队列中的重复任务未执行，删除所述待调度队列中的重复任务或所述任务列表中的重复任务。If the repeated task in the to-be-scheduled queue is not executed, the repeated task in the to-be-scheduled queue or the repeated task in the task list is deleted.
  5. 如权利要求3所述的大数据的任务自动化测试方法，其中，所述去重操作包括： The task automation test method of big data according to claim 3, wherein the deduplication operation comprises:
    若所述待调度队列中的重复任务已提交至所述集群执行，杀掉所述集群执行中的所述重复任务，回收所述重复任务使用的集群资源。And if the repetitive task in the to-be-scheduled queue has been submitted to the cluster for execution, killing the repetitive task in the cluster execution, and reclaiming the cluster resource used by the repetitive task.
  6. 一种大数据的任务自动化测试***，包括：A task automation test system for big data, including:
    数据管理模块，设置为为任务配置执行日期，获取指定日期内需要提交到集群执行的任务列表；The data management module is configured to configure an execution date for the task, and obtain a task list that needs to be submitted to the cluster for execution on the specified date;
    任务管理模块，设置为将所述任务列表加入所述集群的待调度队列；以及a task management module, configured to add the task list to a queue to be scheduled of the cluster;
    任务提交执行模块，设置为提交所述待调度队列中的任务到所述集群执行。The task submission execution module is configured to submit the task in the to-be-scheduled queue to the cluster for execution.
  7. 如权利要求6所述的大数据的任务自动化测试***，其中，所述任务提交执行模块，设置为取当前可用的集群资源，按照预设规则从所述指定日期的任务中选择拟提交到所述集群执行的待执行任务；如果所述当前可用的集群资源满足所述待执行任务的执行条件，提交所述待执行任务至所述集群执行，如果所述当前可用的集群资源不满足所述待执行任务的执行条件，将所述待执行任务置于所述待调度队列尾部。The task automation test system of claim 6, wherein the task submission execution module is configured to take the currently available cluster resources, and select a proposed submission from the tasks of the specified date according to a preset rule. The task to be executed executed by the cluster; if the currently available cluster resource satisfies the execution condition of the task to be executed, submit the task to be executed to the cluster, if the currently available cluster resource does not meet the The execution condition of the task to be executed is placed at the end of the queue to be scheduled.
  8. 如权利要求6或7所述的大数据的任务自动化测试***，其中，所述任务管理模块还设置为，在所述任务列表加入所述集群的待调度队列前，比较所述任务列表中的任务和所述待调度队列中的任务是否有重复，并在所述待调度队列中出现重复任务时，执行去重操作。The task automation module of the big data according to claim 6 or 7, wherein the task management module is further configured to compare the task list before adding the task list to the queue to be scheduled Whether the task and the task in the queue to be scheduled have duplicates, and when a duplicate task occurs in the to-be-scheduled queue, performing a deduplication operation.
  9. 如权利要求8所述的大数据的任务自动化测试***，其中，所述任务管理模块设置为：The task automation test system of big data according to claim 8, wherein said task management module is configured to:
    若所述待调度队列中的重复任务未执行，删除所述待调度队列中的重复任务或所述任务列表中的重复任务。 If the repeated task in the to-be-scheduled queue is not executed, the repeated task in the to-be-scheduled queue or the repeated task in the task list is deleted.
  10. 如权利要求8所述的大数据的任务自动化测试***，其中，所述任务管理模块设置为：The task automation test system of big data according to claim 8, wherein said task management module is configured to:
    若所述待调度队列中的重复任务已提交至所述集群执行，杀掉所述集群执行中的所述重复任务，回收所述重复任务使用的集群资源。And if the repetitive task in the to-be-scheduled queue has been submitted to the cluster for execution, killing the repetitive task in the cluster execution, and reclaiming the cluster resource used by the repetitive task.
  11. 一种非暂态存储介质，存储有计算机可执行指令，所述计算机可执行指令设置为执行权利要求1-5任一项所述的大数据的任务自动化测试方法。A non-transitory storage medium storing computer executable instructions, the computer executable instructions being arranged to perform the task automation test method of the big data of any of claims 1-5.
  12. 一种计算机程序产品，所述计算机程序产品包括存储在非暂态计算机可读存储介质上的计算机程序，所述计算机程序包括程序指令，当所述程序指令被计算机执行时，使所述计算机执行权利要求1-5任一项所述的大数据的任务自动化测试方法。A computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions that, when executed by a computer, cause the computer to execute The task automation test method for big data according to any one of claims 1 to 5.
  13. 一种电子设备，包括至少一个处理器和与所述至少一个处理器通信连接的存储器，所述存储器用于存储可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行时，使所述至少一个处理器执行权利要求1-5任一项所述的大数据的任务自动化测试方法。 An electronic device comprising at least one processor and a memory communicatively coupled to the at least one processor, the memory for storing instructions executable by the at least one processor, the instructions being processed by the at least one When executed, the at least one processor is caused to perform the task automation test method of the big data according to any one of claims 1-5.
PCT/CN2016/103969 2016-06-07 2016-10-31 Task automation testing method and system for big data WO2017211042A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610404001.1A CN107480041A (en) 2016-06-07 2016-06-07 The task automation method of testing and system of a kind of big data
CN201610404001.1 2016-06-07

Publications (1)

Publication Number Publication Date
WO2017211042A1 true WO2017211042A1 (en) 2017-12-14

Family

ID=60577629

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/103969 WO2017211042A1 (en) 2016-06-07 2016-10-31 Task automation testing method and system for big data

Country Status (2)

Country Link
CN (1) CN107480041A (en)
WO (1) WO2017211042A1 (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111291893A (en) * 2020-01-22 2020-06-16 合肥本源量子计算科技有限责任公司 Scheduling method, scheduling system, storage medium, and electronic apparatus
CN111416839A (en) * 2020-02-26 2020-07-14 平安科技（深圳）有限公司 Cluster environment timed task processing method, system, device and storage medium

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109902023B (en) * 2019-03-18 2022-06-03 平安普惠企业管理有限公司 Test code handover control method and device
CN110134520A (en) * 2019-05-27 2019-08-16 眸芯科技（上海）有限公司 The application method and system of integrated circuit scarce resource based on queuing
CN110851363A (en) * 2019-11-12 2020-02-28 广东电网有限责任公司 Cloud testing system and method
CN112817808B (en) * 2019-11-18 2024-04-19 赵伟 Computer cluster maintenance task management method and system
CN113190335B (en) * 2021-05-07 2023-05-26 安徽南瑞中天电力电子有限公司 Multi-task scheduling and collecting method of power collecting terminal and power collecting system
CN113535560B (en) * 2021-07-14 2024-06-25 杭州网易云音乐科技有限公司 Test execution method, device, storage medium and computing equipment

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103645909A (en) * 2013-12-30 2014-03-19 中国烟草总公司湖南省公司 Handling method and device for timed task
CN103678133A (en) * 2013-12-18 2014-03-26 中国科学院深圳先进技术研究院 Task scheduling system for application software cloud testing
CN103678132A (en) * 2013-12-18 2014-03-26 中国科学院深圳先进技术研究院 Application software cloud testing system
CN103870348A (en) * 2012-12-14 2014-06-18 中国电信股份有限公司 Test method and system for concurrent user access
US20150268995A1 (en) * 2010-08-19 2015-09-24 International Business Machines Corporation Selective constant complexity dismissal in task scheduling
CN105045710A (en) * 2015-06-30 2015-11-11 吉林大学 Method for automatically generating test data in cloud computing environment

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150268995A1 (en) * 2010-08-19 2015-09-24 International Business Machines Corporation Selective constant complexity dismissal in task scheduling
CN103870348A (en) * 2012-12-14 2014-06-18 中国电信股份有限公司 Test method and system for concurrent user access
CN103678133A (en) * 2013-12-18 2014-03-26 中国科学院深圳先进技术研究院 Task scheduling system for application software cloud testing
CN103678132A (en) * 2013-12-18 2014-03-26 中国科学院深圳先进技术研究院 Application software cloud testing system
CN103645909A (en) * 2013-12-30 2014-03-19 中国烟草总公司湖南省公司 Handling method and device for timed task
CN105045710A (en) * 2015-06-30 2015-11-11 吉林大学 Method for automatically generating test data in cloud computing environment

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111291893A (en) * 2020-01-22 2020-06-16 合肥本源量子计算科技有限责任公司 Scheduling method, scheduling system, storage medium, and electronic apparatus
CN111291893B (en) * 2020-01-22 2023-06-02 合肥本源量子计算科技有限责任公司 Scheduling method, scheduling system, storage medium and electronic device
CN111416839A (en) * 2020-02-26 2020-07-14 平安科技（深圳）有限公司 Cluster environment timed task processing method, system, device and storage medium
CN111416839B (en) * 2020-02-26 2022-09-23 平安科技（深圳）有限公司 Cluster environment timing task processing method, system, device and storage medium

Also Published As

Publication number Publication date
CN107480041A (en) 2017-12-15

Similar Documents

Publication Publication Date Title
WO2017211042A1 (en) Task automation testing method and system for big data
JP5990192B2 (en) Filtering query data in the data store
CN108255620B (en) Service logic processing method, device, service server and system
CN108549583B (en) Big data processing method and device, server and readable storage medium
US20160139949A1 (en) Virtual machine resource management system and method thereof
CN104199739B (en) A kind of speculating type Hadoop dispatching methods based on load balancing
WO2019062699A1 (en) Resource scheduling method, scheduling server, cloud computing system and storage medium
WO2019148713A1 (en) Sql statement processing method and apparatus, computer device, and storage medium
WO2022151668A1 (en) Data task scheduling method and apparatus, storage medium, and scheduling tool
CN111475564A (en) Streaming data processing method, system, computer equipment and storage medium
CN111045933A (en) Regression strategy updating method and device, storage medium and terminal equipment
US10241828B2 (en) Method and system for scheduling transactions in a data system
Petrov et al. Adaptive performance model for dynamic scaling Apache Spark Streaming
CN111367591B (en) Spark task processing method and device
CN111913793A (en) Distributed task scheduling method, device, node equipment and system
CN106686619B (en) Performance evaluation method and equipment
CN114510317A (en) Virtual machine management method, device, equipment and storage medium
CN112035236B (en) Task scheduling method, device and storage medium based on multi-factor cooperation
Ouyang et al. An approach for modeling and ranking node-level stragglers in cloud datacenters
CN110891083A (en) Agent method for supporting multi-job parallel execution in Gaia
WO2016101115A1 (en) Resource scheduling method and related apparatus
CN112100186B (en) Data processing method and device based on distributed system and computer equipment
CN107885549B (en) Method and system for clearing residual process in TORQUE computing cluster computing node
CN116089248B (en) Write I/O burst distribution prediction method, device, equipment and storage medium
CN111352724B (en) Method and device for realizing security resource selection

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16904491

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16904491

Country of ref document: EP

Kind code of ref document: A1