CN116820727A - Task processing method and device, electronic equipment and storage medium - Google Patents
Task processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN116820727A CN116820727A CN202310890814.6A CN202310890814A CN116820727A CN 116820727 A CN116820727 A CN 116820727A CN 202310890814 A CN202310890814 A CN 202310890814A CN 116820727 A CN116820727 A CN 116820727A
- Authority
- CN
- China
- Prior art keywords
- task
- detection
- processed
- initial
- qualified
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003860 storage Methods 0.000 title claims abstract description 17
- 238000003672 processing method Methods 0.000 title abstract description 15
- 238000001514 detection method Methods 0.000 claims abstract description 182
- 238000000034 method Methods 0.000 claims abstract description 49
- 238000012795 verification Methods 0.000 claims abstract description 42
- 238000012360 testing method Methods 0.000 claims abstract description 30
- 238000012545 processing Methods 0.000 claims abstract description 28
- 230000008569 process Effects 0.000 claims description 13
- 238000010200 validation analysis Methods 0.000 claims 1
- 238000011161 development Methods 0.000 abstract description 12
- 238000012423 maintenance Methods 0.000 abstract description 3
- 238000004519 manufacturing process Methods 0.000 description 9
- 238000004891 communication Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 8
- 238000004590 computer program Methods 0.000 description 7
- 230000002159 abnormal effect Effects 0.000 description 5
- 238000004458 analytical method Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 3
- 230000000903 blocking effect Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 239000002699 waste material Substances 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The disclosure provides a task processing method, a device, an electronic device and a storage medium, wherein the method comprises the following steps: loading an initial task; performing standard detection on the initial task; testing and verifying the initial tasks which are qualified in the standard detection, and determining that the initial tasks which are qualified in the testing and verification are tasks to be processed; running a task to be processed; when the operation of the task to be processed is detected to be finished, acquiring an operation result of the task to be processed; performing data quality detection on the operation result, and prompting the operation result based on the data quality detection result, wherein the data quality detection comprises: the file size of the output of the task to be processed, the scale of the output data amount and the accuracy of the output content. When the method is applied to processing the task, the standard detection of the task is added, so that the task submitted to the running environment meets the unified standard requirement, the difficulty of collaborative development and maintenance is reduced, the verification of the data quality is carried out after the task is completed, and the accuracy and the effectiveness of the verification are improved.
Description
Technical Field
The disclosure relates to the field of computer technology, and in particular, to a task processing method, a task processing device, electronic equipment and a storage medium.
Background
The existing big data task scheduling system can realize scheduling of time dimension and dependency relationship dimension, and development and submission of tasks are performed in a user interface mode. However, the existing task scheduling system does not achieve good support in terms of task development, task execution efficiency optimization and task result quality, so that the problems that the existing task scheduling system is uncontrollable in resource consumption, high in cost, incapable of guaranteeing an execution result, poor in data quality, influenced in customer use and the like occur when the task is processed.
Disclosure of Invention
The disclosure provides a task processing method, a task processing device, electronic equipment and a storage medium, so as to at least solve the technical problems in the prior art.
According to a first aspect of the present disclosure, there is provided a task processing method, the method comprising: loading an initial task; performing standard detection on the initial task; performing test verification on the initial task which is qualified in specification detection, and determining that the initial task which is qualified in test verification is a task to be processed; running the task to be processed; when the operation of the task to be processed is detected to be finished, acquiring an operation result of the task to be processed; performing data quality detection on the operation result, and prompting the operation result based on the data quality detection result, wherein the data quality detection comprises: and whether the file size, the data size and the content of the output of the task to be processed are accurate or not.
In an embodiment, the performing the specification detection on the initial task includes: performing specification detection on the initial task based on a preset task specification, wherein the specification detection comprises grammar detection and format detection; if all the grammar detection and format detection of the initial task are qualified, determining that the specification detection of the initial task is qualified; and if at least one detection failure exists in the grammar detection and the format detection of the initial task, determining that the specification detection of the initial task is failed.
In one embodiment, the method further comprises: and generating a prompt message for the initial task with unqualified standard detection, wherein the prompt message comprises a unqualified reason and a unqualified position, modifying the initial task according to the reason and the position, and carrying out standard detection on the modified initial task until the standard detection of the initial task is qualified.
In an embodiment, the determining the initial task qualified by the test is determined as a task to be processed, including: if the fact that the current time meets the starting processing time of the initial task is detected, all the leading tasks with the dependency relationship with the initial task are run and completed, the initial task is set to be in a ready state, and the initial task is determined to be a task to be processed.
In an embodiment, the running the task to be processed includes: analyzing the task to be processed, splitting the task to be processed into a plurality of subtasks, and determining the dependency relationship among the subtasks; based on the dependency relationship among the plurality of subtasks, the subtasks are operated, the subtasks with the dependency relationship are operated according to the dependency relationship sequence, and the subtasks without the dependency relationship are operated in parallel.
In one embodiment, the method further comprises: the method comprises the steps of adjusting resources allocated to a target task in real time in the running process of the task to be processed, wherein the target task is the task to be processed or the subtask, and the method comprises the following steps: determining an amount of required resources to run the target task and a corresponding amount of allocated resources; if the required resource amount is larger than the allocated resource amount, allocating a specific amount of new resources for the target task, so that the updated allocated resource amount meets a first resource threshold; and if the required resource amount is smaller than the allocated resource amount, recovering a specific amount of resources from the allocated resources of the target task, so that the updated allocated resource amount meets a second resource threshold.
In an embodiment, the prompting the operation result based on the data quality detection result includes: if the data quality detection is qualified, determining that the task to be processed is completed, and sending a normal task completion prompt; if the data quality detection is unqualified and the dependency relationship between the post task and the operation result of the task to be processed is detected, the operation of the post task is blocked, and a first prompt of task failure is sent; if the data quality detection is not qualified and the fact that the running result of the post task and the running result of the task to be processed do not have a dependency relationship is detected, the running of the post task is not blocked, and a second prompt of task failure is sent out.
According to a second aspect of the present disclosure, there is provided a task processing device, the device comprising: the loading module is used for loading the initial task; the first detection module is used for carrying out standard detection on the initial task; the verification module is used for carrying out test verification on the initial task which is qualified in specification detection, and determining that the initial task which is qualified in test verification is a task to be processed; the operation module is used for operating the task to be processed; the acquisition module is used for acquiring the operation result of the task to be processed when the operation of the task to be processed is detected to be finished; the second detection module is used for carrying out data quality detection on the operation result and prompting the operation result based on the data quality detection result, and the data quality detection comprises the following steps: and whether the file size, the data size and the content of the output of the task to be processed are accurate or not.
In an embodiment, the first detection module is specifically configured to perform specification detection on the initial task based on a preset task specification, where the specification detection includes syntax detection and format detection; if all the grammar detection and format detection of the initial task are qualified, determining that the specification detection of the initial task is qualified; and if at least one detection failure exists in the grammar detection and the format detection of the initial task, determining that the specification detection of the initial task is failed.
In an embodiment, the device further comprises: the prompt module is used for generating a prompt message for the initial task with unqualified standard detection, wherein the prompt message comprises a unqualified reason and a unqualified position, so that the initial task is modified according to the reason and the position, and standard detection is carried out on the modified initial task until the standard detection of the initial task is qualified.
In an embodiment, the verification module is further configured to set the initial task to be in a ready state and determine that the initial task is a task to be processed if it is detected that the current time meets a start processing time of the initial task and all the leading tasks having a dependency relationship with the initial task are completed.
In an embodiment, the operation module includes: the analysis sub-module is used for analyzing the task to be processed, splitting the task to be processed into a plurality of sub-tasks and determining the dependency relationship among the plurality of sub-tasks; and the running sub-module is used for running the sub-tasks based on the dependency relationship among the plurality of sub-tasks, wherein the sub-tasks with the dependency relationship run according to the dependency relationship sequence, and the sub-tasks without the dependency relationship run in parallel.
In an embodiment, the device further comprises: the adjusting module is used for adjusting resources allocated to a target task in real time in the running process of the task to be processed, wherein the target task is the task to be processed or the subtask; the adjustment module comprises: a determining sub-module for determining an amount of required resources to run the target task and a corresponding amount of allocated resources; an adjustment sub-module, configured to allocate a specific amount of new resources to the target task if the required resource amount is greater than the allocated resource amount, so that the updated allocated resource amount meets a first resource threshold; and if the required resource amount is smaller than the allocated resource amount, recovering a specific amount of resources from the allocated resources of the target task, so that the updated allocated resource amount meets a second resource threshold.
In an embodiment, the second detection module is further configured to determine that the task to be processed is completed and send a normal task completion prompt if the data quality is detected to be qualified; if the data quality detection is unqualified and the dependency relationship between the post task and the operation result of the task to be processed is detected, the operation of the post task is blocked, and a first prompt of task failure is sent; if the data quality detection is not qualified and the fact that the running result of the post task and the running result of the task to be processed do not have a dependency relationship is detected, the running of the post task is not blocked, and a second prompt of task failure is sent out.
According to a third aspect of the present disclosure, there is provided an electronic device comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the methods described in the present disclosure.
According to a fourth aspect of the present disclosure, there is provided a non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of the present disclosure.
The method, the device, the electronic equipment and the storage medium for processing the task comprise the steps of firstly loading an initial task; performing standard detection on the initial task, performing test verification on the initial task which is qualified in standard detection, and determining the initial task which is qualified in test verification as a task to be processed; running a task to be processed; when the operation of the task to be processed is detected to be finished, an operation result of the task to be processed is obtained, and data quality detection is carried out on the operation result. When the method is applied to processing the task, the standard detection of the task is added in the development stage of the task, so that the task submitted to the production environment meets the unified standard requirement, the difficulty of collaborative development and maintenance is reduced, the test verification is carried out on the initial task qualified in standard detection, the normal operation of the production environment is prevented from being influenced when the problem is caused by directly operating the initial task, the efficiency of the development verification of the task is improved, the verification of the data quality is finally carried out after the task is completed, and the accuracy and the effectiveness of the verification are improved.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.
Drawings
The above, as well as additional purposes, features, and advantages of exemplary embodiments of the present disclosure will become readily apparent from the following detailed description when read in conjunction with the accompanying drawings. Several embodiments of the present disclosure are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings, in which:
in the drawings, the same or corresponding reference numerals indicate the same or corresponding parts.
Fig. 1 shows a schematic implementation flow diagram of a task processing method according to an embodiment of the disclosure;
fig. 2 illustrates a second implementation flow diagram of a task processing method according to an embodiment of the disclosure;
FIG. 3 illustrates a block diagram of a task processing device according to an embodiment of the present disclosure;
fig. 4 shows a schematic diagram of a composition structure of an electronic device according to an embodiment of the disclosure.
Detailed Description
In order to make the objects, features and advantages of the present disclosure more comprehensible, the technical solutions in the embodiments of the present disclosure will be clearly described in conjunction with the accompanying drawings in the embodiments of the present disclosure, and it is apparent that the described embodiments are only some embodiments of the present disclosure, but not all embodiments. Based on the embodiments in this disclosure, all other embodiments that a person skilled in the art would obtain without making any inventive effort are within the scope of protection of this disclosure.
Fig. 1 shows a schematic implementation flow diagram of a task processing method according to an embodiment of the disclosure, including:
step 101, loading an initial task.
And 102, performing standard detection on the initial task.
And in the development stage of the task, after the initial task is loaded, the initial task is subjected to standard detection so as to ensure that the initial task meets the unified standard requirement before running.
And step 103, testing and verifying the initial task which is qualified in the standard detection, and determining that the initial task which is qualified in the test and verification is a task to be processed.
After the initial task is subjected to standard detection, submitting the initial task which is qualified in standard detection to a designated task verification cluster for verification, wherein the environment formed by the designated task verification cluster is an environment consistent with the production environment. The initial tasks which are qualified in standard detection are tested and verified, so that the problem that the normal operation of the production environment is affected when the tasks are directly submitted to the production environment after development is completed can be avoided.
In the test verification process, if the initial task runs successfully on the designated task verification cluster, the initial task is determined to run normally, otherwise, the initial task is determined to not run normally. When the initial task can normally run, inquiring the result data verified by the test, judging whether the running result of the initial task is correct according to the result data, and determining whether the task logic of the initial task is accurate. And determining the initial task which is successfully operated and has the correct operation result as a task to be processed, wherein the task to be processed is a task which can be operated in a production environment.
Through the test verification process, the problems of grammar errors, logic errors, inaccurate data and the like in the initial task with qualified standard detection can be further eliminated, and the abnormal operation of the initial task in the production environment is avoided.
If the initial task test verification passing the standard detection is not passed, generating a prompt message according to the reason of the failure and the position of the failure, carrying out standard detection again on the modified initial task after corresponding modification according to the prompt message, and carrying out test verification again after the standard detection until the initial task can run successfully on the designated task verification cluster, and the result data of the test verification is correct.
Step 104, running the task to be processed.
And when the specification detection and the test verification of the initial task are qualified, determining that the initial task is a task to be processed, and submitting the task to be processed to a formal environment for operation. In the actual running process, the dependency relationship between the tasks to be processed in the preset time period can be determined, and a corresponding directed acyclic graph is generated so as to run the tasks to be processed according to the dependency relationship between the tasks to be processed. The preset time period may be set according to practical situations, for example, may be a day or a week.
And 105, when the operation of the task to be processed is detected to be finished, acquiring an operation result of the task to be processed.
And sending out a prompt message for ending the task operation when the task operation to be processed is ended, and acquiring the operation result of the task to be processed after the prompt message is monitored. When the operation result of the task to be processed is that the operation is successful, the data quality detection can be carried out on the operation result, and when the operation result of the task to be processed is that the operation is failed, the task to be processed is adjusted according to the failure reason prompted in the failure pool and is operated again.
Further, a failure threshold may be set, and if the number of times that the task to be processed fails to run reaches the failure threshold, the task to be processed is not run any more.
Step 106, performing data quality detection on the operation result, and prompting the operation result based on the data quality detection result, wherein the data quality detection comprises: the file size of the output of the task to be processed, the scale of the output data amount and the accuracy of the output content.
And detecting the data quality of the operation result of the task to be processed according to a preset detection standard, wherein the data quality detection comprises the size of a file produced by the task to be processed, the scale of the data volume produced, the range of the data volume produced, whether the content of the produced is correct or not and the like. When the data quality detection result meets the preset detection standard, confirming that the task is normally completed, and generating a task normal completion prompt; and when the data quality detection result does not accord with the preset detection standard, confirming that the task is not completed normally, and generating a task abnormal completion prompt.
According to the task processing method provided by the embodiment of the scheme, after the task processing method is loaded into the initial task, the initial task is subjected to standard detection and test verification, the initial task which is qualified in both standard detection and test verification is determined to be the task to be processed, then the task to be processed is operated, the operation result of the task to be processed is obtained, the operation result is subjected to data quality detection, and the operation result is prompted based on the data quality detection result. When the method is used for processing the task, the standard inspection of the task is added in the development stage of the task, so that the task submitted to the running environment meets the unified standard requirement, the difficulty of collaborative development and maintenance is reduced, the verification of the data quality is carried out after the task is completed, and the accuracy and the effectiveness of the verification are improved.
In one embodiment, performing canonical detection on an initial task includes:
performing specification detection on the initial task based on a preset task specification, wherein the specification detection comprises grammar detection and format detection; if all the grammar detection and format detection of the initial task are qualified, determining that the specification detection of the initial task is qualified; if at least one of grammar detection and format detection of the initial task is unqualified, determining that specification detection of the initial task is unqualified.
The task development has strict standard, and in the stage of task development, the inspection of the task is added, so that the initial task can be ensured to meet the specification requirement. Grammar detection comprises sentence grammar, field naming and the like; the format detection includes writing a format, statement specifications, and the like. When the initial task meets the corresponding specification requirements in terms of grammar detection and format detection, the specification detection of the initial task is considered to be qualified, and if any item which does not meet the corresponding specification requirements in terms of grammar detection and format detection exists, the specification detection of the initial task is considered to be unqualified.
In addition, the specification detection can be expanded and regulated in multiple aspects according to the actually running task types, and different specification detections are provided for different task types to detect whether the task meets the specification.
In one embodiment, the method further comprises:
and generating a prompt message for the initial task with unqualified standard detection, wherein the prompt message comprises the unqualified reason and the unqualified position, modifying the initial task according to the reason and the position, and carrying out standard detection on the modified initial task until the standard detection of the initial task is qualified.
When the specification detection of the initial task is failed, determining the failed reason and the failed position, and generating a prompt message, wherein the prompt message prompts a developer that the initial task specification detection is failed, and the failed specific position and the failed reason, such as the format detection failure of the prompt message as the initial task and the specific part of the statement are failed, so that the developer can correspondingly modify the initial task according to the prompt message. And then carrying out specification detection on the modified initial task, if the specification detection is qualified, carrying out subsequent steps, and if the specification detection is still unqualified, continuing modification until the specification detection of the initial task is qualified.
In one embodiment, determining that the initial task qualified by the test is a task to be processed includes: if the fact that the current time meets the starting processing time of the initial task and all the leading tasks with the dependency relationship with the initial task are completed in running is detected, the initial task is set to be in a ready state, and the initial task is determined to be a task to be processed.
The corresponding running time is set when the task runs, for example, ten points of a certain task are preset to start running, and the task starts running when the task runs at the ten points. After standard detection and test verification of the initial task are qualified, if the current time meets the preset starting processing time of the initial task, judging whether the initial task has a leading task, if not, setting the initial task as a ready state, and determining the initial task as a task to be processed.
If the initial task has a leading task, judging whether all the leading tasks with the dependency relationship with the initial task are completely run, if so, setting the initial task as a ready state, and determining that the initial task is a task to be processed; if one leading task is not run and completed, setting the initial task as a waiting state until all the leading tasks with the dependency relationship with the initial task are completed, and setting the initial task as a ready state.
If the current time does not meet the starting processing time of the preset initial task, the initial task is set to be in a waiting state as well, until the current time meets the starting processing time of the preset initial task and all the leading tasks with the dependency relationship with the initial task are completed, and the initial task is set to be in a ready state.
When detecting that the production environment has spare resources, the task in the ready state can be submitted to the production environment for operation.
In one embodiment, as shown in fig. 2, running the task to be processed includes:
step 201, analyzing a task to be processed, splitting the task to be processed into a plurality of subtasks, and determining the dependency relationship among the subtasks;
step 202, based on the dependency relationship among the plurality of subtasks, running the subtasks with the dependency relationship according to the dependency relationship sequence, and running the subtasks without the dependency relationship in parallel.
Before the task to be processed runs formally, analyzing the task to be processed, splitting the task to be processed into a plurality of subtasks according to analysis results, determining the dependency relationship among the subtasks, and constructing a corresponding directed acyclic graph according to the dependency relationship among the subtasks.
When the task to be processed is operated, the operation is carried out according to the dependency relationship among the subtasks, the subtasks with the dependency relationship are operated according to the dependency relationship sequence, and the subtasks without the dependency relationship can be operated in parallel, so that the task execution speed can be increased, and unnecessary waiting in the task operation process is avoided.
When the task to be processed is disassembled into a plurality of subtasks, the dependent parameters among the subtasks are ensured to be unchanged, and abnormal task operation is avoided.
In addition, if the task to be processed cannot be split into sub-tasks after analysis, the task to be processed can be directly operated.
In one embodiment, the method further comprises: the method comprises the steps of adjusting resources allocated to a target task in real time in the running process of the task to be processed, wherein the target task is the task to be processed or a subtask, and the method comprises the following steps:
determining the required resource quantity for running the target task and the corresponding allocated resource quantity;
if the required resource amount is larger than the allocated resource amount, a specific amount of new resources are allocated to the target task, so that the updated allocated resource amount meets a first resource threshold;
and if the required resource amount is smaller than the allocated resource amount, recovering a specific amount of resources from the allocated resource amount of the target task, so that the updated allocated resource amount meets a second resource threshold.
In the process of running the task to be processed, the resources allocated to the target task are adjusted in real time, if the task to be processed does not split the subtasks, the target task is the task to be processed, and if the task to be processed splits the subtasks, each subtask is the target task.
Firstly, determining the required resource quantity of a target task to be operated, comparing the required resource quantity with the allocated resource quantity of the target task, if the required resource quantity of the target task is larger than the allocated resource quantity, the operation of the target task is failed due to insufficient resources, and then, allocating a specific quantity of new resources for the target task, so that the updated allocated resource quantity of the target task meets a first resource threshold value, and the first resource threshold value is slightly larger than the required resource quantity of the target task so as to ensure the smooth operation of the target task.
If the required resource amount of the target task is smaller than the allocated resource amount, the resources are occupied, so that a specific amount of resources are recovered from the allocated resource amount of the target task, the updated allocated resource amount meets a second resource threshold, and the second resource threshold is slightly larger than the required resource amount of the target task, so that smooth running of the target task is ensured, and the resources are not occupied to cause resource waste.
Further, if the required resource amount of the target task is smaller than the allocated resource amount, comparing the allocated resource amount with a second resource threshold, and if the allocated resource amount is smaller than the second resource threshold, allocating a specific amount of new resources to the target task so that the allocated resource amount of the target task meets the second resource threshold; and if the allocated resource amount is larger than the second resource threshold, recovering a specific amount of resources from the allocated resources of the target task, so that the allocated resource amount of the target task meets the second resource threshold.
If the failure of the target task due to insufficient resources is detected in the running process of the target task, resources can be reassigned for the target task, so that the target task can run successfully.
By adjusting the resources in real time, unnecessary resource waste caused by setting larger memory or CPU resources in advance under the condition that the task needs more resources can be avoided.
In addition, after the task processing is completed, the required resource amount in the running process of the target task or the updated allocated resource amount can be counted, so that the reference can be provided for the allocated resource according to the counted data when the same task is run next time.
In one embodiment, the prompting of the operation result based on the data quality detection result includes:
if the data quality detection is qualified, determining that the task to be processed is completed, and sending a normal completion prompt of the task;
if the data quality detection is unqualified and the dependency relationship between the operation results of the post task and the task to be processed is detected, the operation of the post task is blocked, and a first prompt of task failure is sent;
if the data quality detection is unqualified and the running result of the post task and the task to be processed is detected to have no dependency, the operation of the post task is insufficient, and a second prompt of task failure is sent out.
And (3) carrying out data quality detection on an operation result of task operation, and when all data quality detection items are qualified, determining that the data quality detection is qualified, successfully completing the task to be processed and sending a normal completion prompt of the task. And when the data quality detection item is in a disqualification condition, determining that the data quality detection is disqualification, and sending a task abnormal completion prompt when the task to be processed is not successfully completed.
When the data quality detection is unqualified, if the dependency relationship between the post task of the task to be processed and the operation result of the task to be processed is detected, stopping the operation of the post task, and avoiding the influence of the operation result on the operation of the post task; and sending out a first task failure prompt, wherein the first task failure prompt is an alarm prompt, and prompting a developer to correct the task to be processed as soon as possible. If the fact that the post task of the task to be processed and the operation result of the task to be processed do not have a dependency relationship is detected, namely the operation result has no influence on the operation of the post task, the post task can continue to operate, but a task failure second prompt is sent out, the task failure second prompt is a prompt, and a developer is prompted to attach importance to the task to be processed.
By detecting the data quality of the operation result of the task to be processed, the accuracy and the effectiveness of verification are improved, blocking or non-blocking of the post task can be realized, and a prompt effect can be achieved under the condition of abnormal data quality detection so that a developer can intervene in the processing as soon as possible.
Fig. 3 shows a block diagram of a task processing device according to an embodiment of the present disclosure.
Referring to fig. 3, according to a second aspect of the embodiments of the present disclosure, there is provided a task processing device, the device including: the loading module 301 is configured to load an initial task; the first detection module 302 is configured to perform standard detection on an initial task; the verification module 303 is configured to perform test verification on an initial task that is qualified in specification detection, and determine that the initial task that is qualified in test verification is a task to be processed; the operation module 304 is configured to operate a task to be processed; the acquiring module 305 is configured to acquire an operation result of the task to be processed when detecting that the task to be processed is completed; the second detection module 306 is configured to perform data quality detection on the operation result, and perform prompting of the operation result based on the data quality detection result, where the data quality detection includes: the file size of the output of the task to be processed, the scale of the output data amount and the accuracy of the output content.
In an embodiment, the first detection module 302 is specifically configured to perform specification detection on the initial task based on a preset task specification, where the specification detection includes syntax detection and format detection; if all the grammar detection and format detection of the initial task are qualified, determining that the specification detection of the initial task is qualified; if at least one of the grammar detection and the format detection of the initial task is unqualified, determining that the specification detection of the initial task is unqualified.
In an embodiment, the apparatus further comprises: the prompt module 307 is configured to generate a prompt message for detecting an unqualified initial task, where the prompt message includes a reason for the unqualified initial task and a position of the unqualified initial task, modify the initial task according to the reason and the position, and perform the specification detection on the modified initial task until the specification detection of the initial task is qualified.
In an embodiment, the verification module 303 is further configured to set the initial task to be in a ready state and determine the initial task to be a task to be processed if it is detected that the current time satisfies the start processing time of the initial task and all the preceding tasks having a dependency relationship with the initial task are completed.
In one embodiment, the run module 304 includes: the analysis sub-module 3041 is used for analyzing the task to be processed, splitting the task to be processed into a plurality of sub-tasks and determining the dependency relationship among the plurality of sub-tasks; the running sub-module 3042 is configured to run sub-tasks based on the dependency relationships among the plurality of sub-tasks, where the sub-tasks with the dependency relationships run in a dependency relationship order, and the sub-tasks without the dependency relationships run in parallel.
In an embodiment, the apparatus further comprises: the adjustment module 308, the adjustment module is configured to adjust, in real time, resources allocated to a target task during an operation process of a task to be processed, where the target task is a task to be processed or a subtask, and the adjustment module 308 includes: a determining submodule 3081 for determining a required resource amount for running the target task and a corresponding allocated resource amount; an adjustment sub-module 3082, configured to allocate a specific amount of new resources to the target task if the required amount of resources is greater than the allocated amount of resources, so that the updated allocated amount of resources meets the first resource threshold; and if the required resource amount is smaller than the allocated resource amount, recovering a specific amount of resources from the allocated resources of the target task, so that the updated allocated resource amount meets a second resource threshold.
In an embodiment, the second detection module 306 is further configured to determine that the task to be processed is completed if the data quality is detected to be qualified, and send a normal completion prompt for the task; if the data quality detection is unqualified and the dependency relationship between the operation results of the post task and the task to be processed is detected, the operation of the post task is blocked, and a first prompt of task failure is sent; if the data quality detection is unqualified and the fact that the running result of the post task and the running result of the task to be processed do not have a dependency relationship is detected, the running of the post task is not blocked, and a second prompt of task failure is sent.
According to embodiments of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.
Fig. 4 illustrates a schematic block diagram of an example electronic device 400 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 4, the apparatus 400 includes a computing unit 401 that can perform various suitable actions and processes according to a computer program stored in a Read Only Memory (ROM) 402 or a computer program loaded from a storage unit 408 into a Random Access Memory (RAM) 403. In RAM 403, various programs and data required for the operation of device 400 may also be stored. The computing unit 401, ROM 402, and RAM 403 are connected to each other by a bus 404. An input/output (I/O) interface 405 is also connected to bus 404.
Various components in device 400 are connected to I/O interface 405, including: an input unit 406 such as a keyboard, a mouse, etc.; an output unit 407 such as various types of displays, speakers, and the like; a storage unit 408, such as a magnetic disk, optical disk, etc.; and a communication unit 409 such as a network card, modem, wireless communication transceiver, etc. The communication unit 409 allows the device 400 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
The computing unit 401 may be a variety of general purpose and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 401 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the respective methods and processes described above, for example, a task processing method. For example, in some embodiments, a task processing method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as the storage unit 408. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 400 via the ROM 402 and/or the communication unit 409. When a computer program is loaded into RAM 403 and executed by computing unit 401, one or more steps of one task processing method described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform a task processing method in any other suitable way (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel or sequentially or in a different order, provided that the desired results of the technical solutions of the present disclosure are achieved, and are not limited herein.
Furthermore, the terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In the description of the present disclosure, the meaning of "a plurality" is two or more, unless explicitly defined otherwise.
The foregoing is merely specific embodiments of the disclosure, but the protection scope of the disclosure is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the disclosure, and it is intended to cover the scope of the disclosure. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.
Claims (10)
1. A method of task processing, the method comprising:
loading an initial task;
performing standard detection on the initial task;
performing test verification on the initial task which is qualified in specification detection, and determining that the initial task which is qualified in test verification is a task to be processed;
running the task to be processed;
when the operation of the task to be processed is detected to be finished, acquiring an operation result of the task to be processed;
performing data quality detection on the operation result, and prompting the operation result based on the data quality detection result, wherein the data quality detection comprises: and whether the file size, the data size and the content of the output of the task to be processed are accurate or not.
2. The method of claim 1, wherein said performing canonical detection on the initial task comprises:
performing specification detection on the initial task based on a preset task specification, wherein the specification detection comprises grammar detection and format detection;
if all the grammar detection and format detection of the initial task are qualified, determining that the specification detection of the initial task is qualified;
and if at least one detection failure exists in the grammar detection and the format detection of the initial task, determining that the specification detection of the initial task is failed.
3. The method according to claim 1 or 2, characterized in that the method further comprises:
and generating a prompt message for the initial task with unqualified standard detection, wherein the prompt message comprises a unqualified reason and a unqualified position, modifying the initial task according to the reason and the position, and carrying out standard detection on the modified initial task until the standard detection of the initial task is qualified.
4. The method of claim 1, wherein the determining that the initial task that is qualified for test validation is a task to be processed comprises:
if the fact that the current time meets the starting processing time of the initial task is detected, all the leading tasks with the dependency relationship with the initial task are run and completed, the initial task is set to be in a ready state, and the initial task is determined to be a task to be processed.
5. The method of claim 1, wherein the running the task to be processed comprises:
analyzing the task to be processed, splitting the task to be processed into a plurality of subtasks, and determining the dependency relationship among the subtasks;
based on the dependency relationship among the plurality of subtasks, the subtasks are operated, the subtasks with the dependency relationship are operated according to the dependency relationship sequence, and the subtasks without the dependency relationship are operated in parallel.
6. The method according to claim 1 or 5, characterized in that the method further comprises: the method comprises the steps of adjusting resources allocated to a target task in real time in the running process of the task to be processed, wherein the target task is the task to be processed or the subtask, and the method comprises the following steps:
determining an amount of required resources to run the target task and a corresponding amount of allocated resources;
if the required resource amount is larger than the allocated resource amount, allocating a specific amount of new resources for the target task, so that the updated allocated resource amount meets a first resource threshold;
and if the required resource amount is smaller than the allocated resource amount, recovering a specific amount of resources from the allocated resources of the target task, so that the updated allocated resource amount meets a second resource threshold.
7. The method of claim 1, wherein the prompting of the operation result based on the data quality detection result comprises:
if the data quality detection is qualified, determining that the task to be processed is completed, and sending a normal task completion prompt;
if the data quality detection is unqualified and the dependency relationship between the post task and the operation result of the task to be processed is detected, the operation of the post task is blocked, and a first prompt of task failure is sent;
If the data quality detection is not qualified and the fact that the running result of the post task and the running result of the task to be processed do not have a dependency relationship is detected, the running of the post task is not blocked, and a second prompt of task failure is sent out.
8. A task processing device, the device comprising:
the loading module is used for loading the initial task;
the first detection module is used for carrying out standard detection on the initial task;
the verification module is used for carrying out test verification on the initial task which is qualified in specification detection, and determining that the initial task which is qualified in test verification is a task to be processed;
the operation module is used for operating the task to be processed;
the acquisition module is used for acquiring the operation result of the task to be processed when the operation of the task to be processed is detected to be finished;
the second detection module is used for carrying out data quality detection on the operation result and prompting the operation result based on the data quality detection result, and the data quality detection comprises the following steps: and whether the file size, the data size and the content of the output of the task to be processed are accurate or not.
9. An electronic device, comprising:
At least one processor; and
a memory communicatively coupled to the at least one processor; wherein,,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.
10. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310890814.6A CN116820727A (en) | 2023-07-19 | 2023-07-19 | Task processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310890814.6A CN116820727A (en) | 2023-07-19 | 2023-07-19 | Task processing method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116820727A true CN116820727A (en) | 2023-09-29 |
Family
ID=88124058
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310890814.6A Pending CN116820727A (en) | 2023-07-19 | 2023-07-19 | Task processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116820727A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105589750A (en) * | 2015-07-07 | 2016-05-18 | 杭州华三通信技术有限公司 | CPU (Central Processing Unit) resource scheduling method and server |
| CN111241122A (en) * | 2020-01-07 | 2020-06-05 | 广州虎牙科技有限公司 | Task monitoring method and device, electronic equipment and readable storage medium |
| CN113360269A (en) * | 2021-06-29 | 2021-09-07 | 平安普惠企业管理有限公司 | Task allocation method, device, server and storage medium |
| CN113419839A (en) * | 2021-07-20 | 2021-09-21 | 北京字节跳动网络技术有限公司 | Resource scheduling method and device for multi-type jobs, electronic equipment and storage medium |
| CN114691241A (en) * | 2022-04-19 | 2022-07-01 | 中煤航测遥感集团有限公司 | Task execution method and device, electronic equipment and storage medium |
-
2023
- 2023-07-19 CN CN202310890814.6A patent/CN116820727A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105589750A (en) * | 2015-07-07 | 2016-05-18 | 杭州华三通信技术有限公司 | CPU (Central Processing Unit) resource scheduling method and server |
| CN111241122A (en) * | 2020-01-07 | 2020-06-05 | 广州虎牙科技有限公司 | Task monitoring method and device, electronic equipment and readable storage medium |
| CN113360269A (en) * | 2021-06-29 | 2021-09-07 | 平安普惠企业管理有限公司 | Task allocation method, device, server and storage medium |
| CN113419839A (en) * | 2021-07-20 | 2021-09-21 | 北京字节跳动网络技术有限公司 | Resource scheduling method and device for multi-type jobs, electronic equipment and storage medium |
| CN114691241A (en) * | 2022-04-19 | 2022-07-01 | 中煤航测遥感集团有限公司 | Task execution method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108804215B (en) | Task processing method and device and electronic equipment | |
| CN109710624B (en) | Data processing method, device, medium and electronic equipment | |
| CN109977012B (en) | Joint debugging test method, device, equipment and computer readable storage medium of system | |
| CN113377614B (en) | Call chain information generation method and device, electronic equipment and storage medium | |
| CN116841655A (en) | Script execution method, script execution device, script execution equipment, storage medium and script execution product | |
| CN113778849A (en) | Method, apparatus, device and storage medium for testing code | |
| CN115437961A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN116820727A (en) | Task processing method and device, electronic equipment and storage medium | |
| CN115481594B (en) | Scoreboard implementation method, scoreboard, electronic equipment and storage medium | |
| CN114116688A (en) | Data processing and data quality inspection method, device and readable storage medium | |
| CN116401113B (en) | Environment verification method, device and medium for heterogeneous many-core architecture acceleration card | |
| CN115374010A (en) | Function testing method, device, equipment and storage medium | |
| CN115952100B (en) | Interface testing method, device, system, electronic equipment and storage medium | |
| CN116737428B (en) | Air quality mode operation stability checking method and device and electronic equipment | |
| CN117290113B (en) | Task processing method, device, system and storage medium | |
| CN117609064A (en) | Unit test method and device, electronic equipment and storage medium | |
| CN117609087A (en) | Code processing method, device, equipment and medium | |
| CN115309717A (en) | Database loading method, device, equipment, medium and computer program product | |
| CN115374012A (en) | Automatic regression testing method and device, electronic equipment and storage medium | |
| CN114064002A (en) | Software development kit generation method, device, equipment and storage medium | |
| CN116991825A (en) | Database flashback method, device, equipment and storage medium | |
| CN114238075A (en) | Method, device, equipment, medium and product for testing program component | |
| CN117648252A (en) | Function test method and device for software application, electronic equipment and storage medium | |
| CN116303098A (en) | Application testing method and device, electronic equipment and storage medium | |
| CN114996157A (en) | Method, device, equipment and storage medium for identifying risk of changing code |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |