CN113204692A - Method and device for monitoring execution progress of data processing task - Google Patents
Method and device for monitoring execution progress of data processing task Download PDFInfo
- Publication number
- CN113204692A CN113204692A CN202110587044.9A CN202110587044A CN113204692A CN 113204692 A CN113204692 A CN 113204692A CN 202110587044 A CN202110587044 A CN 202110587044A CN 113204692 A CN113204692 A CN 113204692A
- Authority
- CN
- China
- Prior art keywords
- data processing
- processing task
- data
- time length
- progress
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 154
- 238000012544 monitoring process Methods 0.000 title claims abstract description 47
- 238000000034 method Methods 0.000 title claims abstract description 44
- 238000004364 calculation method Methods 0.000 claims description 10
- 230000008569 process Effects 0.000 abstract description 8
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 2
- 238000007418 data mining Methods 0.000 description 2
- 230000003111 delayed effect Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003796 beauty Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/907—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2465—Query processing support for facilitating data mining operations in structured databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0631—Resource planning, allocation, distributing or scheduling for enterprises or organisations
- G06Q10/06311—Scheduling, planning or task assignment for a person or group
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0637—Strategic management or analysis, e.g. setting a goal or target of an organisation; Planning actions based on goals; Analysis or evaluation of effectiveness of goals
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Human Resources & Organizations (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Strategic Management (AREA)
- Entrepreneurship & Innovation (AREA)
- Economics (AREA)
- General Physics & Mathematics (AREA)
- Educational Administration (AREA)
- Game Theory and Decision Science (AREA)
- Operations Research (AREA)
- Data Mining & Analysis (AREA)
- General Business, Economics & Management (AREA)
- Development Economics (AREA)
- Tourism & Hospitality (AREA)
- Quality & Reliability (AREA)
- General Engineering & Computer Science (AREA)
- Marketing (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Library & Information Science (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application discloses a method and a device for monitoring execution progress of a data processing task. Wherein, the method comprises the following steps: acquiring an identifier of the data processing task, and searching data to be processed corresponding to the data processing task according to the identifier; determining a data amount of data to be processed and a currently available computing resource of a device performing a data processing task; and monitoring the execution progress of the data processing task according to the data volume of the data to be processed and the currently available computing resources. The method and the device solve the technical problems that in the process of processing and analyzing the crowd packet data uploaded by the client by using the CDP system, the execution progress of the data processing task cannot be dynamically monitored in real time, the execution progress of the data processing task cannot be mastered, and the use experience of the user is influenced.
Description
Technical Field
The present application relates to the field of data processing, and in particular, to a method and an apparatus for monitoring an execution progress of a data processing task.
Background
The main function of a Client Data Platform (CDP) system is Data insight, wherein a client uploads crowd Data to the system, and analysis dimensions are set for analysis to obtain a desired result. The speed of analyzing the result by the system is most concerned by the client, the time schedule of task execution is not monitored at present, the client can see the result when the task is executed, or the client fails to obtain the desired result after the task is executed for a long time, so that the user experience is poor and the use of the client is delayed frequently.
At present, only the final result (success/failure) of task execution can be alarmed, and the defects are that the lag is not timely and intelligent, and related personnel cannot be informed to timely process the task in advance. Because it also fails at the business level if the execution time of the task greatly exceeds the time required by the client to reach the result.
Aiming at the problems that the execution progress of a data processing task cannot be monitored dynamically in real time in the process of processing and analyzing the crowd packet data uploaded by a client by using a CDP system at present, so that the execution progress of the data processing task cannot be mastered, and the use experience of a user is influenced, an effective solution is not provided at present.
Disclosure of Invention
The embodiment of the application provides a method and a device for monitoring the execution progress of a data processing task, so as to at least solve the technical problems that the execution progress of the data processing task cannot be monitored dynamically in real time in the process of processing and analyzing crowd packet data uploaded by a client by using a CDP system, the execution progress of the data processing task cannot be mastered, and the use experience of the user is influenced.
According to an aspect of the embodiments of the present application, there is provided a method for monitoring the execution progress of a data processing task, including: acquiring an identifier of the data processing task, and searching data to be processed corresponding to the data processing task according to the identifier; determining a data amount of data to be processed and a currently available computing resource of a device performing a data processing task; and monitoring the execution progress of the data processing task according to the data volume of the data to be processed and the currently available computing resources.
Optionally, the monitoring the execution progress of the data processing task according to the data amount of the data to be processed and the currently available computing resource includes: determining a first time length required for executing a data processing task under the current available computing resource according to the data volume of the data to be processed and the current available computing resource; and comparing the first time length with a first preset time length, and if the first time length exceeds the first preset time length, reducing the data volume of the data to be processed or increasing the available computing resources of the equipment executing the data processing task.
Optionally, the monitoring the execution progress of the data processing task according to the data amount of the data to be processed and the currently available computing resource further includes: acquiring a second time length required by the data processing task to complete the first target progress; comparing the second time length with a second preset time length, wherein the second preset time length is the estimated time length required by the data processing task to complete the first target progress; and if the second time length is not matched with the second preset time length, generating an alarm signal.
Optionally, the obtaining of the second time length required by the data processing task to complete the first target progress includes at least one of the following: acquiring a second time length required by the data processing task to complete the first target progress according to a preset progress interval of the data processing task execution progress; and acquiring a second time length required by the data processing task to complete the first target progress according to a preset time interval.
Optionally, determining currently available computing resources of a device performing a data processing task comprises: and determining a cluster calculation force value of the equipment according to the queue idle resource value, the cluster whole idle resource value of the equipment and a resource value to be released by the equipment within a preset time period, wherein the cluster calculation force value is used for representing the current available calculation resources of the equipment, and the queue is used for processing the data processing task.
Optionally, the method further includes: if the available computing resources are increased, determining a third time length required for completing the second target progress when the remaining progress of the data processing task is executed according to the current completion progress of the data processing task and the newly increased available computing resources; comparing the third time length with a third preset time length, wherein the third preset time length is the estimated time length required by the data processing task to complete the second target progress; and if the third time length is not matched with the third preset time length, generating an alarm signal.
According to another aspect of the embodiments of the present application, there is also provided a device for monitoring the execution progress of a data processing task, including: the acquisition module is used for acquiring the identifier of the data processing task and searching the data to be processed corresponding to the data processing task according to the identifier; a determining module for determining a data amount of data to be processed and a currently available computing resource of a device performing a data processing task; and the monitoring module is used for monitoring the execution progress of the data processing task according to the data volume of the data to be processed and the currently available computing resources.
Optionally, the monitoring module comprises: the determining unit is used for determining a first time length required for executing a data processing task under the current available computing resource according to the data volume of the data to be processed and the current available computing resource; and the comparison unit is used for comparing the first time length with a first preset time length, and reducing the data volume of the data to be processed or increasing the available computing resources of the equipment executing the data processing task if the first time length exceeds the first preset time length.
According to another aspect of the embodiments of the present application, a non-volatile storage medium is further provided, where the non-volatile storage medium includes a stored program, and the method for monitoring the execution progress of the data processing task performed by the device where the non-volatile storage medium is located is controlled during program execution.
According to still another aspect of the embodiments of the present application, there is provided a processor, configured to run a program stored in a memory, where the program runs to perform the above method for monitoring the execution progress of a data processing task.
In the embodiment of the application, the identification of the data processing task is obtained, and the data to be processed corresponding to the data processing task is searched according to the identification; determining a data amount of data to be processed and a currently available computing resource of a device performing a data processing task; the method monitors the execution progress of the data processing task according to the data volume of the data to be processed and the currently available computing resource, and monitors the execution progress of the data processing task through the data volume of the data to be processed and the currently available computing resource of the equipment executing the data processing task, thereby realizing real-time dynamic monitoring of the execution progress of the data processing task, greatly improving the control degree of important tasks and the technical effect of user experience, and further solving the technical problems that the execution progress of the data processing task cannot be monitored dynamically in real time in the process of processing and analyzing crowd packet data uploaded by a client by using a CDP system at present, the execution progress of the data processing task cannot be controlled, and the use experience of the user is influenced.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
FIG. 1 is a flow chart of a method for monitoring progress of execution of a data processing task according to an embodiment of the present application;
fig. 2 is a block diagram of a device for monitoring the progress of execution of a data processing task according to an embodiment of the present application.
Detailed Description
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the application described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
First, some terms or terms appearing in the description of the embodiments of the present application are applicable to the following explanations:
CDP (customer Data platform), a customer Data platform that is based on the first party Data of the enterprise.
And (4) data insights, namely converting data into information through data analysis/mining, combing out factors and action links influencing a service result by combining a service scene, and further correctly attributing the problems and obtaining an improved direction.
The crowd bag is used for classifying the users. Generally, users are classified into several categories according to their device numbers, mobile phone numbers, and the like, which can be repeated. For example, common types of crowd bags include: e-commerce crowd bags, student crowd bags, beauty crowd bags and the like.
In accordance with an embodiment of the present application, there is provided an embodiment of a method for monitoring the progress of execution of a data processing task, it should be noted that the steps illustrated in the flowchart of the drawings may be performed in a computer system such as a set of computer-executable instructions, and that while a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different than here.
Fig. 1 is a flowchart of a method for monitoring the execution progress of a data processing task according to an embodiment of the present application, and as shown in fig. 1, the method includes the following steps:
step S102, acquiring an identifier of a data processing task, and searching data to be processed corresponding to the data processing task according to the identifier;
according to an optional embodiment of the application, the data processing task is a data insight task, the data insight is to convert data into information through data analysis/mining, and factor and action link influencing a service result are combed out by combining a service scene, so that the problem is correctly attributed and an improved direction is obtained.
Step S104, determining the data volume of the data to be processed and the current available computing resources of the equipment executing the data processing task;
when a task is created, a task ID (namely a task identifier) is generated in the database, and after the monitoring module scans a new task ID, the data size of the uploaded crowd packet is calculated under the corresponding storage path. At the same time, the currently available computing resources of the device performing the data processing task are computed.
And step S106, monitoring the execution progress of the data processing task according to the data volume of the data to be processed and the currently available computing resources.
Through the steps, the execution progress of the data processing task is monitored through the data volume of the data to be processed and the current available computing resources of the equipment executing the data processing task, so that the execution progress of the data processing task can be dynamically monitored in real time, and the control degree of important tasks and the technical effect of the use experience of customers are greatly improved.
According to an alternative embodiment of the present application, step S106 is implemented by: determining a first time length required for executing a data processing task under the current available computing resource according to the data volume of the data to be processed and the current available computing resource; and comparing the first time length with a first preset time length, and if the first time length exceeds the first preset time length, reducing the data volume of the data to be processed or increasing the available computing resources of the equipment executing the data processing task.
In the step, the predicted completion time of the data processing task under the currently available computing resources is calculated through an algorithm formula and related personnel are notified. The predicted completion time is then compared to a first predetermined duration (which is predetermined based on business requirements), and if the predicted completion time exceeds the first predetermined duration, either an increase in computational resources or a decrease in data size input may be considered.
The predicted completion time of the data processing task can be predicted by the method, and whether the data processing task can be completed within the specified time or not is judged.
According to another alternative embodiment of the present application, step S106 can also be implemented by: acquiring a second time length required by the data processing task to complete the first target progress; comparing the second time length with a second preset time length, wherein the second preset time length is the estimated time length required by the data processing task to complete the first target progress; and if the second time length is not matched with the second preset time length, generating an alarm signal.
The first target schedule is an execution schedule of any of the data processing tasks. After the task starts to be executed, whether the execution progress of the data processing task (the execution progress of the task can be obtained through the API) is matched with the estimated time or not is continuously monitored, if the execution progress reaches the set progress node, an alarm notification is given (if the task progress is just completed by 30%, but the consumed time reaches 50% of the estimated time).
It should be noted that the mismatch between the second duration and the second preset duration may be that the second duration is not equal to the second preset duration, or that a difference between the second duration and the second preset duration exceeds a preset range.
The method can realize real-time dynamic monitoring of the execution progress of the data processing task.
In some optional embodiments of the present application, the obtaining of the second time length required for the data processing task to complete the first target progress includes at least one of: acquiring a second time length required by the data processing task to complete the first target progress according to a preset progress interval of the data processing task execution progress; and acquiring a second time length required by the data processing task to complete the first target progress according to a preset time interval.
In this step, the second duration required for the data processing task to complete the first target progress may be obtained according to the preset progress interval, for example, the time required for completing 30% of progress, the time required for completing 50% of progress, and the time required for completing 70% of progress (i.e., the preset progress interval is 20%).
The second time length required by the data processing task to complete the first target progress may also be obtained at preset time intervals, for example, the second time length required by the data processing task to complete the first target progress is obtained every 10 seconds.
In other alternative embodiments of the present application, the currently available computing resources of the device performing the data processing task are determined by the following method when step S104 is performed: and determining a cluster calculation force value of the equipment according to the queue idle resource value, the cluster whole idle resource value of the equipment and a resource value to be released by the equipment within a preset time period, wherein the cluster calculation force value is used for representing the current available calculation resources of the equipment, and the queue is used for processing the data processing task.
In this step, the current available computing resources of the device are determined by calculating the cluster computation value of the device executing the data processing task, specifically, the cluster computation value of the device is the queue idle resource value + the cluster whole idle resource value of the device + the queue overuse percentage + the resource value to be released by the task to be completed.
It should be noted that the cluster computation value includes the remaining computation resources of the device CPU and memory, the queues are used to execute the data processing tasks, and one queue may execute one or more tasks.
By the method, the residual computing resources of the equipment executing the data processing task can be accurately calculated.
According to an optional embodiment of the application, after the cluster calculation value of the equipment is calculated, the predicted completion time of the task under the current resource condition is obtained according to the current calculation value of the cluster, the crowd data amount, the query dimension complex coefficient and the experience time coefficient.
According to an optional embodiment of the present application, if the available computing resources are increased, determining a third duration required for completing the second target progress when the remaining progress of the data processing task is executed according to the current completion progress of the data processing task and the newly increased available computing resources; comparing the third time length with a third preset time length, wherein the third preset time length is the estimated time length required by the data processing task to complete the second target progress; and if the third time length is not matched with the third preset time length, generating an alarm signal.
It should be noted that the second target progress is any one of the remaining progresses of the data processing task, for example, the progress of 30% has been completed, and the progress of 50% and the progress of 70% are both the second target progress of the remaining progresses of the data processing task, and are referred to as the second target progress herein, mainly for distinguishing from the first target progress in the foregoing.
In the process that the equipment executes the data processing task, if the cluster idle resources of the equipment are expanded (the computing resources are released after other tasks are completed), the speed of the equipment executing the data processing task is increased due to the increase of the cluster idle resources of the equipment, and at the moment, when the residual progress of the data processing task is recalculated according to the current task progress and the newly added computing resources, the third time required by the second target progress is completed. Then comparing the third time length with a third preset time length, and if the task can be completed within a preset time, no alarm is given; if not, alarming according to the latest progress matching value.
By the method, when the computing resources of the equipment executing the data processing task are expanded, the automatic adjustment of the task execution progress monitoring threshold can be realized.
The overall idea of the method provided by the application is to develop a monitoring module, access the monitoring module to the CDP system by using a monitoring universal interface, monitor the progress of the data insight task and the condition of the computing cluster resource in real time, and realize the dynamic monitoring of the progress of the data insight task by an intelligent algorithm and a dynamic monitoring logic. The method fills the blank of task completion progress prediction and dynamic alarm in the CDP system, allows users and operators to have a clear and relatively accurate concept on task completion time, can acquire the progress condition of the task in real time, timely receives alarm after the task is delayed, and allows corresponding responsible persons to make judgment and measures in advance and inform the responsible persons of the task to users. The control degree of important tasks and the use experience of users are greatly improved.
Fig. 2 is a block diagram of a device for monitoring the execution progress of a data processing task according to an embodiment of the present application, and as shown in fig. 2, the device includes:
the acquiring module 20 is configured to acquire an identifier of the data processing task, and search for to-be-processed data corresponding to the data processing task according to the identifier;
a determining module 22 for determining a data amount of the data to be processed and a currently available computing resource of the device performing the data processing task;
and the monitoring module 24 is configured to monitor the execution progress of the data processing task according to the data amount of the data to be processed and the currently available computing resource.
According to an alternative embodiment of the present application, the monitoring module 24 comprises: the determining unit is used for determining a first time length required for executing a data processing task under the current available computing resource according to the data volume of the data to be processed and the current available computing resource; and the comparison unit is used for comparing the first time length with a first preset time length, and reducing the data volume of the data to be processed or increasing the available computing resources of the equipment executing the data processing task if the first time length exceeds the first preset time length.
It should be noted that, reference may be made to the description related to the embodiment shown in fig. 1 for a preferred implementation of the embodiment shown in fig. 2, and details are not described here again.
The embodiment of the application also provides a nonvolatile storage medium, wherein the nonvolatile storage medium comprises a stored program, and the method for monitoring the execution progress of the data processing task is executed by the equipment where the nonvolatile storage medium is controlled during program operation.
The nonvolatile storage medium stores a program for executing the following functions: acquiring an identifier of the data processing task, and searching data to be processed corresponding to the data processing task according to the identifier; determining a data amount of data to be processed and a currently available computing resource of a device performing a data processing task; and monitoring the execution progress of the data processing task according to the data volume of the data to be processed and the currently available computing resources.
The embodiment of the application also provides a processor, wherein the processor is used for running the program stored in the memory, and the program running is used for executing the above method for monitoring the execution progress of the data processing task.
The processor is used for running a program for executing the following functions: acquiring an identifier of the data processing task, and searching data to be processed corresponding to the data processing task according to the identifier; determining a data amount of data to be processed and a currently available computing resource of a device performing a data processing task; and monitoring the execution progress of the data processing task according to the data volume of the data to be processed and the currently available computing resources.
The above-mentioned serial numbers of the embodiments of the present application are merely for description and do not represent the merits of the embodiments.
In the above embodiments of the present application, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In the embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units may be a logical division, and in actual implementation, there may be another division, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.
The foregoing is only a preferred embodiment of the present application and it should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present application, and these improvements and modifications should also be considered as the protection scope of the present application.
Claims (10)
1. A method for monitoring the execution progress of a data processing task is characterized by comprising the following steps:
acquiring an identifier of a data processing task, and searching data to be processed corresponding to the data processing task according to the identifier;
determining a data amount of the data to be processed and a currently available computing resource of a device executing the data processing task;
and monitoring the execution progress of the data processing task according to the data volume of the data to be processed and the current available computing resource.
2. The method of claim 1, wherein monitoring the progress of the data processing task in accordance with the amount of data of the data to be processed and the currently available computing resources comprises:
determining a first time length required for executing the data processing task under the current available computing resource according to the data volume of the data to be processed and the current available computing resource;
and comparing the first time length with a first preset time length, and if the first time length exceeds the first preset time length, reducing the data volume of the data to be processed or increasing available computing resources of equipment executing the data processing task.
3. The method of claim 1, wherein monitoring the progress of the data processing task in accordance with the amount of data of the data to be processed and the currently available computing resources further comprises:
acquiring a second time length required by the data processing task when the data processing task completes the first target progress;
comparing the second time length with a second preset time length, wherein the second preset time length is estimated to be the time length required by the data processing task to complete the first target progress;
and if the second time length is not matched with the second preset time length, generating an alarm signal.
4. The method of claim 3, wherein obtaining the second duration of time required for the data processing task to complete the first target schedule comprises at least one of:
acquiring a second time length required by the data processing task when the data processing task completes the first target progress according to a preset progress interval of the data processing task execution progress;
and acquiring a second time length required by the data processing task to finish the first target progress according to a preset time interval.
5. The method of claim 1, wherein determining currently available computing resources of a device performing the data processing task comprises:
and determining a cluster calculation force value of the equipment according to the queue idle resource value, the cluster whole idle resource value of the equipment and a resource value to be released by the equipment within a preset time period, wherein the cluster calculation force value is used for representing the current available computing resources of the equipment, and the queue is used for processing the data processing task.
6. The method of claim 1, further comprising:
if the available computing resources are increased, determining a third time length required for completing a second target progress when the remaining progress of the data processing task is executed according to the current completion progress of the data processing task and the newly increased available computing resources;
comparing the third time length with a third preset time length, wherein the third preset time length is estimated to be the time length required by the data processing task to complete the second target progress;
and if the third time length is not matched with the third preset time length, generating an alarm signal.
7. A device for monitoring progress of execution of a data processing task, comprising:
the acquisition module is used for acquiring an identifier of a data processing task and searching data to be processed corresponding to the data processing task according to the identifier;
a determining module for determining a data amount of the data to be processed and a currently available computing resource of a device executing the data processing task;
and the monitoring module is used for monitoring the execution progress of the data processing task according to the data volume of the data to be processed and the current available computing resource.
8. The apparatus of claim 7, wherein the monitoring module comprises:
a determining unit, configured to determine, according to the data size of the to-be-processed data and the currently available computing resource, a first time length required for executing the data processing task under the currently available computing resource;
and the comparison unit is used for comparing the first time length with a first preset time length, and if the first time length exceeds the first preset time length, reducing the data volume of the data to be processed or increasing the available computing resources of the equipment executing the data processing task.
9. A non-volatile storage medium, comprising a stored program, wherein when the program runs, a device in which the non-volatile storage medium is located is controlled to execute the method for monitoring the execution progress of the data processing task according to any one of claims 1 to 6.
10. A processor for running a program stored in a memory, wherein the program when running performs the method of monitoring the progress of the execution of a data processing task according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110587044.9A CN113204692A (en) | 2021-05-27 | 2021-05-27 | Method and device for monitoring execution progress of data processing task |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110587044.9A CN113204692A (en) | 2021-05-27 | 2021-05-27 | Method and device for monitoring execution progress of data processing task |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113204692A true CN113204692A (en) | 2021-08-03 |
Family
ID=77023268
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110587044.9A Pending CN113204692A (en) | 2021-05-27 | 2021-05-27 | Method and device for monitoring execution progress of data processing task |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113204692A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113919877A (en) * | 2021-10-15 | 2022-01-11 | 深圳市酷开网络科技股份有限公司 | Method and device for processing human-circled task progress based on DMP platform and readable storage medium |
| CN114070895A (en) * | 2021-11-15 | 2022-02-18 | 中国联合网络通信集团有限公司 | Data transmission method, control plane network element and user plane network element |
| CN115086317A (en) * | 2022-06-13 | 2022-09-20 | 国网北京市电力公司 | Cable monitoring method and device, nonvolatile storage medium and electronic equipment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1513141A (en) * | 2001-06-05 | 2004-07-14 | �ʼҷ����ֵ�������˾ | Method ands system for assessing progress of task |
| CN110262878A (en) * | 2019-05-06 | 2019-09-20 | 平安科技（深圳）有限公司 | Timed task processing method, device, equipment and computer readable storage medium |
| CN111338791A (en) * | 2020-02-12 | 2020-06-26 | 平安科技（深圳）有限公司 | Method, device and equipment for scheduling cluster queue resources and storage medium |
| CN111813523A (en) * | 2020-07-09 | 2020-10-23 | 北京奇艺世纪科技有限公司 | Duration pre-estimation model generation method, system resource scheduling method, device, electronic equipment and storage medium |
-
2021
- 2021-05-27 CN CN202110587044.9A patent/CN113204692A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1513141A (en) * | 2001-06-05 | 2004-07-14 | �ʼҷ����ֵ�������˾ | Method ands system for assessing progress of task |
| CN110262878A (en) * | 2019-05-06 | 2019-09-20 | 平安科技（深圳）有限公司 | Timed task processing method, device, equipment and computer readable storage medium |
| CN111338791A (en) * | 2020-02-12 | 2020-06-26 | 平安科技（深圳）有限公司 | Method, device and equipment for scheduling cluster queue resources and storage medium |
| CN111813523A (en) * | 2020-07-09 | 2020-10-23 | 北京奇艺世纪科技有限公司 | Duration pre-estimation model generation method, system resource scheduling method, device, electronic equipment and storage medium |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113919877A (en) * | 2021-10-15 | 2022-01-11 | 深圳市酷开网络科技股份有限公司 | Method and device for processing human-circled task progress based on DMP platform and readable storage medium |
| CN114070895A (en) * | 2021-11-15 | 2022-02-18 | 中国联合网络通信集团有限公司 | Data transmission method, control plane network element and user plane network element |
| CN114070895B (en) * | 2021-11-15 | 2023-04-25 | 中国联合网络通信集团有限公司 | Data transmission method, control plane network element and user plane network element |
| CN115086317A (en) * | 2022-06-13 | 2022-09-20 | 国网北京市电力公司 | Cable monitoring method and device, nonvolatile storage medium and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US6973415B1 (en) | System and method for monitoring and modeling system performance | |
| CN111212038B (en) | Open data API gateway system based on big data artificial intelligence | |
| CN113204692A (en) | Method and device for monitoring execution progress of data processing task | |
| CN111768008A (en) | Federal learning method, device, equipment and storage medium | |
| US7778715B2 (en) | Methods and systems for a prediction model | |
| JP6355683B2 (en) | Risk early warning method, apparatus, storage medium, and computer program | |
| US11966319B2 (en) | Identifying anomalies in a data center using composite metrics and/or machine learning | |
| CN105243252B (en) | A kind of method and device of account risk assessment | |
| CN107704387B (en) | Method, device, electronic equipment and computer readable medium for system early warning | |
| CN112187512B (en) | Port automatic expansion method, device and equipment based on flow monitoring | |
| US7369967B1 (en) | System and method for monitoring and modeling system performance | |
| US20180268258A1 (en) | Automated decision making using staged machine learning | |
| CN108696486B (en) | Abnormal operation behavior detection processing method and device | |
| CN113516244A (en) | Intelligent operation and maintenance method and device, electronic equipment and storage medium | |
| CN113312244A (en) | Fault monitoring method, equipment, program product and storage medium | |
| CN115309913A (en) | Deep learning-based financial data risk identification method and system | |
| CN113704018A (en) | Application operation and maintenance data processing method and device, computer equipment and storage medium | |
| CN113723692A (en) | Data processing method, apparatus, device, medium, and program product | |
| CN114138601A (en) | Service alarm method, device, equipment and storage medium | |
| Hani et al. | Support vector regression for service level agreement violation prediction | |
| CN112163154A (en) | Data processing method, device, equipment and storage medium | |
| CN115827232A (en) | Method, device, system and equipment for determining configuration for service model | |
| CN107124314B (en) | data monitoring method and device | |
| AU2021218217A1 (en) | Systems and methods for preventative monitoring using AI learning of outcomes and responses from previous experience. | |
| CN112860956A (en) | Multivariate environmental data analysis method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |