CN113535354A - Method and device for adjusting parallelism of Flink SQL operator - Google Patents
Method and device for adjusting parallelism of Flink SQL operator Download PDFInfo
- Publication number
- CN113535354A CN113535354A CN202110731394.8A CN202110731394A CN113535354A CN 113535354 A CN113535354 A CN 113535354A CN 202110731394 A CN202110731394 A CN 202110731394A CN 113535354 A CN113535354 A CN 113535354A
- Authority
- CN
- China
- Prior art keywords
- task
- parallelism
- flink
- operator
- parameter
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 71
- 238000012545 processing Methods 0.000 claims abstract description 29
- 230000008569 process Effects 0.000 claims abstract description 19
- 230000000007 visual effect Effects 0.000 claims description 10
- 238000001514 detection method Methods 0.000 claims description 6
- 238000012544 monitoring process Methods 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 5
- 230000009286 beneficial effect Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000012800 visualization Methods 0.000 description 3
- 238000009825 accumulation Methods 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24568—Data stream processing; Continuous queries
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method and a device for adjusting the parallelism of Flink SQL operators, relates to the technical field of Internet, and can conveniently and efficiently adjust the parallelism of each operator. The method comprises the following steps: initializing a Flink SQL task and a configured parallelism parameter to each operator in a Flink engine task execution plan; processing the Flink SQL task according to the parallelism parameter, and detecting a backpressure value in the execution process of the Flink SQL task in real time; and carrying out early warning aiming at operators with backpressure values exceeding a threshold value, adjusting the configuration of parallelism parameters and restarting a task execution plan. The device is applied with the method provided by the scheme.
Description
Technical Field
The invention relates to the technical field of internet, in particular to a method and a device for adjusting the parallelism of a Flink SQL operator.
Background
The Apache Flink is an open source computing platform facing distributed data stream processing and batch processing, and can provide functions supporting stream processing and batch processing two types of applications based on the same Flink Runtime (Flink Runtime). The method is more and more favored by developers due to the advantages of supporting high-throughput, low-delay and high-performance stream processing, and supporting exact-once semantics of state-based computing.
With more and more businesses migrating into the Flink (the OCDP platform provides a visual dragging task and a Flink SQL task), the method has higher requirements on the utilization rate of system resources. Because the Flink SQL task is a pure SQL mode, the method is more convenient and faster, more and more developers select the SQL mode to develop the task, the Flink SQL task cannot adjust the parallelism degree aiming at the operator level, and if the parallelism degree of the task level is directly set, the resource waste is caused. Therefore, how to conveniently and efficiently adjust the parallelism of each operator is very necessary in the actual business operation.
Disclosure of Invention
The invention aims to provide a method and a device for adjusting the parallelism of Flink SQL operators, which can conveniently and efficiently adjust the parallelism of each operator.
In order to achieve the above object, a first aspect of the present invention provides a method for adjusting parallelism of a Flink SQL operator, comprising:
initializing a Flink SQL task and a configured parallelism parameter to each operator in a Flink engine task execution plan;
processing the Flink SQL task according to the parallelism parameter, and detecting a backpressure value in the execution process of the Flink SQL task in real time;
and carrying out early warning aiming at operators with backpressure values exceeding a threshold value, adjusting the configuration of parallelism parameters and restarting a task execution plan.
Preferably, before the step of initializing the Flink SQL task and the configured parallelism parameter to each operator in the Flink engine task execution plan, the method further includes:
and before the Flink SQL task is on-line, a developer configures the parallelism parameter of the Flink SQL task through a visual interface.
Preferably, the method for processing the Flink SQL task according to the parallelism parameter and detecting the backpressure value in the execution process of the Flink SQL task in real time comprises the following steps:
and acquiring a back pressure value of the task in the running process in real time according to the monitoring view and the Flink UI in the execution process of the Flink SQL task.
Preferably, the method for performing early warning on the operator with the backpressure value exceeding the threshold, adjusting the configuration of the parallelism parameter, and restarting the task execution plan includes:
comparing the real-time backpressure value of each operator with a corresponding threshold value, and performing early warning when any one or more operators exceed the threshold value;
and reminding a developer to reconfigure the parallelism parameter and restart the task execution plan according to the early warning result.
Further, the method for reminding a developer to reconfigure the parallelism parameter and restart the task execution plan according to the early warning result comprises the following steps:
acquiring a task topology plan from a context example of an execution environment, and analyzing the parallelism parameter of each operator;
readjusting the parallelism logic of the operators based on the parallelism parameters of the operators, and reconfiguring the parallelism parameters;
restarting the task execution plan by the parallelism logic of the readjustment operator and the reconfigured parallelism parameter, and submitting the task execution plan to the Flink engine for execution and operation.
Optionally, the method for reminding the developer to reconfigure the parallelism parameter includes:
and remotely reminding the developer to reconfigure the parallelism parameter by any one or more of APP push, short message, telephone and mail.
Compared with the prior art, the method for adjusting the parallelism of the Flink SQL operator has the following beneficial effects:
the method for adjusting the parallelism of the Flink SQL operator comprises the steps of initializing a Flink SQL task and a configured parallelism parameter to each operator in a Flink engine task execution plan, processing the Flink SQL task by using a Flink engine according to the parallelism parameter, detecting a backpressure value processed by the Flink SQL task in real time in the processing process, visually displaying the backpressure value to a developer, giving an early warning to the operator with the backpressure value exceeding a threshold value, enabling the developer to adjust the configuration of the parallelism parameter in time through a visual UI, saving the adjusted parallelism parameter of the operator, and restarting the task to realize the modification of the parallelism of the operator.
Therefore, the method and the device solve the problem that the parallelism parameter of the operator cannot be adjusted for the Flink SQL task, and improve the capability and efficiency of real-time task processing. In addition, because the developer completes the adjustment on the UI interface, compared with the scheme of operator parallelism adjustment completed by a code writing mode in the prior art, the method has the advantages of quick restart and effectiveness, and enhanced data processing capability of tasks.
A second aspect of the present invention provides a device for adjusting parallelism of a Flink SQL operator, which is applied to the method for adjusting parallelism of a Flink SQL operator in the foregoing technical solution, and the device includes:
the task online unit is used for initializing the Flink SQL task and the configured parallelism parameter to each operator in the task execution plan of the Flink engine;
the operation detection unit is used for processing the Flink SQL task according to the parallelism parameter and detecting a backpressure value in the execution process of the Flink SQL task in real time;
and the processing unit is used for carrying out early warning on the operator with the backpressure value exceeding the threshold value, adjusting the configuration of the parallelism parameter and restarting the task execution plan.
Preferably, the method further comprises the following steps:
and the parameter configuration unit is used for configuring the parallelism parameter of the Flink SQL task by a developer through a visual interface before the Flink SQL task is on line.
Preferably, the operation detection unit is further configured to obtain a backpressure value during task operation in real time according to the monitoring view and the Flink UI during the execution of the Flink SQL task.
Preferably, the processing unit includes:
the threshold comparison module is used for comparing the real-time backpressure value of each operator with a corresponding threshold, and carrying out early warning when any one or more operators exceed the threshold;
and the early warning reminding module is used for reminding a developer to reconfigure the parallelism parameter and restart the task execution plan.
Compared with the prior art, the device for adjusting the parallelism of the Flink SQL operator provided by the invention has the same beneficial effect as the method for adjusting the parallelism of the Flink SQL operator provided by the technical scheme, and the detailed description is omitted here.
A third aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, executes the steps of the method for adjusting the parallelism of the Flink SQL operator.
Compared with the prior art, the beneficial effect of the computer-readable storage medium provided by the invention is the same as that of the method for adjusting the parallelism of the Flink SQL operator provided by the technical scheme, and the description is omitted here.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention and not to limit the invention. In the drawings:
FIG. 1 is a schematic flow chart illustrating a method for adjusting parallelism of a Flink SQ operator according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart illustrating a method for adjusting parallelism of a Flink SQL operator according to an embodiment of the present invention;
fig. 3 is a sequence diagram of an interaction performed by the Flink SQL operator according to an embodiment of the present invention.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in detail below. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example one
Referring to fig. 1-3, the present embodiment provides a method for adjusting parallelism of a Flink SQL operator, including:
initializing a Flink SQL task and a configured parallelism parameter to each operator in a Flink engine task execution plan; processing the Flink SQL task according to the parallelism parameter, and detecting a backpressure value in the execution process of the Flink SQL task in real time; and carrying out early warning aiming at operators with backpressure values exceeding a threshold value, adjusting the configuration of parallelism parameters and restarting a task execution plan.
Considering that the Flink community source code does not provide a scheme for modifying the parallelism of the flank SQL task operator, in order to solve the above problems, the embodiment provides a method for adjusting the parallelism of the Flink SQL operator, first, initializes the Flink SQL task and the configured parallelism parameter to each operator in the execution plan of the Flink engine task, then, the Flink engine is utilized to process the Flink SQL task according to the parallelism parameter, the backpressure value processed by the Flink SQL task needs to be detected in real time in the processing process, meanwhile, the backpressure value is visually displayed to a developer, the task of long-time consumption accumulation is analyzed, the operator with the backpressure value exceeding the threshold value is early warned, and the configuration of the parallelism parameter is timely adjusted by a developer through a visual UI, and then the parallelism parameter of the operator is saved after the adjustment of the parallelism parameter of the operator is finished, and the parallelism of the operator can be modified by restarting a task.
Therefore, the problem that the parallelism parameter of the operator cannot be adjusted for the Flink SQL task is solved, and the capability and the efficiency of real-time task processing are improved. In addition, because the developer completes the adjustment on the UI interface, compared with the scheme of operator parallelism adjustment completed by a code writing mode in the prior art, the scheme provided by the embodiment can be restarted and taken effect quickly, and the capability of processing data by the task is enhanced.
In the above embodiment, before initializing the Flink SQL task and the configured parallelism parameter to each operator in the Flink engine task execution plan in the step, the method further includes:
and before the Flink SQL task is on-line, a developer configures the parallelism parameter of the Flink SQL task through a visual interface. The parallelism parameter is adjusted through the visual UI interface, and compared with a traditional code writing mode, the method has the advantages of being low in operation threshold, high in efficiency, convenient and flexible and the like.
In the above embodiment, the method for processing the Flink SQL task according to the parallelism parameter and detecting the backpressure value in the execution process of the Flink SQL task in real time includes:
and in the execution process of the Flink SQL task, acquiring a back pressure value of the task in operation in real time according to the monitoring view and the Flink UI.
In the above embodiment, the method for performing early warning on an operator with a backpressure value exceeding a threshold, adjusting the configuration of the parallelism parameter, and restarting the task execution plan includes:
comparing the real-time backpressure value of each operator with a corresponding threshold value, and performing early warning when any one or more operators exceed the threshold value; and reminding a developer to reconfigure the parallelism parameter and restart the task execution plan according to the early warning result.
When the method is specifically implemented, the backpressure value shows the resource consumption condition of operators in the task processing process, and according to the early warning rule set by a user, when the backpressure value of one or more operators exceeds a threshold value, early warning can be timely sent to a developer.
In the above embodiment, the method for reminding a developer to reconfigure the parallelism parameter and restart the task execution plan according to the early warning result includes:
acquiring a task topology plan from a context example of an execution environment, and analyzing the parallelism parameter of each operator;
readjusting the parallelism logic of the operators based on the parallelism parameters of the operators, and reconfiguring the parallelism parameters;
restarting the task execution plan by the parallelism logic of the readjustment operator and the reconfigured parallelism parameter, and submitting the task execution plan to the Flink engine for execution and operation.
In the above embodiment, the method for reminding a developer to reconfigure the parallelism parameter includes:
and remotely reminding the developer to reconfigure the parallelism parameter by any one or more of APP push, short message, telephone and mail.
It should be explained that the execution graph in Flink can be divided into four layers: StreamGraph, JobGraph, ExecutionGraph, and physical execution graph.
StreamGraph (task execution plan graph): is an initial graph generated from code written by a developer through the Stream API to represent the topology of the program. The operator parallelism parameter is modified at this layer by the UI in this embodiment.
JobGraph (task optimization graph): the StreamGraph is optimized to generate the JobGraph, which is submitted to the data structure of JobManager. The main optimization is to use a plurality of eligible nodes chain together as a node, which can reduce the serialization/deserialization/transmission consumption required for data flow between nodes.
ExecutionGraph (execution diagram): JobManager generates an ExecutionGraph from JobGraph. The ExecutionGraph is a parallelized version of the JobGraph and is the most core data structure of the scheduling layer.
Physical execution diagram: after the Job is scheduled by the JobManager according to the execution graph, a 'diagram' formed after the Task is deployed on each TaskManager to realize the running Task.
The embodiment obtains the parallelism parameter of each operator in the task StreamGraph from the context of the execution environment (streamexecution environment). And is displayed on the front page through an interface form. Then, analyzing each Operator parallelism parameter Operator Level. After the operator parallelism parameter is obtained, a set of logic capable of modifying the operator parallelism is realized again, then the operator parallelism parameter is reset, and finally the modified operator parallelism logic and the corresponding execution environment are regenerated into a task execution plan (StreamGraph). And submitting the newly generated execution plan to the Flink calculation engine to execute and run. Specifically, the configuration in the parent class is loaded by customizing the subclass of StreamExecutionEnvironment. Acquiring a task execution plan instance, analyzing an operator parallelism set in the execution plan, converting a child object from a context environment instance context of an execution environment, calling a method for realizing the setting of the parallelism setParalleism again, and finally delivering the adjusted task execution plan to be executed by a Flink engine. A developer can check each operator of the task and the parallelism parameter thereof on a front-end page, adjust the parallelism of the operators by combining actual conditions and related indexes and submit the operator parallelism to a background. Rewritten into the execution plan of Flink by the daemon. Namely: the task execution plan is realized again, and the latest adjusted operator parallelism parameter is registered in the task execution plan. When the task is submitted again to run (the task execution plan is restarted), the adjusted Flink SQL task is executed, and therefore the purpose of modifying the parallelism of the operators of the Flink SQL task is achieved.
It can be seen that, the embodiment provides a visualized operation entry to modify the operator parallelism while modifying the StreamGraph parallelism. In addition, the embodiment is also suitable for the visualization task, so that the workload of modifying the code and repackaging the online visualization task is reduced.
In conclusion, the purpose of adjusting the parallelism of the Flink SQL operator is achieved through the scheme of the embodiment, and the problem that a developer cannot modify the parallelism of the Flink SQL task operator originally is solved. Meanwhile, the workload of modifying the parallelism of the FLink visualization task operator by a task developer is greatly reduced, and the problem of the reduction of the consumption capability of the whole task caused by the consumption accumulation of a certain operator can be solved in a very short time.
Example two
The embodiment provides a device for adjusting the parallelism of a Flink SQL operator, which comprises:
the task online unit is used for initializing the Flink SQL task and the configured parallelism parameter to each operator in the task execution plan of the Flink engine;
the operation detection unit is used for processing the Flink SQL task according to the parallelism parameter and detecting a backpressure value in the execution process of the Flink SQL task in real time;
and the processing unit is used for carrying out early warning on the operator with the backpressure value exceeding the threshold value, adjusting the configuration of the parallelism parameter and restarting the task execution plan.
Preferably, the method further comprises the following steps:
and the parameter configuration unit is used for configuring the parallelism parameter of the Flink SQL task by a developer through a visual interface before the Flink SQL task is on line.
Preferably, the operation detection unit is further configured to obtain a backpressure value during task operation in real time according to the monitoring view and the Flink UI during the execution of the Flink SQL task.
Preferably, the processing unit includes:
the threshold comparison module is used for comparing the real-time backpressure value of each operator with a corresponding threshold, and carrying out early warning when any one or more operators exceed the threshold;
and the early warning reminding module is used for reminding a developer to acquire the parallelism parameter of each operator from the context example of the execution environment according to the early warning result, readjusting the parallelism logic of the operator based on the parallelism parameter of each operator, reconfiguring the parallelism parameter, restarting the task execution plan by readjusting the parallelism logic of the operator and the reconfigured parallelism parameter, and submitting the restarted task execution plan to the Flink engine for execution and operation.
Compared with the prior art, the device for adjusting the parallelism of the Flink SQL operator provided by the embodiment of the invention has the same beneficial effect as the method for adjusting the parallelism of the Flink SQL operator provided by the first embodiment, and the detailed description is omitted here.
EXAMPLE III
The present embodiment provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, executes the steps of the method for adjusting the parallelism of the Flink SQL operator.
Compared with the prior art, the beneficial effect of the computer-readable storage medium provided by the embodiment is the same as that of the method for adjusting the parallelism of the Flink SQL operator provided by the above technical scheme, and details are not repeated herein.
It will be understood by those skilled in the art that all or part of the steps in the method for implementing the invention may be implemented by hardware instructions related to a program, the program may be stored in a computer-readable storage medium, and when executed, the program includes the steps of the method of the embodiment, and the storage medium may be: ROM/RAM, magnetic disks, optical disks, memory cards, and the like.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and the changes or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.
Claims (10)
1. A method for adjusting the parallelism of a Flink SQL operator is characterized by comprising the following steps:
initializing a Flink SQL task and a configured parallelism parameter to each operator in a Flink engine task execution plan;
processing the Flink SQL task according to the parallelism parameter, and detecting a backpressure value in the execution process of the Flink SQL task in real time;
and carrying out early warning aiming at operators with backpressure values exceeding a threshold value, adjusting the configuration of parallelism parameters and restarting a task execution plan.
2. The method according to claim 1, wherein before the step of initializing the Flink SQL task and the configured parallelism parameter to each operator in the Flink engine task execution plan, the method further comprises:
and before the Flink SQL task is on-line, a developer configures the parallelism parameter of the Flink SQL task through a visual interface.
3. The method according to claim 1 or 2, wherein the method for processing the Flink SQL task according to the parallelism parameter and detecting the backpressure value in the execution process of the Flink SQL task in real time comprises the following steps:
and acquiring a back pressure value of the task in the running process in real time according to the monitoring view and the Flink UI in the execution process of the Flink SQL task.
4. The method of claim 1, wherein the method of performing early warning for operators with backpressure values exceeding a threshold, adjusting the configuration of parallelism parameters, and restarting a task execution plan comprises:
comparing the real-time backpressure value of each operator with a corresponding threshold value, and performing early warning when any one or more operators exceed the threshold value;
and reminding a developer to reconfigure the parallelism parameter and restart the task execution plan according to the early warning result.
5. The method of claim 4, wherein reminding a developer to reconfigure the parallelism parameter and restart a task execution plan based on an early warning result comprises:
acquiring a task topology plan from a context example of an execution environment, and analyzing the parallelism parameter of each operator;
readjusting the parallelism logic of the operators based on the parallelism parameters of the operators, and reconfiguring the parallelism parameters;
restarting the task execution plan by the parallelism logic of the readjustment operator and the reconfigured parallelism parameter, and submitting the task execution plan to the Flink engine for execution and operation.
6. An apparatus for adjusting parallelism of a Flink SQL operator, comprising:
the task online unit is used for initializing the Flink SQL task and the configured parallelism parameter to each operator in the task execution plan of the Flink engine;
the operation detection unit is used for processing the Flink SQL task according to the parallelism parameter and detecting a backpressure value in the execution process of the Flink SQL task in real time;
and the processing unit is used for carrying out early warning on the operator with the backpressure value exceeding the threshold value, adjusting the configuration of the parallelism parameter and restarting the task execution plan.
7. The apparatus of claim 6, further comprising:
and the parameter configuration unit is used for configuring the parallelism parameter of the Flink SQL task by a developer through a visual interface before the Flink SQL task is on line.
8. The device according to claim 6 or 7, wherein the running detection unit is further configured to obtain a backpressure value during task running in real time according to the monitoring view and the Flink UI during the execution of the Flink SQL task.
9. The apparatus of claim 6, wherein the processing unit comprises:
the threshold comparison module is used for comparing the real-time backpressure value of each operator with a corresponding threshold, and carrying out early warning when any one or more operators exceed the threshold;
and the early warning reminding module is used for reminding a developer to acquire the parallelism parameter of each operator from the context example of the execution environment according to the early warning result, readjusting the parallelism logic of the operator based on the parallelism parameter of each operator, reconfiguring the parallelism parameter, restarting the task execution plan by readjusting the parallelism logic of the operator and the reconfigured parallelism parameter, and submitting the restarted task execution plan to the Flink engine for execution and operation.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method according to any one of the claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110731394.8A CN113535354A (en) | 2021-06-30 | 2021-06-30 | Method and device for adjusting parallelism of Flink SQL operator |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110731394.8A CN113535354A (en) | 2021-06-30 | 2021-06-30 | Method and device for adjusting parallelism of Flink SQL operator |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113535354A true CN113535354A (en) | 2021-10-22 |
Family
ID=78097258
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110731394.8A Pending CN113535354A (en) | 2021-06-30 | 2021-06-30 | Method and device for adjusting parallelism of Flink SQL operator |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113535354A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115760368A (en) * | 2022-11-24 | 2023-03-07 | 中电金信软件有限公司 | Credit business approval method and device and electronic equipment |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130290968A1 (en) * | 2012-04-28 | 2013-10-31 | International Business Machines Corporation | Adjustment of a task execution plan at runtime |
| US20170075693A1 (en) * | 2015-09-16 | 2017-03-16 | Salesforce.Com, Inc. | Handling multiple task sequences in a stream processing framework |
| CN106648904A (en) * | 2017-01-09 | 2017-05-10 | 大连理工大学 | Self-adaptive rate control method for stream data processing |
| US20180287856A1 (en) * | 2017-03-28 | 2018-10-04 | Ca, Inc. | Managing alarms from distributed applications |
| CN109345377A (en) * | 2018-09-28 | 2019-02-15 | 北京九章云极科技有限公司 | A kind of generating date system and Real-time Data Processing Method |
| CN109558232A (en) * | 2018-11-28 | 2019-04-02 | 星环信息科技（上海）有限公司 | Determination method, apparatus, equipment and the medium of degree of parallelism |
| US20190347261A1 (en) * | 2018-05-11 | 2019-11-14 | Qatar Foundation | Apparatus, system, and method for cross-platform data processing |
| CN111143143A (en) * | 2019-12-26 | 2020-05-12 | 北京神州绿盟信息安全科技股份有限公司 | Performance test method and device |
| CN112084016A (en) * | 2020-07-27 | 2020-12-15 | 北京明略软件***有限公司 | Flow calculation performance optimization system and method based on flink |
| CN112084087A (en) * | 2020-08-24 | 2020-12-15 | 上海微亿智造科技有限公司 | Industrial equipment state monitoring and operation and maintenance management method and system |
-
2021
- 2021-06-30 CN CN202110731394.8A patent/CN113535354A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130290968A1 (en) * | 2012-04-28 | 2013-10-31 | International Business Machines Corporation | Adjustment of a task execution plan at runtime |
| US20170075693A1 (en) * | 2015-09-16 | 2017-03-16 | Salesforce.Com, Inc. | Handling multiple task sequences in a stream processing framework |
| CN106648904A (en) * | 2017-01-09 | 2017-05-10 | 大连理工大学 | Self-adaptive rate control method for stream data processing |
| US20180287856A1 (en) * | 2017-03-28 | 2018-10-04 | Ca, Inc. | Managing alarms from distributed applications |
| US20190347261A1 (en) * | 2018-05-11 | 2019-11-14 | Qatar Foundation | Apparatus, system, and method for cross-platform data processing |
| CN109345377A (en) * | 2018-09-28 | 2019-02-15 | 北京九章云极科技有限公司 | A kind of generating date system and Real-time Data Processing Method |
| CN109558232A (en) * | 2018-11-28 | 2019-04-02 | 星环信息科技（上海）有限公司 | Determination method, apparatus, equipment and the medium of degree of parallelism |
| CN111143143A (en) * | 2019-12-26 | 2020-05-12 | 北京神州绿盟信息安全科技股份有限公司 | Performance test method and device |
| CN112084016A (en) * | 2020-07-27 | 2020-12-15 | 北京明略软件***有限公司 | Flow calculation performance optimization system and method based on flink |
| CN112084087A (en) * | 2020-08-24 | 2020-12-15 | 上海微亿智造科技有限公司 | Industrial equipment state monitoring and operation and maintenance management method and system |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115760368A (en) * | 2022-11-24 | 2023-03-07 | 中电金信软件有限公司 | Credit business approval method and device and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107111799B (en) | Job scheduling and monitoring | |
| US20230318952A1 (en) | Method, apparatus and system for real-time optimization of computer-implemented application operations using machine learning techniques | |
| US10534773B2 (en) | Intelligent query parameterization of database workloads | |
| Mattoso et al. | Dynamic steering of HPC scientific workflows: A survey | |
| US20120324454A1 (en) | Control Flow Graph Driven Operating System | |
| US8661441B2 (en) | Transaction load reduction for process completion | |
| US10379920B2 (en) | Processing data to improve a quality of the data | |
| US9098350B2 (en) | Adaptive auto-pipelining for stream processing applications | |
| Song et al. | Reducing energy consumption of smartphones using user-perceived response time analysis | |
| US8671397B2 (en) | Selective data flow analysis of bounded regions of computer software applications | |
| US11126332B2 (en) | Composable events for dynamic user interface composition | |
| Ouyang et al. | Straggler detection in parallel computing systems through dynamic threshold calculation | |
| CN112527474B (en) | Task processing method and device, equipment, readable medium and computer program product | |
| US20120110581A1 (en) | Task cancellation grace periods | |
| US20140258250A1 (en) | Flexible Control Framework Featuring Standalone Rule Engine | |
| CN113535354A (en) | Method and device for adjusting parallelism of Flink SQL operator | |
| CN111782341A (en) | Method and apparatus for managing clusters | |
| US20140372488A1 (en) | Generating database processes from process models | |
| US10592473B2 (en) | Method for improving energy efficiency of map-reduce system and apparatus thereof | |
| WO2013165460A1 (en) | Control flow graph driven operating system | |
| EP3430518A1 (en) | Analysis of recurring processes | |
| CN112363774A (en) | Storm real-time task configuration method and device | |
| JP7204011B2 (en) | Static and runtime analysis of computer program systems | |
| CN117193990B (en) | Scheduling management method, device, equipment and storage medium of http interface | |
| CN112307372B (en) | Data processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |