CN114416326A - Big data control method, device, control system and readable storage medium - Google Patents
Big data control method, device, control system and readable storage medium Download PDFInfo
- Publication number
- CN114416326A CN114416326A CN202011178187.6A CN202011178187A CN114416326A CN 114416326 A CN114416326 A CN 114416326A CN 202011178187 A CN202011178187 A CN 202011178187A CN 114416326 A CN114416326 A CN 114416326A
- Authority
- CN
- China
- Prior art keywords
- service
- running
- determining
- big data
- peak
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a big data management and control method, a big data management and control device, a management and control system and a readable storage medium, wherein the method comprises the following steps: determining a service operation process of each node in a big data cluster; acquiring operation data corresponding to each service operation process within a preset time range; calculating the operation cost corresponding to each service operation process in the preset time range according to the operation data; and generating a process resource allocation strategy according to the operation cost corresponding to each service operation process, and allocating the service operation processes according to the process resource allocation strategy, so that the operation cost corresponding to each service operation process is obtained, and the process resource allocation strategy is generated according to the operation cost, thereby avoiding the control one by one after the occurrence of a data peak period, and being beneficial to the effective operation of a large data cluster and the control of the operation cost.
Description
Technical Field
The invention relates to the technical field of big data, in particular to a big data management and control method, a big data management and control device, a big data management and control system and a readable storage medium.
Background
With the development of big data technology, big data clusters are applied to more and more industries and fields. If the service operation processes of the nodes are concentrated in the same time period to operate or the operation data is overlarge, the operation time of the service operation process is prolonged due to a data peak period, the operation cost corresponding to the service operation process is increased, and the service operation process of the whole big data cluster is broken down in a serious case.
Disclosure of Invention
The invention mainly aims to provide a big data management and control method, a big data management and control device, a management and control system and a readable storage medium, and aims to solve the technical problem of process management and control of a big data cluster based on the running cost corresponding to a business running process.
In order to achieve the above object, the present invention provides a big data management and control method, including the following steps:
determining a service operation process of each node in a big data cluster;
acquiring operation data corresponding to each service operation process within a preset time range;
calculating the operation cost corresponding to each service operation process in the preset time range according to the operation data;
and generating a process resource allocation strategy according to the operation cost corresponding to each service operation process, and allocating the service operation processes according to the process resource allocation strategy.
Optionally, the operation data includes memory consumption, a CPU core count, and an operation time node, and the step of calculating the operation cost corresponding to each service operation process within the preset time range according to the operation data includes:
determining the charging unit price and discount coefficient matched with each service running process according to the running time node, memory consumption and CPU core number of each service running process;
and calculating the operation cost corresponding to each service operation process in the preset time range according to the charging unit price and the discount coefficient.
Optionally, the step of determining the charging unit price and the discount coefficient matched with each service running process according to the running time node, the memory consumption and the number of CPU cores of each service running process includes:
if the operation time node of the service operation process is in the peak operation time period, dividing the service operation process into at least one peak charging time period and/or at least one off-peak charging time according to the peak time node of the peak operation time period and the operation time node of the service operation process;
and determining the target memory consumption and the target operation duration of each charging time period, and determining the charging unit price and the discount coefficient of each charging time period according to the CPU core number, the target memory consumption and the target operation duration.
Optionally, the step of determining the charging unit price of each charging time period according to the number of CPU cores, the target memory consumption amount, and the target running time includes:
determining discount coefficients matched with the charging time periods;
determining the process calculation power of each charging time period according to the target running time, the CPU core number and the target memory consumption of each charging time period;
and determining the charging unit price of each charging time period based on the process calculation power.
Optionally, the step of determining the discount coefficient matched with each charging time period includes:
if the charging time period is the peak charging time period, determining that the discount coefficient of the peak charging time period is a first preset discount coefficient; and
and if the charging time period is the off-peak charging time period, determining that the discount coefficient of the off-peak charging time period is a second preset discount coefficient, wherein the second preset discount coefficient is smaller than the first preset discount coefficient.
Optionally, the step of determining the charging unit price for each charging time period based on the process calculation power includes:
dividing the process calculation power of each charging time period into at least one stage of process calculation power respectively according to a preset step division rule;
and determining the charging unit price matched with the calculation power of each stage.
Optionally, the step of determining the charging unit price and the discount coefficient matched with each service running process according to the running time node, the memory consumption and the number of CPU cores of each service running process further includes:
determining the discount coefficient of each business operation process according to the operation time node of each business operation process;
determining the process calculation power of each business operation process according to the operation time node, the CPU core number and the memory consumption of each business operation process;
and determining the charging unit price of each service operation process based on the process calculation power.
Optionally, the step of determining the discount coefficient of each service running process according to the running time node of each service running process includes:
if the running time node of the business running process is in the peak-peak running time period, determining the discount coefficient of the business running process as a third preset discount coefficient; and
and if the running time node of the business running process is in the off-peak running time period, determining that the discount coefficient of the business running process is a fourth preset discount coefficient, wherein the fourth preset discount coefficient is smaller than the third preset discount coefficient.
Optionally, the step of determining the charging unit price of each service running process based on the process calculation power includes:
dividing the process calculation power of each service operation process into at least one stage of process calculation power respectively according to a preset step division rule;
and determining the charging unit price matched with the calculation power of each stage.
Optionally, the step of generating a process resource allocation policy according to the operation cost corresponding to each service operation process includes:
determining abnormal business operation processes according to the operation cost corresponding to each business operation process;
and generating a process resource allocation strategy corresponding to the abnormal service operation process, wherein the process resource allocation strategy comprises an operation time allocation strategy, an operation capacity allocation strategy and/or an operation speed allocation strategy.
Optionally, after the step of determining the abnormal service running process according to the running cost corresponding to each service running process, the method further includes:
determining a control terminal corresponding to the abnormal service operation process;
and sending an early warning message to the control terminal to inform the control terminal to control the abnormal service running process.
Optionally, after the step of calculating the operation cost corresponding to each service operation process within the preset time range according to the operation data, the method further includes:
and visually outputting the operation cost corresponding to each service operation process.
Further, to achieve the above object, the present invention further provides a big data management and control apparatus, including:
the determining module is used for determining the service running process of each node in the big data cluster;
the acquisition module is used for acquiring operation data corresponding to each service operation process within a preset time range;
the computing module is used for computing the operation cost corresponding to each service operation process in the preset time range according to the operation data;
and the strategy module is used for generating a process resource allocation strategy according to the operation cost corresponding to each service operation process so as to allocate the service operation processes according to the process resource allocation strategy.
Optionally, the calculation module comprises:
the first determining unit is used for determining the charging unit price and the discount coefficient matched with each service running process according to the running time node, the memory consumption and the CPU core number of each service running process;
and the calculating unit is used for calculating the operation cost corresponding to each service operation process in the preset time range according to the charging unit price and the discount coefficient.
Optionally, the first determination unit includes:
a dividing subunit module, configured to divide the service running process into at least one peak charging time period and/or at least one off-peak charging time according to a peak time node of the peak running time period and an operating time node of the service running process, if the operating time node of the service running process is in the peak running time period;
and the determining subunit module is used for determining the target memory consumption and the target operation duration of each charging time period so as to determine the charging unit price and the discount coefficient of each charging time period according to the CPU core number, the target memory consumption and the target operation duration.
Optionally, the policy module comprises:
the first determining unit is used for determining the abnormal service running process according to the running cost corresponding to each service running process;
and the strategy unit is used for generating a process resource allocation strategy corresponding to the abnormal service running process, wherein the process resource allocation strategy comprises a running time allocation strategy, a running capacity allocation strategy and/or a running speed allocation strategy.
Optionally, the policy module further comprises:
the first determining unit is used for determining a control terminal corresponding to the abnormal service running process;
and the early warning unit is used for sending an early warning message to the control terminal so as to inform the control terminal to control the abnormal service running process.
Optionally, the big data management and control apparatus further includes:
and the output module is used for visually outputting the operation cost corresponding to each service operation process.
Further, to achieve the above object, the present invention further provides a monitoring system, where the monitoring system includes a memory, a processor, and a big data management and control program stored in the memory and executable on the processor, and the big data management and control program, when executed by the processor, implements the steps of the big data management and control method as described above.
Further, to achieve the above object, the present invention also provides a readable storage medium, on which a big data management and control program is stored, and when being executed by a processor, the big data management and control program implements the steps of the big data management and control method as described above.
The big data management and control method, the device, the monitoring system and the readable storage medium of the invention determine the service operation process of each node in the big data cluster; acquiring operation data corresponding to each service operation process within a preset time range; calculating the operation cost corresponding to each service operation process in the preset time range according to the operation data; and generating a process resource allocation strategy according to the operation cost corresponding to each service operation process, and allocating the service operation processes according to the process resource allocation strategy, so that the operation cost corresponding to each service operation process is determined by acquiring the operation data corresponding to each service operation process, and the process resource allocation strategy is generated according to the operation cost, thereby avoiding the control one by one after the data peak period, and being beneficial to the effective operation of the big data cluster and the control of the operation cost.
Drawings
FIG. 1 is a schematic diagram of a hardware operating environment according to an embodiment of the monitoring system of the present invention;
FIG. 2 is a flowchart illustrating a big data management and control method according to a first embodiment of the present invention;
FIG. 3 is a flowchart illustrating a big data management and control method according to a second embodiment of the present invention;
FIG. 4 is a flowchart illustrating a big data management and control method according to a third embodiment of the present invention;
fig. 5 is a functional block diagram of a big data management and control apparatus according to a preferred embodiment of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The invention provides a monitoring system, and referring to fig. 1, fig. 1 is a schematic structural diagram of a hardware operating environment according to an embodiment of the monitoring system of the invention.
As shown in fig. 1, the monitoring system may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Wherein a communication bus 1002 is used to enable connective communication between these components. The user interface 1003 may include a Display screen (Display), an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory (e.g., a magnetic disk memory). The memory 1005 may optionally be a memory monitoring system separate from the processor 1001 described above.
Those skilled in the art will appreciate that the hardware configuration of the monitoring system shown in fig. 1 does not constitute a limitation of the monitoring system, and may include more or fewer components than those shown, or some components in combination, or a different arrangement of components.
As shown in fig. 1, a memory 1005, which is a readable storage medium, may include therein an operating system, a network communication module, a user interface module, and a big data handler. The operating system is a program for managing and controlling hardware and software resources of the monitoring system, and supports the operation of a network communication module, a user interface module, a big data management and control program and other programs or software; the network communication module is used to manage and control the network interface 1004; the user interface module is used to manage and control the user interface 1003.
In the hardware structure of the monitoring system shown in fig. 1, the network interface 1004 is mainly used for connecting to a background server and performing data communication with the background server; the user interface 1003 is mainly used for connecting a client (user side) and performing data communication with the client; the processor 1001 may call the big data hypervisor stored in the memory 1005 and perform the following operations:
determining a service operation process of each node in a big data cluster;
acquiring operation data corresponding to each service operation process within a preset time range;
calculating the operation cost corresponding to each service operation process in the preset time range according to the operation data;
and generating a process resource allocation strategy according to the operation cost corresponding to each service operation process, and allocating the service operation processes according to the process resource allocation strategy.
Further, the processor 1001 may call a big data hypervisor stored in the memory 1005, and perform the following operations:
determining the charging unit price and discount coefficient matched with each service running process according to the running time node, memory consumption and CPU core number of each service running process;
and calculating the operation cost corresponding to each service operation process in the preset time range according to the charging unit price and the discount coefficient.
Further, the processor 1001 may call a big data hypervisor stored in the memory 1005, and perform the following operations:
if the operation time node of the service operation process is in the peak operation time period, dividing the service operation process into at least one peak charging time period and/or at least one off-peak charging time according to the peak time node of the peak operation time period and the operation time node of the service operation process;
and determining the target memory consumption and the target operation duration of each charging time period, and determining the charging unit price and the discount coefficient of each charging time period according to the CPU core number, the target memory consumption and the target operation duration.
Further, the processor 1001 may call a big data hypervisor stored in the memory 1005, and perform the following operations:
determining discount coefficients matched with the charging time periods;
determining the process calculation power of each charging time period according to the target running time, the CPU core number and the target memory consumption of each charging time period;
and determining the charging unit price of each charging time period based on the process calculation power.
Further, the processor 1001 may call a big data hypervisor stored in the memory 1005, and perform the following operations:
if the charging time period is the peak charging time period, determining that the discount coefficient of the peak charging time period is a first preset discount coefficient; and
and if the charging time period is the off-peak charging time period, determining that the discount coefficient of the off-peak charging time period is a second preset discount coefficient, wherein the second preset discount coefficient is smaller than the first preset discount coefficient.
Further, the processor 1001 may call a big data hypervisor stored in the memory 1005, and perform the following operations:
dividing the process calculation power of each charging time period into at least one stage of process calculation power respectively according to a preset step division rule;
and determining the charging unit price matched with the calculation power of each stage.
Further, the processor 1001 may call a big data hypervisor stored in the memory 1005, and perform the following operations:
determining the discount coefficient of each business operation process according to the operation time node of each business operation process;
determining the process calculation power of each business operation process according to the operation time node, the CPU core number and the memory consumption of each business operation process;
and determining the charging unit price of each service operation process based on the process calculation power.
Further, the processor 1001 may call a big data hypervisor stored in the memory 1005, and perform the following operations:
if the running time node of the business running process is in the peak-peak running time period, determining the discount coefficient of the business running process as a third preset discount coefficient; and
and if the running time node of the business running process is in the off-peak running time period, determining that the discount coefficient of the business running process is a fourth preset discount coefficient, wherein the fourth preset discount coefficient is smaller than the third preset discount coefficient.
Further, the processor 1001 may call a big data hypervisor stored in the memory 1005, and perform the following operations:
dividing the process calculation power of each service operation process into at least one stage of process calculation power respectively according to a preset step division rule;
and determining the charging unit price matched with the calculation power of each stage.
Further, the processor 1001 may call a big data hypervisor stored in the memory 1005, and perform the following operations:
determining abnormal business operation processes according to the operation cost corresponding to each business operation process;
and generating a process resource allocation strategy corresponding to the abnormal service operation process, wherein the process resource allocation strategy comprises an operation time allocation strategy, an operation capacity allocation strategy and/or an operation speed allocation strategy.
Further, the processor 1001 may call a big data hypervisor stored in the memory 1005, and perform the following operations:
determining a control terminal corresponding to the abnormal service operation process;
and sending an early warning message to the control terminal to inform the control terminal to control the abnormal service running process.
Further, the processor 1001 may call a big data hypervisor stored in the memory 1005, and perform the following operations:
and visually outputting the operation cost corresponding to each service operation process.
The invention also provides a big data management and control method.
Referring to fig. 2, fig. 2 is a flowchart illustrating a big data management and control method according to a first embodiment of the present invention.
While a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown or described herein. Specifically, the big data management and control method of the embodiment includes:
step S10, determining the service running process of each node in the big data cluster;
step S20, acquiring operation data corresponding to each service operation process within a preset time range;
the big data control method in the embodiment is suitable for a big data control system, the big data control system is accessed into a big data cluster and controls the service running process of each node in the big data cluster, and the control content comprises normal control on the node in a normal running state or the service running process and abnormal monitoring on the node in an abnormal state or the service running process. Specifically, a device for providing various service functions is accessed as a node into a big data cluster, a business operation process of each node in the big data cluster is determined, then operation data of each node in the big data cluster is obtained, specifically, when a big data control instruction is received or a big data control condition is reached, operation data corresponding to each business operation process in a preset time range is obtained, wherein the big data control condition includes that a current time node reaches a preset peak time node, or the operation speed of the current business operation process is monitored to be smaller than a preset speed threshold value, and the like, and is not particularly limited, the big data control condition can be set by itself based on the current big data cluster condition, specifically, in the embodiment, the operation data includes memory consumption, a CPU core number and an operation time node, and specifically refers to a start operation time node and an end time node corresponding to the business operation process in the preset time range, it should be noted that, if the service running process starts running before the preset time range, the starting running time node of the service running process in the preset time range is the starting time point of the preset time range, and if the service running process continues running after the preset time range, the ending time node of the service running process in the preset time range is the ending time point of the preset time range, where the memory consumption is the memory amount occupied by the service running data stored when the service running process runs in the preset time range, and the number of CPU cores is the number of CPU cores required to execute the service running process.
Step S30, calculating the operation cost corresponding to each service operation process in the preset time range according to the operation data;
in the step, after obtaining operation data corresponding to each service operation process within a preset time range, calculating an operation cost corresponding to each service operation process within the preset time range according to a preset cost calculation rule, specifically, determining a discount coefficient and a charging unit price corresponding to each service operation process according to the preset cost calculation rule, thereby obtaining an operation cost corresponding to each service operation process, optionally, determining a discount coefficient and a charging unit price corresponding to each service operation process according to a process operation time period corresponding to each service operation process, for example, if a service operation process includes a first service operation process in a peak time period and a second service operation process in an off-peak time period, setting the discount coefficient of the first service operation process as a and the charging unit price as A, setting the discount coefficient of the second service operation process as B and the charging unit price as B, wherein a is greater than B (for example, a is 1.2, B is 0.7), and a is greater than B, it can be understood that, when the service operation process is in the peak time period, the difference of the operation cost between the first service operation process in the peak time period and the second service operation process in the off-peak time period is increased by increasing the discount coefficient and the charging unit price of the service operation process, so as to enhance the representativeness of the operation cost (i.e., the service operation processes corresponding to the peak time period and the off-peak time period can be distinguished by the operation cost), and the like.
Step S40, generating a process resource allocation strategy according to the operation cost corresponding to each service operation process, and allocating the service operation processes according to the process resource allocation strategy.
In this step, after obtaining the operation cost corresponding to each service operation process, optionally, determining whether the operation cost of each service operation process exceeds a cost threshold, if so, determining that the operation time of the service operation process is in a peak period or a peak early warning period, generating a process resource allocation strategy and the like corresponding to the service operation process according to the operation state of the service operation process in the peak period or the peak early warning period, and then allocating the service operation process according to the operation cost, so as to implement shunting management and control of the service operation process, thereby facilitating effective operation of a large data cluster.
Specifically, the step S40 of generating a process resource allocation policy according to the operation cost corresponding to each service operation process, and allocating the service operation processes according to the process resource allocation policy includes:
step S401, determining abnormal business operation processes according to operation costs corresponding to the business operation processes;
step S402, generating a process resource allocation strategy corresponding to the abnormal service running process, wherein the process resource allocation strategy comprises a running time allocation strategy, a running capacity allocation strategy and/or a running speed allocation strategy.
In this step, it should be noted that, since the discount coefficient and the charging unit price of the service running process running in the peak period are both greater than those of the service running process running in the off-peak period, it can be distinguished whether the service running process is running in the peak period by the running cost, specifically, an abnormal service running process is screened out by the running cost, optionally, it is judged whether the running cost corresponding to each service running process or the total service running process of each node exceeds the cost threshold, for example, if the total running cost corresponding to all the service running processes of a node exceeds the cost threshold, all the service running processes corresponding to the node are judged to be abnormal running processes, or if the running cost corresponding to the service running process exceeds the cost threshold, the service running process is taken as an abnormal running process, wherein in the embodiment, the criterion body for determining the abnormal running process is not limited.
Further, after determining the abnormal service running process, generating a process resource allocation strategy corresponding to each abnormal service running process according to the running state of each abnormal service running process, for example, if it is detected that the running speed of the abnormal service running process is slowed down in the later period when storing the service data due to the large memory consumption of the abnormal service running process, so as to cause the running time process and the running cost to be too high, generating a running capacity allocation strategy corresponding to the abnormal service running process, for example, increasing the running capacity corresponding to the abnormal service running process, so as to accelerate the abnormal service running process, or accelerating the running speed of the abnormal service running process, so as to accelerate the abnormal service running process, thereby avoiding the occurrence of the peak period of the big data cluster data, further optionally, determining whether the process of the abnormal service running process can be suspended, if the process of the abnormal business operation process can be suspended, the abnormal business operation process is suspended, and the business operation process is restarted in the off-peak period, so that the operation pressure of the business operation process of the big data cluster is relieved.
The big data management and control method of the invention determines the business operation process of each node in the big data cluster; acquiring operation data corresponding to each service operation process within a preset time range; calculating the operation cost corresponding to each service operation process in the preset time range according to the operation data; and generating a process resource allocation strategy according to the operation cost corresponding to each service operation process, and allocating the service operation processes according to the process resource allocation strategy, so that the operation cost corresponding to each service operation process is determined by acquiring the operation data corresponding to each service operation process, and the process resource allocation strategy is generated according to the operation cost, thereby avoiding the control one by one after the data peak period, and being beneficial to the effective operation of the big data cluster and the control of the operation cost.
Further, based on the first embodiment of the big data management and control method of the present invention, a second embodiment of the big data management and control method of the present invention is provided.
Referring to fig. 3, fig. 3 is a schematic flow chart of a big data management and control method according to a second embodiment of the present invention;
the difference between the second embodiment of the big data management and control method and the first embodiment of the big data management and control method is that the running data includes memory consumption, CPU core count and running time nodes, and the step of calculating the running cost corresponding to each business running process within the preset time range according to the running data includes:
step S301, determining a charging unit price and a discount coefficient matched with each service running process according to the running time node, the memory consumption and the CPU core number of each service running process;
step S302, calculating the operation cost corresponding to each service operation process in the preset time range according to the charging unit price and the discount coefficient.
In the embodiment, the charging unit price and the discount coefficient matched with each service running process are determined according to the running time node, the memory consumption and the CPU core number of each service running process, so that each service running process is charged in a differentiated mode, and running cost charging is reasonable.
Specifically, the step of determining the charging unit price and the discount coefficient matched with each service running process according to the running time node, the memory consumption and the number of CPU cores of each service running process includes:
step S30210, if the operation time node of the service operation process is in the peak operation time period, dividing the service operation process into at least one peak charging time period and/or at least one off-peak charging time according to the peak time node of the peak operation time period and the operation time node of the service operation process;
step S30211, determining a target memory consumption amount and a target operation duration for each charging period, and determining a charging unit price and a discount coefficient for each charging period according to the number of CPU cores, the target memory consumption amount, and the target operation duration.
In this step, if an operation time node of a service operation process is in an on-peak operation time period, an on-peak node of the on-peak operation time period and a off-peak node of an off-peak operation time period adjacent to the on-peak operation time period are determined, the service operation process is divided into at least one on-peak charging time period and/or at least one off-peak charging time period according to the on-peak node, the off-peak node, a start operation time node of the service operation process and an end operation time node of the service operation process, and then a target memory consumption amount and a target operation time period of each charging time period are determined, so that a unit price charging and discount coefficient of each charging time period is determined according to the number of cores of a CPU, the target memory consumption amount and the target operation time period.
Specifically, the step of determining the charging unit price of each charging time period according to the number of CPU cores, the target memory consumption amount, and the target running time includes:
step S30212, determining a discount coefficient matched with each charging time period;
in this step, if the charging time period is the peak charging time period, determining that the discount coefficient of the peak charging time period is a first preset discount coefficient; and
and if the charging time period is the off-peak charging time period, determining that the discount coefficient of the off-peak charging time period is a second preset discount coefficient, wherein the second preset discount coefficient is smaller than the first preset discount coefficient.
Step S30213, determining the process calculation power of each charging time period according to the target running time, the CPU core number and the target memory consumption of each charging time period;
specifically, in this embodiment, the process calculation power is the operation duration + the memory consumption + the number of CPU cores, for example, the unit process calculation power is 1CPU core +2GB memory consumption + 60 seconds of operation duration.
Step S30214, determining the charging unit price for each charging period based on the process calculation power.
In the step, in order to charge each process with different calculation power and realize reasonable running cost charging, a charging rule is adopted:
dividing the process calculation power of each charging time period into at least one stage of process calculation power respectively according to a preset step division rule;
and determining the charging unit price matched with the calculation power of each stage.
For example, the predetermined step division rule is that the charging unit price between 1 and 1000 processes of computational power is 16.66 yuan/computational power, the charging unit price between 1001 and 2000 processes of computational power is 15.25 yuan/computational power, the charging unit price between 2001 and 5000 processes of computational power is 14.00 yuan/computational power, and the charging unit price exceeding 5000 processes of computational power is 13.50 yuan/computational power.
In the embodiment, the charging unit price and the discount coefficient matched with each service running process are determined according to the running time node, the memory consumption and the CPU core number of each service running process, so that the running cost corresponding to each service running process within the preset time range is calculated according to the charging unit price and the discount coefficient of each service running process, the precision rate of the running cost is realized, and the precision rate of the process resource allocation strategy generated according to the running cost is further improved.
Further, based on the first or second embodiment of the big data management and control method of the present invention, a third embodiment of the big data management and control method of the present invention is proposed.
Referring to fig. 4, fig. 4 is a schematic flowchart of a big data management and control method according to a third embodiment of the present invention;
the difference between the third embodiment of the big data management and control method and the first or second embodiment of the big data management and control method is that the step of determining the charging unit price and the discount coefficient matched with each service running process according to the running time node, the memory consumption and the number of CPU cores of each service running process further includes:
s30220, determining a discount coefficient of each business operation process according to the operation time node of each business operation process;
s30221, determining the process calculation power of each business operation process according to the operation time node, the CPU core number and the memory consumption of each business operation process;
and S30222, determining the charging unit price of each service operation process based on the process calculation power.
In this embodiment, another charging rule different from the charging rule of the second embodiment is proposed, specifically:
if the running time node of the business running process is in the peak-peak running time period, determining the discount coefficient of the business running process as a third preset discount coefficient;
and if the running time node of the business running process is in the off-peak running time period, determining that the discount coefficient of the business running process is a fourth preset discount coefficient, wherein the fourth preset discount coefficient is smaller than the third preset discount coefficient.
It should be noted that, if the operation time node of the service operation process is in the peak-peak operation time period, the discount coefficient of the service operation process is set as the discount coefficient corresponding to the peak period, so as to increase the operation cost of the service operation process, and further, the service operation process running in the peak period is configured according to the operation cost, so that the situation that the regulation and control are performed one by one after the data peak period is avoided, and the effective operation of the big data cluster and the management and control of the operation cost are facilitated. .
Further, the step of determining the charging unit price of each service operation process based on the process calculation power includes:
s302221, dividing the process calculation power of each service operation process into at least one stage of process calculation power according to a preset step division rule;
s302222, determining the charging unit price matched with each step of process calculation.
In this step, for example, the predetermined step division rule is that the charging unit price between 1 and 1000 processes of the computational power is 16.66 yuan/computational power, the charging unit price between 1001 and 2000 processes of the computational power is 15.25 yuan/computational power, the charging unit price between 2001 and 5000 processes of the computational power is 14.00 yuan/computational power, and the charging unit price exceeding 5000 processes of the computational power is 13.50 yuan/computational power.
In the embodiment, according to the running time node of each service running process, the discount coefficient of each service running process is determined; determining the process calculation power of each business operation process according to the operation time node, the CPU core number and the memory consumption of each business operation process; and determining the charging unit price of each service running process based on the process calculation power, realizing the precision rate of the running cost, and further improving the precision rate of generating a process resource allocation strategy according to the running cost.
The invention also provides a big data management and control device. Referring to fig. 5, the big data management and control apparatus includes:
the determining module 10 is configured to determine a service running process of each node in the big data cluster;
the acquiring module 20 is configured to acquire operation data corresponding to each service operation process within a preset time range;
the calculating module 30 is configured to calculate, according to the operation data, operation costs corresponding to the operation processes of the services within the preset time range;
and the policy module 40 is configured to generate a process resource allocation policy according to the operation cost corresponding to each service operation process, so as to allocate the service operation processes according to the process resource allocation policy.
Optionally, the calculation module comprises:
the first determining unit is used for determining the charging unit price and the discount coefficient matched with each service running process according to the running time node, the memory consumption and the CPU core number of each service running process;
and the calculating unit is used for calculating the operation cost corresponding to each service operation process in the preset time range according to the charging unit price and the discount coefficient.
Optionally, the first determining unit includes:
a dividing subunit module, configured to divide the service running process into at least one peak charging time period and/or at least one off-peak charging time according to a peak time node of the peak running time period and an operating time node of the service running process, if the operating time node of the service running process is in the peak running time period;
and the determining subunit module is used for determining the target memory consumption and the target operation duration of each charging time period so as to determine the charging unit price and the discount coefficient of each charging time period according to the CPU core number, the target memory consumption and the target operation duration.
Optionally, the policy module comprises:
the first determining unit is used for determining the abnormal service running process according to the running cost corresponding to each service running process;
and the strategy unit is used for generating a process resource allocation strategy corresponding to the abnormal service running process, wherein the process resource allocation strategy comprises a running time allocation strategy, a running capacity allocation strategy and/or a running speed allocation strategy.
Optionally, the policy module further comprises:
the first determining unit is used for determining a control terminal corresponding to the abnormal service running process;
and the early warning unit is used for sending an early warning message to the control terminal so as to inform the control terminal to control the abnormal service running process.
Optionally, the big data management and control apparatus further includes:
and the output module is used for visually outputting the operation cost corresponding to each service operation process.
The specific implementation of the big data management and control apparatus of the present invention is substantially the same as that of the above-mentioned big data management and control method, and is not described herein again.
In addition, the embodiment of the invention also provides a readable storage medium.
The readable storage medium has stored thereon a big data management program, which when executed by the processor implements the steps of the big data management method as described above.
The readable storage medium of the present invention may be a computer readable storage medium, and the specific implementation manner of the readable storage medium is substantially the same as that of each embodiment of the big data management and control method, and will not be described herein again.
The present invention is described in connection with the accompanying drawings, but the present invention is not limited to the above embodiments, which are only illustrative and not restrictive, and those skilled in the art can make various changes without departing from the spirit and scope of the invention as defined by the appended claims, and all changes that come within the meaning and range of equivalency of the specification and drawings that are obvious from the description and the attached claims are intended to be embraced therein.
Claims (10)
1. A big data management and control method is characterized by comprising the following steps:
determining a service operation process of each node in a big data cluster;
acquiring operation data corresponding to each service operation process within a preset time range;
calculating the operation cost corresponding to each service operation process in the preset time range according to the operation data;
and generating a process resource allocation strategy according to the operation cost corresponding to each service operation process, and allocating the service operation processes according to the process resource allocation strategy.
2. The big data management and control method according to claim 1, wherein the operation data includes memory consumption, CPU core count, and operation time nodes, and the step of calculating the operation cost corresponding to each service operation process within the preset time range according to the operation data includes:
determining the charging unit price and discount coefficient matched with each service running process according to the running time node, memory consumption and CPU core number of each service running process;
and calculating the operation cost corresponding to each service operation process in the preset time range according to the charging unit price and the discount coefficient.
3. The big data management and control method according to claim 2, wherein the step of determining the charging unit price and the discount coefficient matched with each service running process according to the running time node, the memory consumption and the number of CPU cores of each service running process comprises:
if the operation time node of the service operation process is in the peak operation time period, dividing the service operation process into at least one peak charging time period and/or at least one off-peak charging time according to the peak time node of the peak operation time period and the operation time node of the service operation process;
and determining the target memory consumption and the target operation duration of each charging time period, and determining the charging unit price and the discount coefficient of each charging time period according to the CPU core number, the target memory consumption and the target operation duration.
4. The big data management and control method according to claim 2, wherein the step of determining the charging unit price and the discount coefficient matched with each service running process according to the running time node, the memory consumption and the number of CPU cores of each service running process further comprises:
determining the discount coefficient of each business operation process according to the operation time node of each business operation process;
determining the process calculation power of each business operation process according to the operation time node, the CPU core number and the memory consumption of each business operation process;
and determining the charging unit price of each service operation process based on the process calculation power.
5. The big data management and control method according to claim 4, wherein the step of determining the discount coefficient of each business operation process according to the operation time node of each business operation process comprises:
if the running time node of the business running process is in the peak-peak running time period, determining the discount coefficient of the business running process as a third preset discount coefficient; and
and if the running time node of the business running process is in the off-peak running time period, determining that the discount coefficient of the business running process is a fourth preset discount coefficient, wherein the fourth preset discount coefficient is smaller than the third preset discount coefficient.
6. The big data management and control method according to claim 1, wherein the step of generating the process resource allocation policy according to the operation cost corresponding to each service operation process comprises:
determining abnormal business operation processes according to the operation cost corresponding to each business operation process;
and generating a process resource allocation strategy corresponding to the abnormal service operation process, wherein the process resource allocation strategy comprises an operation time allocation strategy, an operation capacity allocation strategy and/or an operation speed allocation strategy.
7. The big data management and control method according to claim 6, wherein after the step of determining the abnormal service running process according to the running cost corresponding to each service running process, the method further comprises:
determining a control terminal corresponding to the abnormal service operation process;
and sending an early warning message to the control terminal to inform the control terminal to control the abnormal service running process.
8. The big data management and control device is characterized by comprising:
the determining module is used for determining the service running process of each node in the big data cluster;
the acquisition module is used for acquiring operation data corresponding to each service operation process within a preset time range;
the computing module is used for computing the operation cost corresponding to each service operation process in the preset time range according to the operation data;
and the strategy module is used for generating a process resource allocation strategy according to the operation cost corresponding to each service operation process so as to allocate the service operation processes according to the process resource allocation strategy.
9. A monitoring system comprising a memory, a processor, and a big data hypervisor stored on the memory and executable on the processor, the big data hypervisor when executed by the processor implementing the steps of the big data hypervisor method of any of claims 1-7.
10. A readable storage medium, wherein a big data management and control program is stored on the readable storage medium, and when executed by a processor, the big data management and control program implements the steps of the big data management and control method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011178187.6A CN114416326A (en) | 2020-10-28 | 2020-10-28 | Big data control method, device, control system and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011178187.6A CN114416326A (en) | 2020-10-28 | 2020-10-28 | Big data control method, device, control system and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114416326A true CN114416326A (en) | 2022-04-29 |
Family
ID=81260314
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011178187.6A Pending CN114416326A (en) | 2020-10-28 | 2020-10-28 | Big data control method, device, control system and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114416326A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115271901A (en) * | 2022-07-30 | 2022-11-01 | 苏州浪潮智能科技有限公司 | Artificial intelligence platform based resource charging method and system |
-
2020
- 2020-10-28 CN CN202011178187.6A patent/CN114416326A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115271901A (en) * | 2022-07-30 | 2022-11-01 | 苏州浪潮智能科技有限公司 | Artificial intelligence platform based resource charging method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107832126B (en) | Thread adjusting method and terminal thereof | |
| CN109451051B (en) | Service request processing method and device, electronic equipment and storage medium | |
| CN110196767B (en) | Service resource control method, device, equipment and storage medium | |
| CN110196770B (en) | Cloud system memory data processing method, device, equipment and storage medium | |
| CN109766172B (en) | Asynchronous task scheduling method and device | |
| CN111338791A (en) | Method, device and equipment for scheduling cluster queue resources and storage medium | |
| CN111142788B (en) | Data migration method and device and computer readable storage medium | |
| CN112650575B (en) | Resource scheduling method, device and cloud service system | |
| CN111638953B (en) | Method, device and storage medium for realizing GPU virtualization | |
| CN111190739A (en) | Resource allocation method and device, electronic equipment and storage medium | |
| CN115617497B (en) | Thread processing method, scheduling component, monitoring component, server and storage medium | |
| CN114637603A (en) | Resource control method and device, computer equipment and storage medium | |
| CN113688001A (en) | Dynamic balancing method and device for server hard disk power consumption, terminal and storage medium | |
| CN108415765B (en) | Task scheduling method and device and intelligent terminal | |
| CN111586140A (en) | Data interaction method and server | |
| CN114954105A (en) | Battery replacement method and device, electronic equipment and storage medium | |
| CN114416326A (en) | Big data control method, device, control system and readable storage medium | |
| CN213876703U (en) | Resource pool management system | |
| CN112463361A (en) | Method and equipment for distributing elastic resources of distributed computation | |
| CN109284275B (en) | Cloud platform virtual machine file system monitoring method and device | |
| CN108804152B (en) | Method and device for adjusting configuration parameters | |
| CN111857996B (en) | Interrupt processing method, system, equipment and computer readable storage medium | |
| CN113285890B (en) | Gateway flow distribution method and device, electronic equipment and storage medium | |
| CN115437794A (en) | I/O request scheduling method and device, electronic equipment and storage medium | |
| JP3031278B2 (en) | Automatic resource assignment method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |