CN116166813A - Management method, system, equipment and storage medium for big data automation operation and maintenance - Google Patents
Management method, system, equipment and storage medium for big data automation operation and maintenance Download PDFInfo
- Publication number
- CN116166813A CN116166813A CN202211618788.3A CN202211618788A CN116166813A CN 116166813 A CN116166813 A CN 116166813A CN 202211618788 A CN202211618788 A CN 202211618788A CN 116166813 A CN116166813 A CN 116166813A
- Authority
- CN
- China
- Prior art keywords
- data
- enterprise
- information
- management
- global
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007726 management method Methods 0.000 title claims abstract description 97
- 238000012423 maintenance Methods 0.000 title claims abstract description 53
- 230000002159 abnormal effect Effects 0.000 claims abstract description 45
- 238000013079 data visualisation Methods 0.000 claims abstract description 44
- 238000012545 processing Methods 0.000 claims abstract description 11
- 238000005516 engineering process Methods 0.000 claims abstract description 10
- 238000000034 method Methods 0.000 claims description 29
- 238000004590 computer program Methods 0.000 claims description 17
- 238000001514 detection method Methods 0.000 claims description 7
- 238000010219 correlation analysis Methods 0.000 claims description 6
- 230000007246 mechanism Effects 0.000 claims description 5
- 238000013507 mapping Methods 0.000 claims description 3
- 230000006870 function Effects 0.000 description 8
- 230000000694 effects Effects 0.000 description 6
- 230000005856 abnormality Effects 0.000 description 5
- 238000011161 development Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 4
- 230000001360 synchronised effect Effects 0.000 description 4
- 238000013523 data management Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013499 data model Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/36—Creation of semantic tools, e.g. ontology or thesauri
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/3331—Query processing
- G06F16/334—Query execution
- G06F16/3344—Query execution using natural language analysis
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The application discloses a management method, a system, equipment and a storage medium for big data automation operation and maintenance, wherein the management method for big data automation operation and maintenance comprises the following steps: acquiring enterprise operation data based on cluster indexes; synchronizing and calculating enterprise operation data to obtain enterprise data information; storing the enterprise data information to a database by adopting a distributed storage technology; and carrying out standardization processing on the enterprise data information stored in the database to obtain global data information, and generating a global data visualization map based on the global data information so that the management and control end calls the global data visualization map to carry out cluster management on enterprise operation data. The enterprise operation data is subjected to cluster management through the global data visualization map, so that the operation and maintenance efficiency and accuracy are improved, the dynamic and automatic integrated management of the enterprise operation data is realized, the complexity of the data operation and maintenance management is reduced, and the abnormal data is quickly found and positioned.
Description
Technical Field
The present disclosure relates to the field of information management, and in particular, to a method, system, device, and storage medium for managing automated operation and maintenance of big data.
Background
With the development of computers and big data, each big enterprise has entered a digitalized operation stage from informatization, and the technical system of the enterprise also covers relational databases, reports and the like.
However, the enterprise data are scattered in different systems and databases, so that the data are difficult to retrieve and find for enterprise management staff, business staff, data development staff and operation and maintenance staff, the value of the data is not fully reflected, the enterprise is difficult to manage and find problems from the data, and the phenomena of data island and repeated data storage and development are formed.
Therefore, the above technical problems are to be solved.
Disclosure of Invention
The embodiment of the application provides a management method, a system, equipment and a storage medium for large data automation operation and maintenance, which are used for solving or partially solving the problems that data of enterprises are scattered in different systems and databases, and the data are difficult to retrieve and find for enterprise management personnel, business personnel, data development personnel and operation and maintenance personnel, so that the value of the data is not fully reflected, the data are difficult to manage and find problems from the enterprises, and the phenomena of data island and repeated data storage and development are formed.
A management method for big data automation operation and maintenance comprises the following steps:
acquiring enterprise operation data based on cluster indexes;
synchronizing and calculating enterprise operation data to obtain enterprise data information;
storing the enterprise data information into a database by adopting a distributed storage technology so that a management and control end uniformly manages and controls the enterprise data information;
and carrying out standardization processing on the enterprise data information stored in the database to obtain global data information, and generating a global data visualization map based on the global data information so that the management and control end calls the global data visualization map to carry out cluster management on enterprise operation data.
The present application may be further configured in a preferred example to: before generating the global data visualization map based on the global data information, further comprising:
mapping enterprise operation data into a database table by adopting TDengine data modeling;
the database table is converted into a MapReduce task through an SQL engine and is used for generating a global data visualization map.
The present application may be further configured in a preferred example to: based on the cluster index, obtaining enterprise operation data includes:
and performing anomaly detection and correlation analysis on the enterprise operation source data to obtain enterprise operation data.
The present application may be further configured in a preferred example to: after synchronizing and calculating the enterprise operation data to obtain enterprise data information, the method further comprises the following steps:
acquiring a check point, and determining abnormal information of enterprise operation data based on the check point and the enterprise data information;
based on a preset alarm mechanism and abnormal information, an alarm prompt carrying the abnormal information is sent to the management and control end, so that the management and control end can check the abnormal information.
The present application may be further configured in a preferred example to: after generating the global data visualization map based on the global data information, further comprising:
and sending the global data visualization map to a resource panel so that the management and control end lays out the global data visualization map.
The present application may be further configured in a preferred example to: before sending the alarm prompt carrying the abnormal information to the management and control end, the method further comprises the following steps:
counting the number of alarms and obtaining an alarm number threshold;
comparing the alarm times with an alarm times threshold value to obtain an alarm times comparison result;
based on the alarm times comparison result, whether to suspend sending alarm prompt carrying abnormal information to the control end is determined.
The present application may be further configured in a preferred example to: after determining the anomaly information for the enterprise operational data based on the checkpoint and the enterprise data information, further comprising:
acquiring historical alarm data;
based on the historical alarm data, the abnormal information is analyzed, the abnormal type of the abnormal information is determined, and the abnormal type is used for sending alarm prompts of different types to the management and control end based on the abnormal type.
The second purpose of the application is to provide a management system for big data automation operation and maintenance.
The second object of the present application is achieved by the following technical solutions:
a management system for automated operation and maintenance of big data, comprising:
the enterprise operation data acquisition module is used for acquiring enterprise operation data based on cluster indexes;
the enterprise data information obtaining module is used for synchronizing and calculating enterprise operation data to obtain enterprise data information;
the enterprise data information storage module is used for storing enterprise data information to a database by adopting a distributed storage technology so that the management and control end can uniformly manage and control the enterprise data information;
and the global data visualization map generation module is used for carrying out standardized processing on enterprise data information stored in the database to obtain global data information, and generating a global data visualization map based on the global data information so that the management and control end calls the global data visualization map to carry out cluster management on enterprise operation data.
The third object of the present application is to provide an electronic device.
The third object of the present application is achieved by the following technical solutions:
an electronic device comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein the processor realizes the management method of the big data automation operation and maintenance when executing the computer program.
A computer readable storage medium storing a computer program which when executed by a processor implements the above-described method of managing big data automation operation and maintenance.
In summary, the present application includes the following beneficial technical effects:
according to the management method, the system, the equipment and the storage medium for the big data automation operation and maintenance, the enterprise operation data is acquired according to the cluster index, then the enterprise operation data is synchronized and calculated to obtain enterprise data information, and the enterprise data information is stored in the database by adopting the distributed storage technology, so that the management and control end performs unified management and control on the enterprise data information; and performing standardization processing on the enterprise data information stored in the database to obtain global data information, and generating a global data visualization map according to the global data information, so that the management and control end calls the global data visualization map to perform cluster management on enterprise operation data. According to the method, the enterprise operation data is subjected to cluster management through the global data visualization map, so that the operation and maintenance efficiency and accuracy are improved, the dynamic and automatic comprehensive management of the enterprise operation data is realized, the complexity of the data operation and maintenance management is reduced, and the abnormal data is quickly found and positioned.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following description of the drawings that are needed for the description of the embodiments of the present application will be given in brief, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart illustrating a method for managing big data automation operation and maintenance according to an embodiment of the present application;
FIG. 2 is a flowchart illustrating a method for managing big data automation operation and maintenance according to a first embodiment of the present disclosure;
FIG. 3 is a schematic diagram of a management system for automated operation and maintenance of big data according to an embodiment of the present application;
fig. 4 is a schematic diagram of an electronic device according to an embodiment of the application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.
Embodiments of the present application are described in further detail below with reference to the drawings attached hereto.
The embodiment of the application provides a management method for big data automation operation and maintenance, and the main flow of the method is described as follows:
referring to fig. 1, S10, enterprise operation data is acquired based on a cluster index.
The cluster indexes comprise an operation monitoring index, a component performance index, a system index and the like.
Specifically, according to cluster indexes set by the system, such as an operation monitoring index, a component performance index, a system index and the like, data corresponding to the indexes are collected, and then enterprise operation data are obtained.
The step S10 serves to improve the authenticity and reliability of the enterprise operation data.
And S20, synchronizing and calculating the enterprise operation data to obtain enterprise data information.
The enterprise data information comprises component performance data, system operation data, job process data and the like.
Specifically, the enterprise operation data is divided into blocks according to PK (Primary Key constraint), and then divided into a plurality of tasks for parallel reading, namely, parallel reading is realized in a full-scale stage. The full quantity and the increment can be automatically switched, and the switching of the lockless consistency is performed through a lockless algorithm during switching. After switching to the increment stage, the independent task is used for analyzing enterprise operation data of the increment part, so that the full increment integrated reading is realized. After entering the increment stage, the job can be modified and released by the control end by no longer needing resources, so that enterprise data information is obtained.
The step S20 serves to improve the efficiency of data operation and maintenance management.
S30, storing the enterprise data information into a database by adopting a distributed storage technology so that the management and control end can uniformly manage and control the enterprise data information.
The distributed storage technology is adopted to use the disk space on each machine in the enterprise through a network, the scattered storage resources form virtual storage equipment, enterprise data information is stored in a database of the virtual storage equipment in a scattered mode, and therefore the management and control end is enabled to conduct unified management and control on the enterprise data information.
The step S30 has the effects of reducing the repeated storage condition of the data by uniformly controlling the enterprise data information, improving the working efficiency of data operation and maintenance management and improving the data utilization rate of the enterprise data information.
And S40, carrying out standardization processing on the enterprise data information stored in the database to obtain global data information, and generating a global data visualization map based on the global data information so that the management and control end calls the global data visualization map to carry out cluster management on enterprise operation data.
The standardized processing refers to unified data standard and index caliber. Clustering refers to providing a group of network resources for users as a whole, wherein the individual computer systems are nodes of the cluster, and a management end in cluster management can randomly add and delete the nodes of the cluster system.
Specifically, the enterprise data information stored in the database is unified with the data standard and the index caliber, automation and online of the enterprise data information are achieved, the BI data visualization tool is adopted to generate a global data visualization map, and then the management and control end calls the global data visualization map to conduct cluster management on enterprise operation data.
The step S40 has the effects of realizing centralized management of enterprise data, improving the efficiency of data operation and maintenance management and reducing the complexity of enterprise operation data management.
According to the management method, the system, the equipment and the storage medium for the big data automation operation and maintenance, the enterprise operation data is acquired according to the cluster index, then the enterprise operation data is synchronized and calculated to obtain enterprise data information, and the enterprise data information is stored in the database by adopting the distributed storage technology, so that the management and control end performs unified management and control on the enterprise data information; and performing standardization processing on the enterprise data information stored in the database to obtain global data information, and generating a global data visualization map according to the global data information, so that the management and control end calls the global data visualization map to perform cluster management on enterprise operation data. According to the method, the enterprise operation data is subjected to cluster management through the global data visualization map, so that the operation and maintenance efficiency and accuracy are improved, the dynamic and automatic comprehensive management of the enterprise operation data is realized, the complexity of the data operation and maintenance management is reduced, and the abnormal data is quickly found and positioned.
In some possible embodiments, step S10, that is, based on the cluster index, acquires enterprise operation data, includes:
s101, performing anomaly detection and correlation analysis on enterprise operation source data to obtain enterprise operation data.
The correlation analysis refers to analyzing two or more variable elements with correlation, so as to measure the correlation degree of two variable factors. There is a certain association or probability between elements of the correlation to be able to perform the correlation analysis.
The abnormality detection firstly defines a group of enterprise operation data such as CPU utilization rate, memory utilization rate, file verification and the like when the system is in a normal condition, and then analyzes the enterprise operation data to further determine whether abnormality occurs.
The step S101 serves to improve the security and reliability of enterprise operation data management.
In some possible embodiments, after step S20, that is, after synchronizing and calculating the enterprise operation data, the method further includes:
s21, acquiring a check point, and determining abnormal information of enterprise operation data based on the check point and the enterprise data information.
S22, based on a preset alarm mechanism and abnormal information, an alarm prompt carrying the abnormal information is sent to the control end, so that the control end can check the abnormal information.
The check point is a check node arranged on the system according to the requirement of the control end. The preset alarm mechanism is set by the control end in the system in advance, and the control end can be configured in a self-defined way for a period of time without processing the abnormal information, so that the alarm information can be automatically pushed to the control end with a higher level.
The step S21 and the step S22 have the effect of improving the reliability and timeliness of enterprise operation data detection.
In some possible embodiments, after step S21, that is, after determining the anomaly information of the enterprise operation data based on the checkpoints and the enterprise data information, further includes:
s23, acquiring historical alarm data.
S24, analyzing the abnormal information based on the historical alarm data, and determining the abnormal type of the abnormal information, wherein the abnormal type is used for sending alarm prompts of different types to the management and control end based on the abnormal type.
Wherein the exception types include bursty exceptions and regular exceptions.
Specifically, historical alarm data stored in a system are acquired, abnormal information is compared with the historical alarm data for analysis, whether the abnormal information belongs to sudden abnormality or regular abnormality is determined, alarm prompts corresponding to different abnormality types preset in the system are sent to a management and control end, real-time alarm statistics and display functions are provided, and the abnormal information is presented in a time line mode.
The alarm prompt comprises a short message, a mailbox, a WeChat, a nail and other alarm pushing modes.
The step S23 and the step S24 have the function of improving the timeliness and the reliability of sending the alarm prompt.
In some possible embodiments, before step S22, that is, before sending the alert prompt carrying the abnormal information to the management and control end, the method further includes:
s25, counting the number of alarms and acquiring an alarm number threshold.
S26, comparing the alarm times with alarm times threshold values to obtain alarm times comparison results.
And S27, determining whether to suspend sending the alarm prompt carrying the abnormal information to the management and control end based on the alarm times comparison result.
The alarm frequency threshold is data information preset in advance and stored in the system.
Specifically, counting the number of alarms in a certain time, such as the number of alarms in five minutes, comparing the number of alarms in the time with a preset alarm number threshold value of a system to obtain an alarm number comparison result, and if the number of alarms is smaller than or equal to the alarm number threshold value, suspending sending an alarm prompt carrying abnormal information to a management and control terminal; if the alarm times are greater than the alarm times threshold, continuing to send alarm prompts carrying abnormal information to the control end until the control end processes the abnormal information and pauses sending the alarm prompts.
The steps S25 to S27 have the functions of reducing the repeatability of sending the alarm prompt and improving the efficiency and reliability of data operation and maintenance management.
In some possible embodiments, before step S40, that is, before generating the global data visualization map based on the global data information, the method further includes:
s41, mapping enterprise operation data into a database table by adopting TDengine data modeling.
S42, converting the database table into a MapReduce task through an SQL engine for generating a global data visualization map.
The TDengine data modeling is a data model which adopts one acquisition point and one table. MapReduce is a programming model for parallel operation of large-scale datasets that achieves reliability by distributing large-scale operations on the datasets to each node on the network.
Specifically, a single-column super table is created by adopting TDengine data modeling, acquisition index IDs are added in tag items, and acquisition indexes of different data types in enterprise operation data are classified into different super tables, for example, numerical values and character strings are respectively classified into two types of corresponding super tables, so that a database table is finally obtained. Then, defining SQL grammar rules by utilizing Antlr, completing SQL lexicon, converting SQL into an abstract grammar Tree AST Tree, traversing the AST Tree, abstracting a basic composition unit QueryBlock of the query, traversing the QueryBlock, translating into an execution operation Tree Operator Tree, performing Operator Tree transformation by a logic layer optimizer, reducing MapReduce job, reducing the buffer data volume, traversing the Operator Tree, translating into a MapReduce task, and finally performing MapReduce task transformation by a physical layer optimizer, generating a final execution plan, and further generating a global data visualization map.
The QueryBlock is a most basic component unit of SQL and comprises three parts, namely an input source, a calculation process and an output, and simply a QueryBlock is a sub-query.
In some possible embodiments, after step S40, that is, after generating the global data visualization map based on the global data information, further includes:
s43, the global data visualization map is sent to the resource panel, so that the management and control end lays out the global data visualization map.
Specifically, the management and control end can automatically layout according to the requirements by sending the global data visualization map to the built-in resource panel, and the multi-dimensional multi-view self-defining resource panel.
The step S43 has the effects that through the operation of the full-flow visual interface, the use and learning cost is greatly reduced by arranging the dragging workflow components, the enterprises can conveniently open the closed loops of the data, the platform and the application, and the efficiency, the convenience, the sharing and the reliability of the data operation and maintenance management are improved.
According to the management method for the big data automation operation and maintenance, as shown in fig. 2, the safety and reliability of enterprise operation data management are improved by performing anomaly detection and correlation analysis on enterprise operation source data; the alarm mechanism and the abnormal information are preset, and an alarm prompt carrying the abnormal information is sent to the management and control end, so that the reliability and timeliness of enterprise operation data detection are improved; comparing the alarm times with alarm times threshold preset by the system, determining whether to suspend sending alarm prompts carrying abnormal information to the management and control end, reducing the repeatability of sending the alarm prompts, and improving the efficiency and reliability of data operation and maintenance management; through the visual interface operation of the whole process, the drag workflow component arrangement greatly reduces the use and learning cost, is convenient for enterprises to open closed loops of data, platforms and applications, and improves the efficiency, convenience, sharing and reliability of data operation and maintenance management.
In another embodiment of the application, a management system for automated operation and maintenance of big data is disclosed.
Referring to fig. 3, the management system of the big data automation operation and maintenance includes:
the enterprise operation data acquisition module 10 is configured to acquire enterprise operation data based on the cluster index.
The enterprise data information module 20 is configured to synchronize and calculate enterprise operation data to obtain enterprise data information.
The enterprise data information storage module 30 is configured to store enterprise data information to a database by using a distributed storage technology, so that a management and control end performs unified management and control on the enterprise data information.
The global data visualization map generation module 40 is configured to perform standardization processing on the enterprise data information stored in the database to obtain global data information, and generate a global data visualization map based on the global data information, so that the management and control end invokes the global data visualization map to perform cluster management on the enterprise operation data.
The big data automatic operation and maintenance management system provided in this embodiment can achieve the same technical effects as the foregoing embodiments due to the functions of each module and the logic connection between each module, and the principle analysis can see the relevant description of the steps of the big data automatic operation and maintenance management method, which is not repeated here.
For specific limitations regarding the management system of the big data automation operation and maintenance, reference may be made to the above limitation of the management method of the big data automation operation and maintenance, and details thereof will not be repeated here. The modules in the management system for big data automation operation and maintenance can be realized in whole or in part by software, hardware and combination thereof. The above modules may be embedded in hardware or independent of a processor in the device, or may be stored in software in a memory in the device, so that the processor may call and execute operations corresponding to the above modules.
In an embodiment, an electronic device is provided, which may be a monitoring terminal, and an internal structure diagram thereof may be as shown in fig. 4. The electronic device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the electronic device is configured to provide computing and control capabilities. The memory of the electronic device includes a non-volatile medium, an internal memory. The non-volatile medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile media. The database of the electronic equipment is used for storing data to be stored in the management method of the big data automation operation and maintenance. The network interface of the electronic device is used for communicating with an external terminal through a network connection. The computer program, when executed by the processor, implements a method for managing automated operation and maintenance of big data.
In an embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, where the processor implements the method for managing big data automation operation and maintenance according to the above embodiment when executing the computer program, for example, step S10 to step S40 shown in fig. 1. Alternatively, the processor, when executing the computer program, implements the functions of the modules/units of the management system for automated operation and maintenance of big data in the above embodiment, such as the functions of the modules 10 to 40 shown in fig. 3. To avoid repetition, no further description is provided here.
In an embodiment, a computer readable storage medium is provided, where a computer program is stored on the computer readable storage medium, where the computer program when executed by a processor implements the method for managing big data automation operation and maintenance in the foregoing embodiment, or where the computer program when executed by a processor implements functions of each module/unit in the big data automation operation and maintenance management system in the foregoing system embodiment. To avoid repetition, no further description is provided here.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable medium that when executed comprises the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the system is divided into different functional units or modules to perform all or part of the above-described functions.
The above embodiments are only for illustrating the technical solution of the present application, and are not limiting thereof; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application, and are intended to be included in the scope of the present application.
Claims (10)
1. The management method for the automatic operation and maintenance of the big data is characterized by comprising the following steps of:
acquiring enterprise operation data based on cluster indexes;
synchronizing and calculating the enterprise operation data to obtain enterprise data information;
storing the enterprise data information into a database by adopting a distributed storage technology so that a management and control end uniformly manages and controls the enterprise data information;
and carrying out standardization processing on the enterprise data information stored in the database to obtain global data information, and generating a global data visualization map based on the global data information so that the management and control end calls the global data visualization map to carry out cluster management on the enterprise operation data.
2. A method of managing automated big data operations and maintenance according to claim 1, further comprising, prior to said generating a global data visualization map based on said global data information:
mapping the enterprise operation data into a database table by adopting TDengine data modeling;
and converting the database table into a MapReduce task through an SQL engine, and generating the global data visualization map.
3. The method for managing automated operation and maintenance of big data according to claim 1, wherein the obtaining the enterprise operation data based on the cluster index comprises:
and performing anomaly detection and correlation analysis on the enterprise operation source data to obtain enterprise operation data.
4. The method for managing big data automation operation and maintenance according to claim 1, further comprising, after synchronizing and calculating the enterprise operation data to obtain enterprise data information:
acquiring a check point, and determining abnormal information of the enterprise operation data based on the check point and the enterprise data information;
based on a preset alarm mechanism and the abnormal information, sending an alarm prompt carrying the abnormal information to the control end so that the control end can check the abnormal information.
5. The method of claim 1, further comprising, after the generating the global data visualization map based on the global data information:
and sending the global data visualization map to a resource panel so that the control end lays out the global data visualization map.
6. The method for managing big data automation operation and maintenance according to claim 4, further comprising, before said sending an alarm prompt carrying said anomaly information to said management and control terminal:
counting the number of alarms and obtaining an alarm number threshold;
comparing the alarm times with the alarm times threshold value to obtain an alarm times comparison result;
and determining whether to suspend sending the alarm prompt carrying the abnormal information to the control terminal based on the alarm times comparison result.
7. The method of claim 4, further comprising, after said determining anomaly information for said enterprise operating data based on said checkpoint and said enterprise data information:
acquiring historical alarm data;
and analyzing the abnormal information based on the historical alarm data, and determining the abnormal type of the abnormal information, wherein the abnormal type is used for sending alarm prompts of different types to the management and control end based on the abnormal type.
8. A management system for automated operation and maintenance of big data, comprising:
the enterprise operation data acquisition module is used for acquiring enterprise operation data based on cluster indexes;
the enterprise data information obtaining module is used for synchronizing and calculating the enterprise operation data to obtain enterprise data information;
the enterprise data information storage module is used for storing the enterprise data information to a database by adopting a distributed storage technology so that a management and control end uniformly manages and controls the enterprise data information;
and the global data visualization map generation module is used for carrying out standardized processing on the enterprise data information stored in the database to obtain global data information, and generating a global data visualization map based on the global data information so that the management and control end calls the global data visualization map to carry out cluster management on the enterprise operation data.
9. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements a method for managing big data automation operations according to any of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the method for managing big data automation operation and maintenance according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211618788.3A CN116166813A (en) | 2022-12-15 | 2022-12-15 | Management method, system, equipment and storage medium for big data automation operation and maintenance |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211618788.3A CN116166813A (en) | 2022-12-15 | 2022-12-15 | Management method, system, equipment and storage medium for big data automation operation and maintenance |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116166813A true CN116166813A (en) | 2023-05-26 |
Family
ID=86415439
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211618788.3A Pending CN116166813A (en) | 2022-12-15 | 2022-12-15 | Management method, system, equipment and storage medium for big data automation operation and maintenance |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116166813A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117591607A (en) * | 2024-01-19 | 2024-02-23 | 杭州青橄榄网络技术有限公司 | Data synchronization management method and system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105119750A (en) * | 2015-09-08 | 2015-12-02 | 南京联成科技发展有限公司 | Distributed information security operation and maintenance management platform based on massive data |
| CN109840268A (en) * | 2018-12-23 | 2019-06-04 | 国网浙江省电力有限公司 | A kind of universe data map construction method based on enterprise information model |
| WO2021184630A1 (en) * | 2020-03-19 | 2021-09-23 | 平安国际智慧城市科技股份有限公司 | Method for locating pollutant discharge object on basis of knowledge graph, and related device |
| CN114386774A (en) * | 2021-12-21 | 2022-04-22 | 中国中煤能源集团有限公司 | CPIM-based three-dimensional visual full-life-cycle management platform for coal preparation plant |
| CN115150589A (en) * | 2022-06-30 | 2022-10-04 | 合肥恒翔电子科技有限公司 | Video monitoring operation and maintenance management system for coal mine enterprise |
-
2022
- 2022-12-15 CN CN202211618788.3A patent/CN116166813A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105119750A (en) * | 2015-09-08 | 2015-12-02 | 南京联成科技发展有限公司 | Distributed information security operation and maintenance management platform based on massive data |
| CN109840268A (en) * | 2018-12-23 | 2019-06-04 | 国网浙江省电力有限公司 | A kind of universe data map construction method based on enterprise information model |
| WO2021184630A1 (en) * | 2020-03-19 | 2021-09-23 | 平安国际智慧城市科技股份有限公司 | Method for locating pollutant discharge object on basis of knowledge graph, and related device |
| CN114386774A (en) * | 2021-12-21 | 2022-04-22 | 中国中煤能源集团有限公司 | CPIM-based three-dimensional visual full-life-cycle management platform for coal preparation plant |
| CN115150589A (en) * | 2022-06-30 | 2022-10-04 | 合肥恒翔电子科技有限公司 | Video monitoring operation and maintenance management system for coal mine enterprise |
Non-Patent Citations (1)
| Title |
|---|
| 阿里云云栖号: "基于 Flink CDC 实现海量数据的实时同步和转换", pages 1 - 27, Retrieved from the Internet <URL:https://segmentfault.com/a/1190000042219864> * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117591607A (en) * | 2024-01-19 | 2024-02-23 | 杭州青橄榄网络技术有限公司 | Data synchronization management method and system |
| CN117591607B (en) * | 2024-01-19 | 2024-05-07 | 杭州青橄榄网络技术有限公司 | Data synchronization management method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107908672B (en) | Application report realization method, device and storage medium based on Hadoop platform | |
| CN107491345B (en) | Method for writing picture data and distributed NewSQ L database system | |
| US20190213115A1 (en) | Utilizing artificial intelligence to test cloud applications | |
| JP2016015124A (en) | Computer device, processing method, and computer program | |
| WO2021083713A1 (en) | Ml-based event handling | |
| CN112527774A (en) | Data center building method and system and storage medium | |
| CN106293891B (en) | Multidimensional investment index monitoring method | |
| US10922640B2 (en) | Smart template for predictive analytics | |
| CN111459698A (en) | Database cluster fault self-healing method and device | |
| AU2021244852B2 (en) | Offloading statistics collection | |
| Vershinin et al. | Performance analysis of PostgreSQL, MySQL, microsoft SQL server systems based on TPC-H tests | |
| CN117149873A (en) | Data lake service platform construction method based on flow batch integration | |
| CN116166813A (en) | Management method, system, equipment and storage medium for big data automation operation and maintenance | |
| US20230130738A1 (en) | System and method for providing sql quality analysis and sql optimization in relational database | |
| Balliu et al. | A big data analyzer for large trace logs | |
| CN114201328A (en) | Fault processing method and device based on artificial intelligence, electronic equipment and medium | |
| US20220027365A1 (en) | Enterprise resource planning system, server and supervision method of sql queries in such a system or server | |
| CN114416849A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN114238463A (en) | Calculation engine control method and device for distributed index calculation | |
| CN113961582A (en) | Data processing method, device, equipment and storage medium | |
| CN116739336A (en) | Power grid disaster early warning method and system based on multi-source heterogeneous data fusion model | |
| CN116384921A (en) | Execution method and device of operation and maintenance event, storage medium and electronic equipment | |
| US10909117B2 (en) | Multiple measurements aggregated at multiple levels of execution of a workload | |
| US20230185817A1 (en) | Multi-model and clustering database system | |
| Wei et al. | Big data analysis service platform building for complex product manufacturing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |