CN110809029A - Management method for receiving multiple operation and maintenance index data based on memory queue - Google Patents
Management method for receiving multiple operation and maintenance index data based on memory queue Download PDFInfo
- Publication number
- CN110809029A CN110809029A CN201910987702.6A CN201910987702A CN110809029A CN 110809029 A CN110809029 A CN 110809029A CN 201910987702 A CN201910987702 A CN 201910987702A CN 110809029 A CN110809029 A CN 110809029A
- Authority
- CN
- China
- Prior art keywords
- data
- receiving
- maintenance
- index data
- index
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Computer And Data Communications (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a management method for receiving multiple operation and maintenance index data based on a memory queue, which provides three data receiving modes, including TCP receiving, HTTP receiving and Kafka receiving. And uses a concurrentlinkedque queue to secure threads when executing concurrently. The management method can be applied to an intelligent operation and maintenance platform in an electric power system, developers can select a data communication mode only by following an operation and maintenance index data format formulated by the method, code development in the communication aspect is not needed, received data can be effectively managed, and a dream database is used for data storage, Redis is used for caching of data query, and opensdb is used for real-time display of front-end data.
Description
Technical Field
The application belongs to the field of power grid dispatching automation, and particularly relates to a management method for receiving multiple operation and maintenance index data.
Background
In the process of building an intelligent operation and maintenance platform of the power system, the acquisition end sends the acquired operation and maintenance data to the server end in various forms, for example, the operation and maintenance index data is sent through Kafka or HTTP; how to receive multiple sending modes in a unified way by a server is a problem to be solved. The open source intelligent operation and maintenance monitoring system used in the industry at present is zabbix, open-falcon, promemeus and the like.
The zabbix system is an enterprise-level open source solution providing distributed system monitoring and network monitoring functions based on a WEB interface. zabbix can monitor various network parameters to ensure the safe operation of the server system; and provides a flexible notification mechanism to allow system administrators to quickly set aside various existing problems.
zabbix consisted of 2 parts, zabbix server and an optional component zabbix agent.
The zabbix server can provide functions of monitoring the state of a remote server/network, collecting data and the like by SNMP, zabbix agent, ping, port monitoring and the like, and can be operated on platforms such as Linux, Solaris, HP-UX, AIX, FreeBSD, open BSD, OSX and the like.
Advantages of the Zabbix system include: (1) the most common monitoring mode of zabbix is that the monitoring index is deep and detailed and has pertinence. (2) The built-in monitoring function is powerful, and built-in monitoring items are rich. (3) The TCP mode realizes communication, and the reliability is also guaranteed.
However, Zabbix, although providing a powerful monitoring function, requires installing agent clients one by one on the monitored machine, is cumbersome to deploy, and does not conform to the complex field environment of the power system. Moreover, the communication mode is a TCP mode, the agent and server communication modes are not configurable and must conform to the mode provided by zabbix, and the flexibility is not good; moreover, zibbix is easy to enter the door, but the difficulty of secondary development is high; the operation and maintenance system is heavy.
Therefore, a new technical solution is needed to solve the above problems.
Disclosure of Invention
The purpose of the invention is as follows: the invention provides an HBase-based smart grid acquisition and monitoring data storage system and a storage method, and aims to reasonably store information and fully utilize resources to improve the loading and access performance of acquisition and monitoring data.
The technical scheme is as follows: in order to achieve the purpose, the intelligent power grid acquisition monitoring data storage system based on HBase adopts the following technical scheme:
a management method for receiving multiple operation and maintenance index data based on a memory queue comprises the following steps:
step 1), combing operation and maintenance index data according to a formulated data format for effective data receiving, data storage and data display;
step 2), providing three data receiving communication modes: the method comprises the steps of TCP communication, HTTP communication and Kafka communication modes, and is used for receiving operation and maintenance index data;
and step 3), integrating various databases for storing data, including a relational database, a time sequence database and a memory cache database, and effectively storing the data.
Further, in step 1), the data source includes but infrastructure index, operation index, business system index, third party's index; the message is in a json format, and the data type supports String, int and Float; the data format comprises host names, component names, affiliated systems, index groups, index names, index values and reporting periods.
Further, in the step 2), spring is embedded into Netty, and a TCP and HTTP communication mode is integrated through Netty, so that a high-performance and high-reliability network application framework is provided; the spring boot is embedded into the Kafka, and a configurable mechanism for communicating to the Kafka is provided; a convurrentlinkedque queue is used to ensure thread security when executing concurrently.
Further, in the step 3), three data storage modes are integrated through spring, a relational database is integrated, and all metadata and index data are stored; integrating Redis, storing the latest data into Redis, and providing a health diagnosis interface for the outside; integrating the opentsdb to provide data support for the rich presentation of the front-end data.
Has the advantages that: compared with the prior art, the invention has the beneficial effects that:
the operation and maintenance index data can be effectively managed according to the data access specification of the method, component names, affiliated systems, index groups and the like are defined in the specification, and various operation and maintenance data received by various data sources are sorted according to the specification and then stored in a database, so that the data is more organized, and the subsequent data display and application are facilitated; the three data communication modes provided comprise TCP communication, HTTP communication and Kafka communication, cover the general communication mode of the power system, can be seamlessly butted with each scheduling system of the power system, can collect and process operation and maintenance index data without intrusiveness, and developers can put more energy on data service application without considering the realization of codes too much.
Drawings
FIG. 1 is a system architecture diagram of the present invention.
Detailed Description
The following describes the management method for receiving multiple operation and maintenance index data based on the memory queue in further detail with reference to the accompanying drawings.
The following is the data access specification of the present invention.
The message adopts json format, index group-index item data format.
{
"endpoint":"dev1",
"system":"NariCloud",
"component":"eth1",
"metricgroup":"IPInterface",
"metric":"AdminStatus",
"datatype":"String/int/Float",
"value":1,
"step":60,
"counterType":"GAUGE/COUNTER",
"timestamp":1524724608,
}
endpoint, this one is typically the host hostname;
component: the generalized components of the device may be a network card and a CPU which exist physically, and may be middleware components such as a DB;
system: systems to which devices belong, e.g. south Rayleigh clouds
metricgroup: index group, induction of indexes,// unified management of operation and maintenance platform
metric is the unified management of index item,// operation and maintenance platform
value is the value of the index;
step, reporting time period;
the COUNTER type only supports GAUGE and COUNTER according to the reported type, so that the GAUGE is uniformly used;
timestamp, unit second, reported by the index;
fig. 1 shows a system architecture diagram of the present invention. Data sources should include, but are not limited to, infrastructure metrics, operational metrics, business system metrics, third party metrics; the communication mode supports Kafka, HTTP (POST) and TCP modes; and storing the data after accessing to a relational database, Redis and OpenTsdb, and sending an alarm.
In addition, the present invention has many specific implementations and ways, and the above description is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be construed as the protection scope of the present invention. All the components not specified in the present embodiment can be realized by the prior art.
Claims (5)
1. A management method for receiving multiple operation and maintenance index data based on a memory queue is characterized by comprising the following steps:
step 1), combing operation and maintenance index data according to a formulated data format for effective data receiving, data storage and data display;
step 2), providing three data receiving communication modes: the method comprises the steps of TCP communication, HTTP communication and Kafka communication modes, and is used for receiving operation and maintenance index data;
and step 3), integrating various databases for storing data, including a relational database, a time sequence database and a memory cache database, and effectively storing the data.
2. The method for managing the reception of multiple operation and maintenance index data based on the memory queue according to claim 1, wherein: in the step 1), the data source comprises infrastructure indexes, operation indexes, service system indexes and third-party indexes; the message is in a json format, and the data type supports String, int and Float; the data format comprises host names, component names, affiliated systems, index groups, index names, index values and reporting periods.
3. The method for managing the reception of multiple operation and maintenance index data based on the memory queue according to claim 1, wherein: in the step 2), spring is embedded into Netty, and a TCP and HTTP communication mode is integrated through Netty, so that a high-performance and high-reliability network application framework is provided; the spring boot is embedded in Kafka, providing a configurable mechanism for communicating to Kafka.
4. The method for managing the reception of the multiple kinds of operation and maintenance index data based on the memory queue as claimed in claim 3, wherein: in step 2), a ConcurrentLinkedQueue queue is used to ensure the thread security during concurrent execution.
5. The method for managing the reception of multiple operation and maintenance index data based on the memory queue according to claim 1, wherein: in the step 3), three data storage modes are integrated through spring, a relational database is integrated, and all metadata and index data are stored; integrating Redis, storing the latest data into Redis, and providing a health diagnosis interface for the outside; integrating the opentsdb to provide data support for the rich presentation of the front-end data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910987702.6A CN110809029A (en) | 2019-10-17 | 2019-10-17 | Management method for receiving multiple operation and maintenance index data based on memory queue |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910987702.6A CN110809029A (en) | 2019-10-17 | 2019-10-17 | Management method for receiving multiple operation and maintenance index data based on memory queue |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110809029A true CN110809029A (en) | 2020-02-18 |
Family
ID=69488770
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910987702.6A Pending CN110809029A (en) | 2019-10-17 | 2019-10-17 | Management method for receiving multiple operation and maintenance index data based on memory queue |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110809029A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111597087A (en) * | 2020-05-15 | 2020-08-28 | 山东汇贸电子口岸有限公司 | Method for manufacturing cloud monitoring index |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105515192A (en) * | 2015-12-23 | 2016-04-20 | 国网山东省电力公司电力科学研究院 | Monitoring and early warning system and method for power transmission and transformation equipment load data access electric power system |
| CN106452881A (en) * | 2016-10-21 | 2017-02-22 | 用友网络科技股份有限公司 | Operation and maintenance data processing system and method based on cloud + terminal mode |
| CN107239380A (en) * | 2017-05-10 | 2017-10-10 | 杭州铭师堂教育科技发展有限公司 | A kind of message queue monitoring method based on zabbix |
| CN107846468A (en) * | 2017-11-16 | 2018-03-27 | 北京卫星信息工程研究所 | Car networking application system and its control method based on cloud computing technology |
| CN108270860A (en) * | 2018-01-16 | 2018-07-10 | 宇星科技发展(深圳)有限公司 | The acquisition system and method for environmental quality online monitoring data |
| CN108363777A (en) * | 2018-02-09 | 2018-08-03 | 浙江省地理信息中心 | Isomery geographic information public service platform operation/maintenance data junction method based on Agent |
| CN109034521A (en) * | 2018-06-07 | 2018-12-18 | 国电南瑞科技股份有限公司 | A kind of intelligent O&M architecture design method of dispatching of power netwoks control system |
| CN109408240A (en) * | 2018-10-31 | 2019-03-01 | 四川长虹电器股份有限公司 | Terminal device data collection system based on Netty frame |
| CN110210705A (en) * | 2019-04-29 | 2019-09-06 | 德邦物流股份有限公司 | A kind of data analysing method and system |
| US20190306121A1 (en) * | 2018-04-03 | 2019-10-03 | Bank Of America Corporation | System for Managing Communication Ports Between Servers |
-
2019
- 2019-10-17 CN CN201910987702.6A patent/CN110809029A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105515192A (en) * | 2015-12-23 | 2016-04-20 | 国网山东省电力公司电力科学研究院 | Monitoring and early warning system and method for power transmission and transformation equipment load data access electric power system |
| CN106452881A (en) * | 2016-10-21 | 2017-02-22 | 用友网络科技股份有限公司 | Operation and maintenance data processing system and method based on cloud + terminal mode |
| CN107239380A (en) * | 2017-05-10 | 2017-10-10 | 杭州铭师堂教育科技发展有限公司 | A kind of message queue monitoring method based on zabbix |
| CN107846468A (en) * | 2017-11-16 | 2018-03-27 | 北京卫星信息工程研究所 | Car networking application system and its control method based on cloud computing technology |
| CN108270860A (en) * | 2018-01-16 | 2018-07-10 | 宇星科技发展(深圳)有限公司 | The acquisition system and method for environmental quality online monitoring data |
| CN108363777A (en) * | 2018-02-09 | 2018-08-03 | 浙江省地理信息中心 | Isomery geographic information public service platform operation/maintenance data junction method based on Agent |
| US20190306121A1 (en) * | 2018-04-03 | 2019-10-03 | Bank Of America Corporation | System for Managing Communication Ports Between Servers |
| CN109034521A (en) * | 2018-06-07 | 2018-12-18 | 国电南瑞科技股份有限公司 | A kind of intelligent O&M architecture design method of dispatching of power netwoks control system |
| CN109408240A (en) * | 2018-10-31 | 2019-03-01 | 四川长虹电器股份有限公司 | Terminal device data collection system based on Netty frame |
| CN110210705A (en) * | 2019-04-29 | 2019-09-06 | 德邦物流股份有限公司 | A kind of data analysing method and system |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111597087A (en) * | 2020-05-15 | 2020-08-28 | 山东汇贸电子口岸有限公司 | Method for manufacturing cloud monitoring index |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109857613B (en) | Automatic operation and maintenance system based on collection cluster | |
| CN107423198B (en) | EAM platform monitoring management method and system | |
| EP2791825B1 (en) | System and method for monitoring and managing data center resources in real time incorporating manageability subsystem | |
| CN110908879A (en) | Method, device, terminal and storage medium for reporting buried point data | |
| EP2324406B1 (en) | Method and apparatus for monitoring the performance of a power delivery control system | |
| US20110296394A1 (en) | Systems and methods for generating cached representations of encoded package profile | |
| CN103414579A (en) | Cross-platform monitoring system applicable to cloud computing and monitoring method thereof | |
| CN112162821B (en) | Container cluster resource monitoring method, device and system | |
| CN105429791A (en) | Distributed service state detection device and method | |
| US20120191665A1 (en) | Integrated Distribution Management System Channel Adapter | |
| CN109597837B (en) | Time sequence data storage method, time sequence data query method and related equipment | |
| CN105490864A (en) | Business module monitoring method based on OSGI | |
| CN114090366A (en) | Method, device and system for monitoring data | |
| Cejka et al. | Java embedded storage for time series and meta data in Smart Grids | |
| CN115280741A (en) | System and method for autonomous monitoring and recovery in hybrid energy management | |
| CN110636116B (en) | Multidimensional data acquisition system and method | |
| Weng et al. | Kmon: An in-kernel transparent monitoring system for microservice systems with ebpf | |
| CN110809029A (en) | Management method for receiving multiple operation and maintenance index data based on memory queue | |
| CN113570347A (en) | RPA operation and maintenance method for micro-service architecture system | |
| WO2015184850A1 (en) | Method, apparatus and system for collecting stale data | |
| US9189362B2 (en) | Method for measuring the performance of a target server housing a dynamic monitoring tool | |
| US20060053021A1 (en) | Method for monitoring and managing an information system | |
| CN117194156A (en) | Unified monitoring operation and maintenance management method and system for multi-cloud platform | |
| CN106201193A (en) | A kind of method and apparatus showing application message in the terminal | |
| US20230125154A1 (en) | Intelligent upgrade to a debug load operation for an electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200218 |