CN111752808A - Method for implementing data sharing exchange service operation monitoring system - Google Patents
Method for implementing data sharing exchange service operation monitoring system Download PDFInfo
- Publication number
- CN111752808A CN111752808A CN202010616561.XA CN202010616561A CN111752808A CN 111752808 A CN111752808 A CN 111752808A CN 202010616561 A CN202010616561 A CN 202010616561A CN 111752808 A CN111752808 A CN 111752808A
- Authority
- CN
- China
- Prior art keywords
- service
- data
- monitoring
- log
- platform
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 64
- 238000000034 method Methods 0.000 title claims abstract description 24
- 238000012545 processing Methods 0.000 claims abstract description 25
- 238000012423 maintenance Methods 0.000 claims abstract description 17
- 230000002159 abnormal effect Effects 0.000 claims abstract description 6
- 238000001556 precipitation Methods 0.000 claims abstract description 3
- 238000005516 engineering process Methods 0.000 claims description 14
- 238000007726 management method Methods 0.000 claims description 13
- 238000004458 analytical method Methods 0.000 claims description 8
- 230000008569 process Effects 0.000 claims description 7
- 230000007246 mechanism Effects 0.000 claims description 4
- 230000002085 persistent effect Effects 0.000 claims description 4
- 238000013500 data storage Methods 0.000 claims description 2
- 238000000605 extraction Methods 0.000 claims description 2
- 239000012634 fragment Substances 0.000 claims description 2
- 238000002347 injection Methods 0.000 claims description 2
- 239000007924 injection Substances 0.000 claims description 2
- 230000010354 integration Effects 0.000 claims description 2
- 230000002688 persistence Effects 0.000 claims description 2
- 230000001419 dependent effect Effects 0.000 claims 1
- 238000011161 development Methods 0.000 abstract description 2
- 239000000243 solution Substances 0.000 description 4
- 239000003795 chemical substances by application Substances 0.000 description 3
- 238000007689 inspection Methods 0.000 description 3
- 238000012550 audit Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 241000282326 Felis catus Species 0.000 description 1
- 238000009825 accumulation Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010223 real-time analysis Methods 0.000 description 1
- 239000013049 sediment Substances 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3089—Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents
- G06F11/3093—Configuration details thereof, e.g. installation, enabling, spatial arrangement of the probes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Quality & Reliability (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides a method for realizing a data sharing exchange service operation monitoring system, which belongs to the technical field of Java Web application development. The method mainly comprises monitoring of infrastructure, monitoring of service calling, monitoring of exchange tasks, auditing of user operation, processing of operation and maintenance problems and the like. By operating the monitoring platform, the problems of alarming from abnormal conditions are found in time and are quickly positioned, then the operation and maintenance work order is processed in time, and finally the integrated information system monitoring operation and maintenance scheme of knowledge precipitation is formed.
Description
Technical Field
The invention relates to Java Web application development technology, in particular to an implementation method for a data sharing switching service operation monitoring system.
Background
The data sharing and exchanging platform comprises a plurality of subsystems at present, and mainly comprises a national government affair information sharing website, a government affair information resource directory system, a uniform resource management system, a resource application and auditing system, a service gateway, an exchanging system and the like. Although each subsystem carries different services, the service association is close, and once problems occur, the positioning and the solution are difficult. At present, a data sharing exchange platform is mainly used for receiving user client feedback or manually checking system logs to find problems, so that the processing efficiency is low, the human resource consumption is serious, and the problem positioning and solving speed is far from meeting the requirement.
The traditional monitoring application system mode is as follows: simple analysis and processing of the log on each server by using conventional linux tools (such as cat, tail, sed, awk, grep, etc.) can be basically regarded as command-level operations, and the operation has low cost and high speed, but is difficult to reuse and can only complete basic operations. Since the cost of distributed management becomes higher and higher after the number of servers increases, centralized management is performed by collecting logs on each machine on a specific server using a tool such as rsyslog. The problem that this brings is that the log volume increases sharply, and the need is basically difficult to satisfy by the management of little workshop type. The system has low automation degree and large consumption of human resource cost.
Disclosure of Invention
In order to solve the technical problems, the invention provides a method for realizing a data sharing exchange service operation monitoring system, which solves the overall monitoring of a data sharing exchange platform, can timely find problems in the operation condition of the platform and the operation condition of services on the platform, and timely processes the problems through modes of alarming, work orders and the like, thereby ensuring the stability of the platform operation and the normal operation of the services.
The technical scheme of the invention is as follows:
the implementation method for the data sharing switching service operation monitoring system needs to uniformly monitor the data sharing switching platform in order to ensure the normal operation of each subsystem of the data sharing switching platform.
The operation monitoring system accesses log output by the unified login system, user operation logs output by the subsystems, service call logs of the service gateway, exchange task operation logs generated by the exchange channel and the like to the unified monitoring system through a big data assembly; and accessing other service statistical data, alarm information and the like into the operation management system storage library through an REST interface, an ETL extraction mode and the like.
And by accessing the structured service data and the unstructured log data, the data of the bottom infrastructure and the service operation of the whole data sharing exchange platform are brought into the monitoring system.
After the operation monitoring system collects basic data of the data sharing exchange platform, the next step is to process the collected data. The log information collected by the operation monitoring system is cleaned and filtered through the big data assembly, the log information is processed, then the fragment indexing is carried out, the log data is put into the big data storage, and the retrieval service is provided for the upper layer (the visualization layer). And (3) running statistical data and system running condition data acquired by the monitoring system, analyzing and processing the data through a timing task and the like, and timely performing service alarm and the like according to a service configuration rule of the shared switching platform. And processing the basic data into an alarm library based on the basic data of the service library, and forming a knowledge library by the knowledge precipitation based on the alarm library and in cooperation with operation and maintenance management functions such as work order processing and the like.
The monitoring platform technical architecture is constructed using JAVA EE related technology. According to the given overall architecture design and the Spring Boot technical architecture which is mainstream in the industry at present, the overall technical architecture of the whole monitoring platform is given by combining the characteristics of the monitoring system.
The whole platform architecture is hierarchically divided into three layers, namely: a presentation layer, a business layer, and a persistence layer.
The presentation layer is generally built on freemarkerTemplates technology, and is added with traditional HTML, CSS and JAVASCRIPT technologies to perform interface presentation in a browser.
The service layer is generally constructed based on the spring4.3 technology, dependency injection and AOP technology in the 4.3 technology are introduced, and the expandability of the system is enhanced. On the other hand, the processing of large data volume, such as the processing of data like service call logs, etc. is combined with the current big data processing component EKL to perform storage and index analysis; and a callback mechanism required in the data integration process is realized by using JAVA multithreading and an event monitoring mechanism.
The persistent layer is constructed based on mybatis technology, the system platform packages the persistent layer, and all sql in the system is stored in a specific XML file instead of being stored in Java classes. By adopting the mode, the sql and the service code are separated, the long sql sentences are prevented from being spliced in java codes, and the reusability of the codes is improved.
The invention has the advantages that
And the big data component is utilized to carry out unified management and analysis on the logs of all the subsystems of the data sharing exchange platform. The standardized logs are created according to the services of the shared switching platform, and the logs and the services can be perfectly combined. Technically, based on a big data log processing component, a plurality of logs are effectively managed by combining a perfect standard log processing mode. The problems of high concurrency, huge log quantity and the like of a service gateway and an exchange channel can be met. In terms of service, a service log is formulated by combining the sharing platform, monitoring is carried out close to the service, abnormal conditions on the service can be found in time, and complex service judgment can be given to a system for processing. After the problem is found, the alarm notification can be timely carried out according to the configured business rule, and the work order can be automatically generated and distributed to operation and maintenance personnel. After the work order is processed, the work order can become knowledge sediment, and the knowledge accumulation is formed for the quick processing of the subsequent problems.
Drawings
Fig. 1 is a schematic workflow diagram of the entire monitoring.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer and more complete, the technical solutions in the embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention, and based on the embodiments of the present invention, all other embodiments obtained by a person of ordinary skill in the art without creative efforts belong to the scope of the present invention.
The invention relates to a method for realizing a data sharing exchange service operation monitoring system, which is used for carrying out integral monitoring on a data sharing exchange platform close to service data. And the whole monitoring of the shared exchange platform is carried out through the acquisition and processing of the structured data and the unstructured log data. The structured data comprises directory resource release conditions, resource application audit conditions and the like, and the unstructured log data comprises user login logs, user operation audit logs, service call logs, exchange task operation logs, exchange node routing inspection logs and the like.
The operation and maintenance monitoring function is automated to the maximum extent based on service configuration, and the service configuration mainly comprises the following steps: alarm notification rule configuration, alarm work order rule configuration, service call alarm configuration, service dial test alarm configuration, exchange task alarm configuration, and the like. Through configuration, the system can inform the patrol operators of how to warn, whether the warning automatically generates a work order and distributes operation and maintenance processing personnel, whether the service call generates the warning according to the business rule, whether the service dial test generates the warning according to the business rule, and whether the exchange task generates the warning according to the business rule, and brings the alarm into the automatic processing range of the system.
As shown in fig. 1, the service work content of the whole monitoring system is specifically described as follows:
the operation monitoring system mainly comprises the functions of a system monitoring module, a service monitoring module, a log auditing module, an operation and maintenance management module, a message center module, a configuration management module and the like.
For the treatment of alarm information and work order problems, the message center can be used for timely transmitting and receiving message notification, and various message notification modes are provided, mainly including in-station mail, short message, WeChat reminding and the like.
The system has a configuration center, and personalized configuration is carried out on the alarm notification reminding rule, the rule of automatically generating the work order by the alarm, the log storage period, the key service and the like, so that the personalized configuration requirements of the user are met.
The monitoring system carries out integral monitoring on the infrastructure supporting the operation of the service platform through a system monitoring module and mainly comprises infrastructure components such as system servers, middleware, databases and the like. The monitoring indexes of all monitored objects are flexibly configured by flexibly configuring monitoring items, monitoring templates and the like, custom monitoring frequency is supported, and monitoring results are flexibly obtained. And for the data exceeding the threshold value of the monitoring item, giving an alarm in time through a self-defined monitoring alarm rule, and informing a user in real time in a mode of mail, in-station mail, short message and the like. And early warning is carried out on monitoring items which are possibly abnormal through auditing and analyzing the monitoring results.
The system monitors the service through the service log and the service structured data. The log data mainly comprises service call logs, exchange task running condition logs, front node polling logs and the like; the business structured data mainly comprises application auditing conditions, directory resource providing conditions and the like. Under the condition of high concurrent call, the service agent system generates a large amount of call logs and exchange task running logs in the exchange channel, and the core functions of the existing monitoring system are real-time collection of large batches of logs, multi-log source collection and real-time monitoring analysis. The monitoring system monitors the calling condition of a calling party, the running condition of a service gateway and the health condition of an agent interface in real time through the real-time acquisition and analysis of the service calling log, and solves the problems that the problems are difficult to position, the calling analysis is not real-time and the like in the high concurrency process of the service agent; the monitoring system immediately discovers the condition that the preposed node is not timely checked with the exchange task by analyzing the running condition of the exchange task, and immediately informs a worker to intervene by means of alarming and the like. Through the log, operation and maintenance personnel can position the problem in time, and the problems of large data volume and difficult positioning in exchange subscription are solved.
Logging log and operation log of the user are closely related to the behavior of the user, logging and operation related auditing are carried out by collecting the related log in real time, the operation behavior of the user and the pressure bearing condition of each system module are known, abnormal operation of the user is monitored, and auditing analysis is carried out in time. And the abnormal operation of the user carries out real-time alarm according to the alarm rule configured by the administrator.
For the problems found in system monitoring and log real-time analysis auditing, a problem work order is generated in time through the operation and maintenance management function of the monitoring system, and the problems of early warning, abnormity, fault and consultation are classified and then distributed to an operation and maintenance worker of the system for processing, the processing process is supervised, and the problems are guaranteed to be solved in time. And operation and maintenance personnel can make a reasonable inspection plan, the daily inspection work is standardized and streamlined, and the integral operation of the service platform is powerfully guaranteed. For the found problems, the monitoring system carries out statistical analysis and centralized display through platform operation and maintenance conditions, platform operation conditions, knowledge base distribution conditions and the like, so that operation and maintenance personnel and system managers can conveniently inquire the platform operation state at any time, and the monitoring and management of the platform operation problems are realized.
The above description is only a preferred embodiment of the present invention, and is only used to illustrate the technical solutions of the present invention, and not to limit the protection scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (8)
1. A method for implementing a data sharing switching service operation monitoring system is characterized in that,
the data sharing exchange platform is integrally monitored from a bottom infrastructure to an upper layer service through acquisition and processing of structured data and unstructured log data, and the monitoring platform is operated to realize quick positioning from timely finding of abnormal conditions to warning problems and timely processing to an operation and maintenance work order.
2. The method of claim 1,
the method comprises the steps of monitoring infrastructure, monitoring service invocation, monitoring exchange tasks, auditing user operation and processing operation and maintenance problems.
3. The method of claim 1,
the operation monitoring system accesses a login log output by the unified login system, a user operation log output by more than one subsystem, a service call log of the service gateway and an exchange task operation log generated by the exchange channel into the unified monitoring system through a big data component.
4. The method of claim 3,
and accessing other service statistical data and alarm information into the operation management system storage through an REST interface and an ETL extraction mode.
5. The method according to claim 3 or 4,
after the operation monitoring system collects basic data of the data sharing exchange platform, the next step is to process the collected data.
6. The method of claim 5,
the log information collected by the operation monitoring system is cleaned and filtered through the big data assembly, the log information is processed, then the fragment indexing is carried out, the log data is put into the big data storage, and the retrieval service is provided for the upper layer.
7. The method of claim 5,
the service statistical data and the system operation condition data collected by the operation monitoring system are analyzed and processed through a timing task, and service alarm is timely performed according to the service configuration rule of the shared switching platform; and processing the basic data into an alarm library based on the basic data of the service library, and forming a knowledge library by the knowledge precipitation based on the alarm library and in cooperation with operation and maintenance management functions such as work order processing and the like.
8. The method of claim 1,
the platform architecture is hierarchically divided into three layers, namely: a presentation layer, a service layer and a persistence layer;
the presentation layer is constructed on a freemarkerTemplates technology, and is added with HTML, CSS and JAVASCRIPT technologies to display an interface in a browser;
the service layer is generally constructed based on the spring4.3 technology, and the dependent injection and AOP technology in the spring4.3 technology is introduced; on the other hand, the processing of large data volume combines the current big data processing component EKL to carry out storage and index analysis; a multithreading and event monitoring mechanism of JAVA is used for realizing a callback mechanism required in the data integration process;
the persistent layer is constructed based on mybatis technology, the platform packages the persistent layer, and all sql in the system is stored in a specified XML file.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010616561.XA CN111752808A (en) | 2020-07-01 | 2020-07-01 | Method for implementing data sharing exchange service operation monitoring system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010616561.XA CN111752808A (en) | 2020-07-01 | 2020-07-01 | Method for implementing data sharing exchange service operation monitoring system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111752808A true CN111752808A (en) | 2020-10-09 |
Family
ID=72676795
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010616561.XA Pending CN111752808A (en) | 2020-07-01 | 2020-07-01 | Method for implementing data sharing exchange service operation monitoring system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111752808A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112988514A (en) * | 2021-03-17 | 2021-06-18 | 浪潮云信息技术股份公司 | Monitoring method and system for exchange of base table and file |
| CN113516337A (en) * | 2021-03-25 | 2021-10-19 | 中国雄安集团数字城市科技有限公司 | Method and device for monitoring data security operation |
| CN113537824A (en) * | 2021-08-03 | 2021-10-22 | 南方电网数字电网研究院有限公司 | Power grid data full link monitoring system |
| WO2022073406A1 (en) * | 2020-10-10 | 2022-04-14 | 华为技术有限公司 | Network intent monitoring method, network intent monitoring system and storage medium |
| CN115221338A (en) * | 2022-09-08 | 2022-10-21 | 平安银行股份有限公司 | Knowledge graph construction method and system and computer equipment |
-
2020
- 2020-07-01 CN CN202010616561.XA patent/CN111752808A/en active Pending
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022073406A1 (en) * | 2020-10-10 | 2022-04-14 | 华为技术有限公司 | Network intent monitoring method, network intent monitoring system and storage medium |
| CN112988514A (en) * | 2021-03-17 | 2021-06-18 | 浪潮云信息技术股份公司 | Monitoring method and system for exchange of base table and file |
| CN112988514B (en) * | 2021-03-17 | 2023-02-21 | 浪潮云信息技术股份公司 | Monitoring method and system for exchange of base table and file |
| CN113516337A (en) * | 2021-03-25 | 2021-10-19 | 中国雄安集团数字城市科技有限公司 | Method and device for monitoring data security operation |
| CN113537824A (en) * | 2021-08-03 | 2021-10-22 | 南方电网数字电网研究院有限公司 | Power grid data full link monitoring system |
| CN115221338A (en) * | 2022-09-08 | 2022-10-21 | 平安银行股份有限公司 | Knowledge graph construction method and system and computer equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111752808A (en) | Method for implementing data sharing exchange service operation monitoring system | |
| CN112653586B (en) | Time-space big data platform application performance management method based on full link monitoring | |
| CN112612675B (en) | Distributed big data log link tracking method and system under micro-service architecture | |
| CN108197261A (en) | A kind of wisdom traffic operating system | |
| CN112965874B (en) | Configurable monitoring alarm method and system | |
| CN110581773A (en) | automatic service monitoring and alarm management system | |
| CN111858251B (en) | Data security audit method and system based on big data computing technology | |
| CN113179173B (en) | Operation and maintenance monitoring system for expressway system | |
| CN111581104A (en) | DevOps research and development operation integration-based measurement method | |
| CN110209518A (en) | A kind of multi-data source daily record data, which is concentrated, collects storage method and device | |
| Diotalevi et al. | Collection and harmonization of system logs and prototypal Analytics services with the Elastic (ELK) suite at the INFN-CNAF computing centre | |
| CN113746703B (en) | Abnormal link monitoring method, system and device | |
| CN116205396A (en) | Data panoramic monitoring method and system based on data center | |
| CN112269727A (en) | Monitoring and alarming method and system based on log information | |
| CN108897669A (en) | Using monitoring method and equipment | |
| CN117422434A (en) | Wisdom fortune dimension dispatch platform | |
| CN114167181A (en) | Method and system for monitoring local and allopatric line fault tracing | |
| CN116755992B (en) | Log analysis method and system based on OpenStack cloud computing | |
| CN109522349B (en) | Cross-type data calculation and sharing method, system and equipment | |
| CN116795631A (en) | Service system monitoring alarm method, device, equipment and medium | |
| CN110677271A (en) | Big data alarm method, device, equipment and storage medium based on ELK | |
| CN115408236A (en) | Log data auditing system, method, equipment and medium | |
| CN114531338A (en) | Monitoring alarm and tracing method and system based on call chain data | |
| CN114090529A (en) | Log management method, device, system and storage medium | |
| CN112506920A (en) | Information unified collecting and monitoring method and system for multiple financial systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |