CN110809017A - Data analysis application platform system based on cloud platform and micro-service framework - Google Patents

Abstract

The invention discloses a data analysis application platform system based on a cloud platform and a micro-service framework, which comprises: the system comprises a basic resource management platform, a data access module, a data storage module, a data service module and a system management module, wherein the system is based on the establishment of a virtual machine environment constructed by an Openstack platform, the function of a quasi-real-time data service platform data center is fully exerted, a small-core and large-periphery system is constructed, the requirements of a user on simple service application functions, easy deployment, easy expansion and quick online use are met, a data analysis application platform system based on a cloud platform and a micro-service framework is provided, and the system mainly comprises a platform application layer which mainly comprises modules of data access, data storage, data service, system management and the like.

Description

Data analysis application platform system based on cloud platform and micro-service framework

Technical Field

The invention relates to a data resource management platform technology in the power utilization industry, in particular to a data analysis application platform system based on a cloud platform and a micro-service framework.

Background

With the development and the secondary revolution of the internet, the enterprise is changed from informatization to datamation, and the datamation and intellectualization of the service are realized through the construction of a cloud platform. The southern power grid company serving as a national first-level service enterprise should pay more attention to the conversion process of production and service, construct intelligent and data service type power grid enterprises, and pay more attention to the quality and satisfaction degree of customer service. With the advance of various levels of data centers of the Yunnan power grid company, the Yunnan power grid company is developing a large data platform construction, integrating data and models of a CSGII system and a production system, and realizing unified and standardized management of the production system, an information system and a business application system. Particularly, along with the popularization and implementation of a CSGII enterprise management system and the provincial level concentration of related professional systems, a large amount of business data are concentrated in network provinces companies, local power supply departments cannot master data generated in the production and operation processes of the local power supply departments, the network provinces lack effective means of data backflow at present, but along with the requirement of lean management, the demands of each business department on data analysis and application are increased day by day, and the data analysis and application of data support distribution network operation monitoring, power grid planning construction, customer service and the like are used as a 'short board' in the lean management aspect of the Yuxi power supply departments. In order to better develop the service attributes of a power grid enterprise for the application and the information construction of the city business, a foundation cloud platform is urgently required to be built on the surface of the city to build the micro-service application, but at present, no such platform service system exists.

Disclosure of Invention

In order to solve the problems and defects in the prior art, the virtual machine environment constructed based on the Openstack platform is constructed, the functions of a data center of a quasi-real-time data service platform are fully exerted, a system with a small core and a large periphery is constructed, and the requirements of a user on simple service application functions, easy deployment, easy expansion and quick online use are met.

Data access: and accessing the distribution network operation data and the scheduling electric quantity data from the service system.

Data storage: the method comprises the steps of storing distribution network model data by using MySQL, storing distribution network operation real-time data by using hbase, and storing latest real-time data by using Redis so as to ensure quick access of applications.

Data service: and a micro-service framework is adopted to provide data services for distribution network monitoring, scheduling electric quantity monitoring and scheduling daily line loss calculation.

A system module: and the system is responsible for system safety management, system management, performance monitoring and the like.

The display layer mainly comprises a mobile APP, a large screen display and a webpage.

Compared with the prior art, the invention has the beneficial effects that:

(1) through research of cloud computing-based micro-service application, basic resource management platform construction based on a cloud computing technology is carried out, and requirements of users on simple service application functions, easy deployment, easy expansion and quick online use are met by utilizing a micro-service framework and combining a basic resource management platform.

(2) The method has the advantages that the function of a quasi-real-time data service platform data center is fully exerted, a supporting platform with small core and large periphery is constructed, and meanwhile, the model based on the quasi-real-time data service platform monitors key business indexes such as distribution network operation monitoring, alarming, business analysis, scheduling report forms and line loss management.

(3) Meanwhile, aiming at several main business modules of customer service, distribution transformer monitoring, monitoring alarm, business analysis, scheduling report forms, line loss management and the like, the method expands computing resources based on a model and data resources of a quasi-real-time data service platform, optimizes a distribution network model, monitors key business and indexes, analyzes customer electricity utilization information, and realizes micro-service application based on a cloud computing platform.

(4) Through application construction under a micro-service architecture, service analysis such as distribution transformer monitoring, monitoring alarm and the like is completed, customer electricity utilization information monitoring is achieved, customer electricity utilization service is optimized, day line loss management is supported, the problems of standard unification, difference shielding and component multiplexing among applications are solved, a core and a complex service module are packaged into a platform in a component mode, and the traditional complex front-end process is more transferred to a back end. And the service which is more flexible, more personalized, faster in response and more extensible is provided.

Drawings

FIG. 1 is a block diagram of the architecture of the present system;

FIG. 2 is a schematic diagram of the platform deployment architecture of the present system;

FIG. 3 is a graph of line loss calculations classified by voltage class for the present system;

FIG. 4 is a diagram of a power monitoring interface of the system;

FIG. 5 is a diagram of a display interface of line loss information

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the accompanying drawings in conjunction with specific embodiments. It should be understood that the description is intended to be exemplary only, and is not intended to limit the scope of the present invention. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present invention.

A cloud computing (cloud computing) platform is a way to implement processing of large-scale computing information, and it uniformly organizes and calls various information and communication technology resources through a network. From the development of cloud computing, the industry has seen the development momentum of cloud computing for a long time, and the current cloud computing development is mainly led and innovated by various IT enterprises in the United states.

In foreign countries, cloud computing programs have been introduced by leaders of the IT industry (IBM, Google, Microsoft, Amazon, VMware, etc.), and a great deal of scientific research effort has been put into them, so that the development of cloud computing is emphasized, and many colleges and research institutions worldwide are developing related researches. Among various cloud computing development modes, open-source cloud computing is an important direction for cloud computing development, and is even considered as a final trend for future development of cloud computing. Achieving flexible, free resource allocation and usage is the ultimate goal of cloud computing in enterprise applications. At present, the mainstream open source cloud computing platforms mainly include OpenStack, OpenNebula, Enomaly, Eucalyptus, cloudstock, and the like.

OpenStack is an open source project initiated by Rackspace corporation with NASA. OpenStack is capable of controlling and managing resources such as computing, storage, and networking of a data center, while providing a Web access interface and an API similar to Amazon EC2 for administrators and users to dynamically obtain needed resources on demand. Currently, the OpenStack foundation has 269 enterprise members and 12306 personal members worldwide, and OpenStack is adopted by more and more manufacturers and cloud computing service providers and applied to actual production environments, including IBM, HP, RedHat, Intel, Microsoft, etc. in the industry.

On the other hand, chinese users have high acceptance for OpenStack, and have been tried and practiced in various ways. The OpenStack-based approach is tried when developing and deploying their own cloud platforms for fun, cyber, etc. first, then OpenStack is selected by 360, Aiqiyi, Baidu, Jingdong, Youyou, Mei Tuo, etc. in succession.

Parallel programming techniques: the cloud computing achieves the purpose of collecting computing resources with high efficiency and low cost through a parallel programming mode, simultaneously ensures the parallel execution of complex programs in the background, and can realize the transparentization for users and programmers. The Map Reduce is a mode adopted by cloud computing, namely, a large task is divided into a plurality of subtasks, and the tasks are sequentially scheduled and distributed through the Map step and the Reduce step, so that the large task is finally processed.

Virtualization technology: virtualization technology is the basis of cloud computing. The virtualization technology is a technology for simulating hardware through software under the condition that real hardware conditions are insufficient and the requirement on the computing capacity of required hardware equipment is not high, so that the OS, application programs and hardware bottom equipment can be classified, the number of the equipment is greatly increased, and the requirements of different users can be met more flexibly. Virtualization technology enables the performance of a computer to be more flexible and efficient, and hardware can be utilized more comprehensively. The technologies mainly adopted at present include full virtualization, paravirtualization, operating system virtualization, CPU virtualization and the like.

A rapid deployment technique: in the application of cloud computing, a rapid deployment technology is an important core technology. Because in cloud computing, a system needs to be able to achieve a fast response to a user's needs to provide personalized needs required by the user, these need to deploy support for technology quickly.

High-speed network technology: the high-speed network technology is a core technology for supporting cloud computing, because without the high-speed network technology, resource sharing provided by the cloud computing cannot be realized. All resources in cloud computing are provided in the form of services, and all of these resources are transmitted over a network. The main communication in the cloud computing environment is CS (Client-Server) and SS (Server-Server).

Data storage technology: the data storage technology in cloud computing generally refers to a cloud storage technology, and the cloud storage technology is similar to the cloud computing technology and has the characteristics of high flexibility, expansibility and multiple tenants. The cloud service system can utilize technologies such as distributed storage, large-scale cluster and grid to form a virtual whole with different areas and different types of storage equipment through the network, and management software is used on an upper layer to manage storage resources on a lower layer to provide data storage and data access cloud services for users. Through the cloud storage technology, a user can use the cloud storage service in a dynamic on-demand mode, and the user does not need to consider the difference between the storage devices of the bottom layer and how the background controls and manages the devices. In addition, cloud storage also enables high fault tolerance of the system through redundant and distributed data.

The resource management technology comprises the following steps: resource management techniques are the management of computing resources, including hardware resources (e.g., CPUs, network devices, storage devices, etc.) and software resources (e.g., operating systems, programs, and applications). The resource management technology forms the resources into logical resources through abstraction, and the logical resources are integrated to serve the user as a single integrated resource pool. The user can directly use the resources required by the user without being concerned about the position of the underlying physical resources and how to implement the process.

The micro-service architecture is to adopt a group of services to construct an application, the services are independently deployed in different processes, and different services communicate through some lightweight interaction mechanisms.

The SG-UAP micro-service basic framework mainly comprises: routing gateway, registration center, configuration center, service administration and the like. The SG-UAP micro application framework adopts Spring-boot, Spring-closed and other open source technologies which are mainstream in the industry, and all indexes reach the advanced level in the industry.

Spring Boot is a completely new framework provided by the Pivotal team, and is designed to simplify the initial building and development process of new Spring applications. The framework uses a specific way to configure, thereby eliminating the need for developers to define a templated configuration. The Spring Boot is not a framework, and is a collection of libraries fundamentally, and Spring boots can be used by introducing corresponding dependence on maven or gradle items without self-management of the versions of the libraries.

The Spring Cloud provides a tool for developers to quickly build in a distributed system (configuration management, service discovery, fusing, routing, control bus, distributed session and the like), and the developers using the Spring Cloud can quickly start service or build application and can quickly interface with Cloud platform resources.

OpenStack is an open-source cloud computing platform and aims to achieve simplicity, large-scale scalability and rich functions. OpenStack provides open source software, establishes public and private clouds, provides an operating platform or toolset for deploying the clouds, and aims to: the cloud running as virtual computing or storage service is helped to be organized, and extensible and flexible cloud computing is provided for public clouds, private clouds, big clouds and small clouds.

Example 1:

a data analysis application platform system based on a cloud platform and a micro-service framework comprises: the system comprises a basic resource management platform, a data access module, a data storage module, a data service module and a system management module; wherein:

1) a base resource management platform comprising:

the virtualized resource management module is used for integrating the resource management equipment through a virtualization technology, connecting the resource management equipment to a cloud platform for management, and providing an external basic service interface;

the resource scheduling management module is used for effectively monitoring and managing physical resources and virtual resources, establishing a service model, and providing functions of elastic calculation, load balancing, dynamic migration, supply on demand and automatic deployment by extracting the service model; the unified management, the unified distribution, the unified deployment, the unified monitoring and the unified backup of hardware resources and virtual resources are realized;

2) data access module, includes:

the model data access module is used for accessing and synchronizing the model data in an incremental mode, after the model data is accessed, if the source-end model data changes, the changes are sent to the big data platform in an event mode, and the big data platform performs incremental updating on the model data;

3) a data storage module comprising:

the large data platform storage module comprises a MySQL module and an opensdb + hbase module, and is respectively used for storing scheduling and distribution network model data and power grid operation time sequence data;

4) a data service module comprising:

the dispatching daily line loss calculation module is used for calculating and analyzing the line loss, and calculating and counting daily line loss data according to the subareas, the distribution areas and the divided voltage grades; displaying the day line loss data according to the region, the transformer area, the circuit and the voltage grade in multiple dimensions; the system is used for analyzing the obviously abnormal line loss data and can check whether the data is caused by the data quality problem;

the dispatching electric quantity monitoring module is used for establishing a hanging relation between electric quantity data and a model according to the dispatching model, monitoring the electric quantity collection of the city and the ground, finding out the loss, abnormality and the like of the electric quantity in time and giving an alarm;

the distribution network operation monitoring module is used for monitoring the distribution network operation condition, and the distribution network operation condition comprises distribution transformer equipment operation, static parameters, equipment load condition and real-time electric energy;

the application display module is used for constructing the front end of the mobile application through the APP application, large-screen display and a webpage, realizing the mobile application of the basic resource management platform, facilitating the user to quickly and conveniently acquire basic resource data, key indexes and other functions at any time and any place, and realizing the access of different users to related data according to business requirements through authority control;

the model and data query module is used for providing a model query function so as to facilitate the user to browse and check the data; the system is used for displaying line loss information in a chart mode according to voltage grades, power supply areas, transformer areas, lines and the like, and highlighting abnormal line loss; the scheduling electric quantity information is analyzed according to statistics of lines, gateways, stations and the like, and is displayed in a chart form;

5) the system management module includes:

and the monitoring alarm service module is used for monitoring the running state of the cloud platform, the running state of the big data platform, the running state of the data access server, the running state of the data micro-service server and the running state of the application server, analyzing and alarming the abnormity and generating an abnormity analysis report.

Wherein, the system management module still includes:

and the system authority management follows the southern power grid information safety protection management specification, unified authority verification is carried out through a provincial company authority management platform, and the user authority mainly comprises: access rights, editing rights; the user authority is maintained by a System Administrator (SA), and after the user successfully logs in, corresponding operation can be carried out according to the own authority;

the log management module is used for helping the platform operation and maintenance personnel to manage and maintain the platform, and the logs are divided into system operation logs and operation logs; the system operation log is used for automatically generating record information according to configuration when each module of the system operates; the operation log provides records of operations performed by a user through an interface, and the records comprise configuration parameter modification, data addition, deletion, modification, check, import, export, node master-slave switching and the like;

the backup and recovery module is used for providing data backup and recovery functions when data damage and other conditions occur due to accidents of hardware failure, misoperation of an administrator and the like in the operation of the platform system so as to ensure data safety.

The basic resource management platform is an infrastructure platform which is built based on an openstack cloud platform, a big data platform and a container management platform.

Wherein, the front-end construction through large-screen display comprises: and PC (personal computer) end access is provided for dispatching line loss, dispatching electric quantity monitoring and distribution network operation monitoring, a BS (base station) framework is adopted, micro-service is adopted as a background service end to provide data for a front-end page, and the distribution network operation monitoring realizes a large-screen display function at the same time.

Wherein, the data storage module still includes: and the cloud platform time sequence data storage is used for storing the numerical data with the time marks, the storage objects are acquired measuring point data and time sequence data, the time sequence database independently stores real-time values and historical values according to the data time mark sequence, and IO (input/output) consumption of data operation is reduced through technologies such as data compression, file blocking and data time mark indexing so as to improve the efficiency.

The backup and recovery module is also used for full backup and incremental backup during backup, the full backup is to backup all contents, and the incremental backup only backs up all the changed data since the last backup; and during recovery, the method is also used for recovering the platform mirror image, checking the running condition of the real-time data platform mirror image service, checking the numerical values, the time labels and the data refreshing condition of the same measuring points in the database and judging whether the mirror image software runs well or not.

The virtualized resource management module also comprises a micro-service component, the micro-service component realizes the calling between micro-services through service registration and service discovery, thereby realizing the decoupling between the micro-services, and is used for providing a uniform access entrance for the front-end application by the routing gateway under the micro-service architecture mode, and realizing the calling of the target service through the routing strategy, thereby simplifying the condition that each micro-service needs to expose access information to the front-end application; the configuration service provides storage of unified dynamic configuration information under multiple environments for the micro-service, and synchronizes the updated information to the micro-service application in a push-pull mode under each operating environment.

Example 2:

(1) infrastructure construction

In a 'big platform + micro service' architecture, 3 sets of infrastructure platforms, namely a cloud platform based on openstack, a big data platform and a container management platform, need to be built.

(2) Basic storage platform

The XSKY enterprise-level data platform architecture provides data persistence, fusion calculation, built-in protection and distribution copying functions among multiple data centers, heterogeneous private clouds and mixed clouds. The storage stores standard protocols for interfacing with applications, such as block storage, and may interface via standard protocols such as iSCSI, FC, RBD, etc., file storage interfaces via protocols such as NFS, CIFS, etc., and object storage interfaces via protocols such as S3.

The system has three layers of architectures, namely an IaaS (infrastructure as a service) layer, a PaaS (platform as a service) layer and a SaaS (software as a service) layer.

An IaaS (infrastructure as a service) layer mainly comprises a network, basic resources and data resources, the infrastructure as a service is realized through a hardware virtualization technology and bottom layer resource packaging, and the layer of service is mainly built based on OpenStack.

The PaaS layer mainly provides microservice. The Spring Boot is a set of rapid configuration scaffolds of the Spring, a single micro service can be rapidly developed based on the Spring Boot, and the Spring Cloud is a Cloud application development tool realized based on the Spring Boot; spring boots focus on single individuals for quick, convenient integration, and Spring Cloud is a service governance framework that focuses on the whole.

The SaaS layer provides applications. And acquiring monitoring data by utilizing the micro service API, and displaying the data access monitoring condition by using spring as a micro application framework. And generating a blending report by using a FineReport report function.

(3) Cloud platform construction

OpenStack is an open-source cloud computing platform and aims to achieve simplicity, large-scale scalability and rich functions. OpenStack provides open source software, establishes public and private clouds, provides an operating platform or toolset for deploying the clouds, and aims to: the cloud running as virtual computing or storage service is helped to be organized, and extensible and flexible cloud computing is provided for public clouds, private clouds, big clouds and small clouds.

Based on OpenStack and distributed storage technology, the functions of an IaaS (infrastructure as a service) layer are realized on the basis of the existing server virtualization, and a high-elasticity and high-reliability running platform is provided for self-built systems, mobile applications and micro applications by providing computing resources and storage resources in a self-service mode.

OpenStack integrates storage capacity, computing capacity, network resources and the like through virtualization to form a cloud computing system, and IaaS service is provided for users. The cloud platform provides the virtualized packaged basic resources, network resources and data storage for the micro applications and the micro services. The virtual machine technology is the key for realizing cloud computing, in an OpenStack platform, a virtual machine is created through Nova, the mirror image and storage of the virtual machine are mainly provided through Glance and shader, and the network service of the virtual machine is mainly realized through Neutron. In addition, the platform realizes the storage of objects through Swift, and is mainly used for storing files such as a circle backup volume and a Glance mirror image file.

The cloud platform also needs to provide a routing gateway for the micro-service, in OpenStack, the function is provided by a Neutron project, and in Neutron, a Neutron-server is responsible for receiving API requests and forwarding the requests to other components for execution; neutron agents and neutron plugins are responsible for enabling and disabling ports, creating networks and subnets, providing IP addresses, etc. plugins and agents differ in the vendor and technology they use in a particular cloud, commonly used agents are L3, DHCP and plug-in agents, etc.

The cloud platform is used as a data source of the micro-service, and data in the platform is stored in a distributed mode, so that network and I/O bottlenecks encountered by mass data throughput in a single machine mode are effectively avoided. Meanwhile, data are stored in a plurality of nodes, and the integrity and the safety of the data when a single-node fault occurs are guaranteed. Aiming at time sequence data, in order to realize high-speed writing and query efficiency, a time sequence database independently stores a real-time value and a historical value according to a data time mark sequence, and reduces I/O (input/output) consumption of data operation through technologies such as data compression, file blocking, data time mark indexing and the like so as to improve efficiency.

(4) Big data platform construction

The platform needs to access scheduling data and distribution network data, and the data comprises structured model data and unstructured time sequence data. The model data is relatively fixed and can be stored in a Relational database MySQL, which is one of the most popular Relational database management systems, and in terms of WEB applications, MySQL is the best RDBMS (Relational database management System) application software.

The time sequence data has the characteristics of high concurrency, low time delay, large data volume and high reliability requirement, so the openstb + hbase mode is adopted for storage. The HBase is a distributed column storage system constructed on the HDFS; HBase is developed based on the Google BigTable model, a typical key/value system; HBase is an important member in an Apache Hadoop ecosystem and is mainly used for mass real-time data storage; logically, the HBase stores data in tables, rows and columns.

For the latest real-time data, in order to accelerate the access speed and improve the application efficiency, the latest real-time data can be stored in a Redis database.

(5) Container management platform set-up

Based on the Docker container technology, the platform as a service (PaaS) layer function is realized, and an operating environment is provided for the micro-service.

The virtualization technology changes a modern computing mode, can improve the use efficiency of system resources, eliminate the dependency relationship between an application program and bottom hardware, and meanwhile strengthen the portability and the safety of loads. Virtualization essentially reproduces the fact that the entire physical server runs an application as one virtual machine, acting as an abstract server resource, with each virtual machine being able to acquire a unique operating system and load. However, the virtual machine technology has a trouble that each instance needs to run a complete copy of the client operating system and a large number of applications contained therein, thereby generating a heavy load, which affects the working efficiency and performance of the virtual machine. Against the background of such a demand, container technology has emerged. The container is more efficient than the traditional virtualization technology, the container does not virtualize or abstract the whole hardware but only abstracts the application or part of the application, the virtualization with the granularity means that resources are not wasted in redundant parts, and simultaneously, the CPU, the memory and the storage requirements can be reduced, so the container technology is also a virtualization technology.

The container virtualization has the light weight characteristic, the required memory space is less, the very fast starting speed is provided, the container creating speed is much higher than that of a virtual machine, because the virtual machine must retrieve 10GB to 20GB of operating systems from a storage system, the operating system kernel of a host server is used by the workload in the container, the step is avoided, and the container can be started in twenty-one seconds; the container virtualization enables the application to be formatted in a standard manner before being placed in the container. In the container, each type of application moves on the network in the same manner. In this way, the container may be moved through the internet or an intranet; container virtualization provides a higher level isolation mechanism, many applications run under a host operating system, all applications share some operating system libraries and the kernel of the operating system, and running containers are prevented from conflicting with each other; container virtualization can break a large application down into many applets, each within a respective container. Such as: discounting the luxury web site Gilt breaks seven large applications down into 300 micro-services, and a small team maintains each service and can roll back/recover quickly if a problem occurs with one micro-service. The container virtualization technology is higher in safety, and containers can be accessed independently. Modifying code changes of one layer may be performed without affecting other layers. In this way, code is more secure to change than in a typical monolithic application.

Docker is an LXC-based advanced container engine sourced by PaaS provider dotCloud, with source code hosted on gitubs, based on ***-issued open source programming language Go. Docker allows developers to package their applications and dependencies into a portable container and then distribute them to mainstream Linux machines. The containers are completely sandboxed, do not have any interface with each other (like the app of the iPhone), have low performance overhead, and can be easily run in machines and data centers. The significance of the method can be compared with the invention of a container, the container standardizes the freight target, and Docker standardizes the application program. Before the birth of Docker, the procedure of deploying an application program by a programmer on a server side is as follows: installation → configuration → operation. Unlike the traditional deployment mode, with Docker, the procedure for the programmer to deploy the application on the server side is as follows: copy → run.

(6) Data access

The platform requires to access model data and real-time data of a dispatching and distribution network, and different strategies are adopted for data access aiming at different characteristics of the model data and the real-time data. The model data is relatively fixed, the change frequency is not high, access and synchronization can be carried out in an incremental mode, after the model data is accessed, if the source end model data changes, the changes are sent to the platform in an event mode, and the platform carries out incremental updating on the model data. The real-time data transmission quantity is large, and the data can be accessed by the platform in an active calling mode.

Aiming at the conditions of model loss, real-time data loss and the like, the platform provides the functions of manually calling and testing the model and the real-time data.

(7) Micro-services

And the line loss is calculated and analyzed, and the daily line loss data is calculated and counted according to the voltage grades of the subareas, the station subareas and the branch voltage grades. And displaying the daily line loss data according to a plurality of dimensions such as areas, transformer areas, lines, voltage grades and the like. And analyzing the line loss data with obvious abnormity after calculation, and checking whether the line loss is abnormal due to the data quality problem. The graph is the line loss calculation classified by voltage class.

The platform provides a dispatching electric quantity monitoring function, establishes a hanging connection relation between electric quantity data and a model according to a dispatching model, monitors the electric quantity collection of the city, finds out the loss, abnormality and the like of the electric quantity in time, and gives an alarm. The figure shows a power monitoring interface.

The method is used for monitoring the operation condition of a distribution network, and mainly relates to the operation of distribution and transformation equipment, static parameters, equipment load conditions, real-time electric energy and the like at present. The method has the advantages of realizing line voltage, current and load monitoring, power quality analysis, voltage qualification rate, line fault rate, distribution transformer fault rate, defect elimination and timeliness monitoring, operation event monitoring, power failure overall process analysis, heavy overload analysis, line trip analysis, low voltage analysis, fault reason analysis and voltage abnormal condition monitoring.

(8) PC terminal and large screen display

And aiming at providing PC (personal computer) end access for scheduling line loss, scheduling electric quantity monitoring and distribution network operation monitoring, a BS (base station) architecture is adopted for realizing. Namely, micro-service is adopted as a background server to provide data for a front-end page. And simultaneously developing and realizing a large-screen display function during the operation monitoring of the distribution network.

(9) Mobile APP display

Based on Android and IOS platforms, the mobile application front end is constructed in a native mode, mobile application of a basic resource management platform is achieved, users can conveniently and quickly obtain basic resource data and key indexes at any time and any place, and access of different users to related data according to business requirements is achieved through authority control.

At present, mobile application integrated platforms are gradually popularized and used, and need to be accessed after APP development is completed.

And performing single sign-on access according to the mobile application integration platform specification and the related SDK, linking an intranet by using a VPN (virtual private network) provided by the mobile integration SDK, entering a special safety channel to enhance the safety, enabling the system to meet the requirement of the development safety standard of the application system of the Yunnan power grid company, submitting related audit materials, and completing related third-party tests so as to realize auditing, testing and putting on shelf on the mobile application integration platform.

Example 3: a data analysis application platform system framework based on a cloud platform and a micro-service framework comprises:

fig. 1 shows the overall architecture of the system. Computing resources and network resources are integrated by the cloud platform, and a storage resource management platform is integrated by the XSky to provide storage resources for the cloud platform.

In order to reasonably utilize hardware resources, openstack is utilized to build a cloud computing platform, various hardware resources are managed, and resources are reasonably distributed for application. And (3) realizing rapid iteration and hot updating of the application by using a docker container technology, and building a container management platform by using K8S. And constructing a big data platform for storing distribution network and scheduling model data and real-time data.

The platform application layer mainly comprises modules of data access, data storage, data service, system management and the like.

Data access: and accessing the distribution network operation data and the scheduling electric quantity data from the service system.

Data storage: the method comprises the steps of storing distribution network model data by using MySQL, storing distribution network operation real-time data by using hbase, and storing latest real-time data by using Redis so as to ensure quick access of applications.

Data service: and a micro-service framework is adopted to provide data services for distribution network monitoring, scheduling electric quantity monitoring and scheduling daily line loss calculation.

A system module: and the system is responsible for system safety management, system management, performance monitoring and the like.

The display layer mainly comprises a mobile APP, a large screen display and a webpage.

As shown in fig. 2, which is a platform deployment architecture, the hardware server implements centralized management and uniform allocation of computing resources, storage resources, and network resources through a virtualization technology. 7 cloud hosts are created through a cloud platform, and the cloud host 1 serves as a data access server to obtain data from a data source server; the cloud hosts 2-4 are used as big data clusters to construct a data storage platform; the cloud hosts 5 and 6 are used as micro service servers and provide data services; the cloud host 7 provides an application service to the outside as an application server.

Design of basic resource management platform

OpenStack integrates storage capacity, computing capacity, network resources and the like through virtualization to form a cloud computing system, and IaaS service is provided for users. The cloud platform provides the virtualized packaged basic resources, network resources and data storage for the micro applications and the micro services. The virtual machine technology is the key for realizing cloud computing, in an OpenStack platform, a virtual machine is created through Nova, the mirror image and storage of the virtual machine are mainly provided through Glance and shader, and the network service of the virtual machine is mainly realized through Neutron. In addition, the platform realizes the storage of objects through Swift, and is mainly used for storing files such as a circle backup volume and a Glance mirror image file.

The cloud platform also needs to provide a routing gateway for the micro-service, in OpenStack, the function is provided by a Neutron project, and in Neutron, a Neutron-server is responsible for receiving API requests and forwarding the requests to other components for execution; neutron agents and neutron plugins are responsible for enabling and disabling ports, creating networks and subnets, providing IP addresses, etc. plugins and agents differ in the vendor and technology they use in a particular cloud, commonly used agents are L3, DHCP and plug-in agents, etc.

The cloud platform is used as a data source of the micro-service, and data in the platform is stored in a distributed mode, so that network and I/O bottlenecks encountered by mass data throughput in a single machine mode are effectively avoided. Meanwhile, data are stored in a plurality of nodes, and the integrity and the safety of the data when a single-node fault occurs are guaranteed. Aiming at time sequence data, in order to realize high-speed writing and query efficiency, a time sequence database independently stores a real-time value and a historical value according to a data time mark sequence, and reduces IO consumption of data operation through technologies such as data compression, file blocking, data time mark indexing and the like so as to improve efficiency.

Virtualization platform design

Virtualization technology: the software application can be isolated from the underlying hardware through virtualization technology, which comprises a split mode of dividing a single resource into a plurality of virtual resources and an aggregation mode of integrating a plurality of resources into one virtual resource. Virtualization technologies can be divided into storage virtualization, computing virtualization, network virtualization, and the like according to objects, and computing virtualization is further divided into system level virtualization, application level virtualization, and desktop virtualization. The platform builds a cloud management platform through openstack to realize virtualization of storage resources, computing resources and network resources. The cloud management platform can realize the functions of uniformly scheduling and rapidly delivering cloud machine resources, monitoring the running performance of the whole computer platform in real time, reporting logs and the like; meanwhile, a user interface is realized, and a user can apply and manage own cloud resources through authority control. The unified management, the unified distribution, the unified deployment, the unified monitoring and the unified backup of hardware resources and software resources are realized. The system comprises a hardware basic implementation layer, a virtualization and resource pooling layer and a resource scheduling and management automation layer.

Big data platform data storage design

The platform needs to access scheduling data and distribution network data, and the data comprises structured model data and unstructured time sequence data. The model data is relatively fixed and can be stored in a Relational database MySQL, which is one of the most popular Relational database management systems, and in terms of WEB applications, MySQL is the best RDBMS (Relational database management System) application software.

The time sequence data has the characteristics of high concurrency, low time delay, large data volume and high reliability requirement, so the openstb + hbase mode is adopted for storage. The HBase is a distributed column storage system constructed on the HDFS; HBase is developed based on the Google BigTable model, a typical key/value system; HBase is an important member in an Apache Hadoop ecosystem and is mainly used for mass real-time data storage; logically, the HBase stores data in tables, rows and columns.

For the latest real-time data, in order to accelerate the access speed and improve the application efficiency, the latest real-time data can be stored in a Redis database.

Data access design

The application platform provides a standard data access function, and accesses the real-time data of the service system required by the platform support application to the big data platform.

Access protocol

The application platform should support a unified data communication protocol to standardize data access of different service systems, and should support extension of the communication protocol for facilitating access of the existing service system. The platform should support at least the following communication protocols:

1) general protocol: the universal protocol is a communication message transmission format established by the platform for realizing the efficient and uniform access of the real-time data of each service system;

2)104, specification: a power communication standard protocol;

3) e language: power system data markup language compliant with IEC61970/61968 specifications.

4)102, stipulation: electric energy communication protocol

Other protocols include metering automation protocol (including large client, distribution transformer, low-voltage meter reading), JMS message mechanism and 61850 protocol.

Accessed service system

And various service systems, marketing systems, production systems and the like accessed by the platform, input model data and the like.

Access configuration management

The data analysis application platform accesses real-time data from various service systems, and the service systems are data sources of the data analysis application platform. The access configuration management is mainly responsible for the unified management of communication parameters between the data analysis application platform and each data source system. The main functions include: the data source system has the functions of addition, deletion, modification, query and the like:

1) increase: the function of adding a data source system is supported, and parameters for communication with the data source system are set during adding;

2) and (3) deleting: the function of supporting the deletion of the data source system;

3) modifying: the function of modifying the configuration parameters of the data source system is supported;

4) and (3) inquiring: and supporting the function of querying a data source system.

The data source system configuration parameters mainly comprise: communication protocol, communication mode, port address, etc.

Access handling

The access processing means: according to the access configuration, the data analysis application platform is communicated with the data source service system through a specified communication protocol and a communication mode, receives real-time data of the source service system, and converts the received data into a data format inside the data analysis application platform big data platform.

The access processing functions mainly include:

1) establishing a communication link with each source service system according to the configuration;

2) according to a communication protocol, communicating with a source service system and receiving real-time data;

3) and carrying out format conversion on the accessed data.

4) During data access, the modification of the configuration parameters should be automatically adapted, and the access process should not be restarted due to the change of the communication parameters.

Access monitoring management

The data analysis application platform provides an access monitoring function and realizes the visualization and the controllability of data access. The main functions include:

1) providing an interface to facilitate real-time monitoring of link connection state, data traffic, connection time, packet information, and the like;

2) providing an interface to facilitate statistics and queries of connection states, running time, down time, data traffic, and the like;

3) providing an interface to facilitate control operations such as running, stopping and the like on the communication link;

4) and providing an alarm function when the states of the access source service system, the network channel and the like are abnormal.

Access exception handling

The access exception handling is to solve the problem of data loss caused by the exception condition (if the network is interrupted, etc.) in the access process with the source service system, the system provides an alarm mechanism, and provides a statistical analysis function for the exception condition to provide a reference for the operation and maintenance of the platform system.

The access exception handling function mainly comprises:

1) for the abnormal operation condition of the access source service system, after the abnormal condition is recovered, an additional recording mechanism is provided to ensure that the data is not lost;

2) and functions of inquiring and counting access abnormal information are provided.

The query content comprises: time of occurrence, cause, etc.;

the statistical functions include: statistics of the frequency of occurrence of anomalies, etc.

Data access

The platform provides data services externally in a micro-service mode, and provides corresponding micro-services for scheduling day line loss calculation, scheduling electric quantity monitoring, distribution network operation monitoring application and mobile APP application. The mode of providing data by the micro service is restful, the mode has the advantages of strong universality and easy integration, and meanwhile, the micro service can also realize load balance of access.

Scheduling daily line loss calculation service

As shown in fig. 3, the calculation and analysis of the line loss are realized, and the daily line loss data are calculated and counted according to the partition, the distribution area and the partial voltage grade. And displaying the day line loss data according to multiple dimensions such as areas, transformer areas, lines, voltage levels and the like. And analyzing the line loss data with obvious abnormity after calculation, and checking whether the line loss is abnormal due to the data quality problem. Such as line loss calculations classified by voltage class.

Scheduling power monitoring service

As shown in fig. 4, the platform provides a scheduling electric quantity monitoring function, establishes a hitching relationship between electric quantity data and a model according to a scheduling model, monitors the electric quantity collection in the city, finds out the loss, abnormality and the like of the electric quantity in time, and gives an alarm. The figure shows a power monitoring interface.

Distribution network operation monitoring service

The method is used for monitoring the operation condition of a distribution network, and mainly relates to the operation of distribution and transformation equipment, static parameters, equipment load conditions, real-time electric energy and the like at present.

And (3) monitoring the distribution transformer load: and monitoring and alarming the equipment load according to relevant standards of the Yunnan power grid company on overload operation of the distribution transformer. Such as: the overload is determined when the single-phase current and the three-phase current of the oil-immersed transformer exceed 100% of the rated quantity, exceed 80% of the rated quantity, exceed 10 times in one month and last for 2 hours each time.

Analyzing distribution network load characteristics: and monitoring and early warning the load condition of the distribution transformer by using data such as date, season, holidays, tourists, weather, industrial processing characteristics and the like.

The distribution network monitoring function also comprises electric energy power quality analysis, operation index monitoring such as voltage qualification rate, line fault rate, distribution transformer fault rate and failure elimination timeliness rate, operation event analysis, power failure full process analysis, heavy overload analysis, line tripping, low voltage and fault reason analysis and voltage abnormal condition monitoring.

Alert service

And designing an alarm service component in a background management interface, wherein the content comprises the running state of a monitoring cloud platform, the running state of a big data platform, the running state of a data access server, the running state of a data micro-service server and the running state of an application server, and analyzing and alarming the abnormality.

And monitoring the line loss, the electric quantity and the running state of the distribution network, giving an alarm for the occurrence of the abnormality, and generating an abnormality analysis report.

Chart presentation application

As shown in fig. 5, the line loss information is displayed in a graph form according to the voltage class, the power supply area, the station area, the line, and the like, and the abnormal line loss is highlighted.

And (4) carrying out statistical analysis on the scheduling electric quantity information according to lines, gateways, stations and the like, and showing the scheduling electric quantity information in a chart form.

System management

The platform authority management follows the southern power grid information safety protection management standard, unified authority verification is carried out through the province company authority management platform, and the user authority mainly comprises the following steps: access rights, editing rights, etc. The user authority is maintained by a System Administrator (SA), and after the user logs in successfully, corresponding operation can be carried out according to the own authority.

Log management

The log management function is mainly used for helping platform operation and maintenance personnel to better manage and maintain the platform, and logs can be divided into system operation logs, operation logs and the like. The system operation log is mainly record information automatically generated according to configuration when each module of the system operates, and the operation log is a record of operation performed by a user through an interface and comprises configuration parameter modification, data addition, deletion, modification, check, import, export, node master-slave switching and the like.

The log management function shall include log query, delete and export functions.

1) Log query: the query log is supported by time period or type and their combination condition.

2) And log deletion: in order to control the excessive storage space occupied by the log records, the platform provides automatic and manual log deleting functions.

The automatic deletion is to automatically delete the records meeting the deletion condition in the system operation according to the configuration strategy.

Manual deletion is the provision of a tool for a user to perform a delete log record operation.

And log export: the function of exporting the log in a fixed format (such as CSV, XML and the like) is supported.

Backup and restore

In the operation process of the platform system, accidents such as hardware faults and misoperation of an administrator sometimes occur, so that data damage and the like occur. The platform should provide data backup and recovery functions to ensure data security.

1) Content of backup

Configuration data: equipment, measurement and station configuration data, etc.

History data: and (4) measuring point historical data stored in a big data platform or a relational database.

And (3) graphic data: and graphic and picture data such as wiring diagrams, tidal current diagrams and the like for WEB display.

Making a system: operating system configuration information, program files, and logs, among others.

2) Backup management

The automatic backup and the manual backup can be realized.

A backup mode: there are two ways of full backup and incremental backup. A full backup is a backup of the entire content, and an incremental backup only backs up all changes since the last backup. The fully backed up data alone may be used to restore the system, but to restore the incremental backup, it is necessary to gradually restore from the previous full backup.

The backup method comprises the following steps: for real-time data backup, the real-time and historical data stored by the large data platform HBase have a plurality of backups during access, so that manual backup is not needed; for the model data stored in the relational database, adopting a self-contained export tool of the relational database for backup; for files, backup is carried out by adopting methods such as compression and the like; the backup of the operating system generally adopts a method of cloning an image.

Monitoring the backup condition: and displaying the backup progress and the content information.

And (3) backup exception handling: and giving detailed prompt when abnormality occurs in the backup process.

3) Recovery management

And (3) data recovery: including configuration data, historical data, image data, etc., which are stored in a real-time database or a relational database, and data consistency is guaranteed in data recovery.

And (3) platform recovery: and rapidly restoring the platform to the previous state according to the backup.

Platform mirroring: checking the running condition of the real-time data platform mirror image service; and checking the numerical values, the time labels and the data refreshing conditions of the same measuring points in the database, and judging whether the mirror image software runs well.

4) Filing management

The primary content archived by the platform includes: filing a log file, filing a backup file, and filing a historical data file generated by a real-time database;

the archive policy setting includes: setting filing period and filing file naming rule as required;

and (3) inquiry statistics of the archived files: archived files may be retrieved based on date; the archived files may be queried based on conditions.

It is to be understood that the above-described embodiments of the present invention are merely illustrative of or explaining the principles of the invention and are not to be construed as limiting the invention. Therefore, any modification, equivalent replacement, improvement and the like made without departing from the spirit and scope of the present invention should be included in the protection scope of the present invention. Further, it is intended that the appended claims cover all such variations and modifications as fall within the scope and boundaries of the appended claims, or the equivalents of such scope and boundaries.

Claims (7)

1. A data analysis application platform system based on a cloud platform and a micro-service framework is characterized by comprising: the system comprises a basic resource management platform, a data access module, a data storage module, a data service module and a system management module; wherein:

1) a base resource management platform comprising:

the virtualized resource management module is used for integrating the resource management equipment through a virtualization technology, connecting the resource management equipment to a cloud platform for management, and providing an external basic service interface;

the resource scheduling management module is used for effectively monitoring and managing physical resources and virtual resources, establishing a service model, and providing functions of elastic calculation, load balancing, dynamic migration, supply as required and automatic deployment by extracting the service model; the unified management, unified distribution, unified deployment, unified monitoring and unified backup of hardware resources and virtual resources are realized;

2) data access module, includes:

the model data access module is used for accessing and synchronizing the model data in an incremental mode, after the model data is accessed, if the source-end model data changes, the changes are sent to the big data platform in an event mode, and the big data platform performs incremental updating on the model data;

3) a data storage module comprising:

the large data platform storage module comprises a MySQL module and an opensdb + hbase module, and is respectively used for storing scheduling and distribution network model data and power grid operation time sequence data;

4) a data service module comprising:

the dispatching daily line loss calculation module is used for calculating and analyzing the line loss, and calculating and counting daily line loss data according to the subareas, the substation areas and the divided voltage grades; displaying the day line loss data according to the region, the transformer area, the line and the voltage grade in multiple dimensions; the system is used for analyzing the obviously abnormal line loss data and can check whether the data is caused by the data quality problem;

the dispatching electric quantity monitoring module is used for establishing a hanging connection relation between electric quantity data and a model according to a dispatching model, monitoring the electric quantity collection of the city, finding out the loss, abnormality and the like of the electric quantity in time and giving an alarm;

the distribution network operation monitoring module is used for monitoring the distribution network operation condition, wherein the distribution network operation condition comprises distribution transformer equipment operation, static parameters, equipment load condition and real-time electric energy;

the application display module is used for constructing the front end of the mobile application through the APP application, large-screen display and a webpage, realizing the mobile application of the basic resource management platform, facilitating the user to quickly and conveniently acquire basic resource data, key indexes and other functions at any time and any place, and realizing the access of different users to related data according to business requirements through authority control;

the model and data query module is used for providing a model query function so as to facilitate users to browse and check data; the system is used for displaying line loss information in a chart mode according to voltage grades, power supply areas, transformer areas, lines and the like, and highlighting abnormal line loss; the scheduling electric quantity information is analyzed according to statistics of lines, gateways, stations and the like, and is displayed in a chart form;

5) the system management module includes:

and the monitoring alarm service module is used for monitoring the running state of the cloud platform, the running state of the big data platform, the running state of the data access server, the running state of the data micro-service server and the running state of the application server, analyzing and alarming the abnormity and generating an abnormity analysis report.

2. The data analysis application platform system according to claim 1, wherein the system management module further comprises:

the authority management module, system authority management follows south electric wire netting information safety protection management standard, carries out unified authority through province company authority management platform and verifies, and the user authority mainly includes: access rights, editing rights; the user authority is maintained by a System Administrator (SA), and after the user successfully logs in, corresponding operation can be carried out according to the own authority;

the log management module is used for helping the platform operation and maintenance personnel to manage and maintain the platform, and the logs are divided into system operation logs and operation logs; the system operation log is used for automatically generating record information according to configuration when each module of the system operates; the operation log provides records of operations performed by a user through an interface, and the records comprise configuration parameter modification, data addition, deletion, modification, check, import, export, node master-slave switching and the like;

the backup and recovery module is used for providing data backup and recovery functions when data damage and other conditions occur due to accidents of hardware failure, misoperation of an administrator and the like in the operation of the platform system so as to ensure data safety.

3. The data analysis application platform system according to claim 1, wherein the basic resource management platform is an infrastructure platform constructed by an openstack-based cloud platform, a big data platform and a container management platform.

4. The data analysis application platform system according to claim 1, wherein the front-end construction through large-screen presentation comprises: and PC (personal computer) end access is provided for dispatching line loss, dispatching electric quantity monitoring and distribution network operation monitoring, a BS (base station) framework is adopted, micro-service is adopted as a background server, data are provided for a front-end page, and the large-screen display function is realized while the distribution network operation monitoring is carried out.

5. The data analysis application platform system according to claim 1, wherein the data storage module further comprises: the cloud platform time sequence data storage is used for collecting measuring point data and time sequence data aiming at numerical data with time marks, the time sequence database independently stores real-time values and historical values according to data time scale sequences, and IO consumption of data operation is reduced through data compression, file blocking, data time scale indexing and other technologies, so that efficiency is improved.

6. The data analysis application platform system according to claim 2, wherein the backup and recovery module is further configured to perform a full backup and an incremental backup during backup, the full backup is to backup all contents, and the incremental backup is to backup only all changed data since the last backup; and during recovery, the method is also used for recovering the platform mirror image, checking the running condition of the real-time data platform mirror image service, checking the numerical values, the time labels and the data refreshing condition of the same measuring points in the database and judging whether the mirror image software runs well or not.

7. The data analysis application platform system according to claim 1, wherein the virtualized resource management module further comprises a micro service component, the micro service component realizes invocation between micro services through service registration and service discovery, so as to realize decoupling between micro services, and is used for the routing gateway to provide a uniform access entry for the front-end application in the micro service architecture mode, and realize invocation of a target service through a routing policy, so as to simplify that each micro service needs to expose access information to the front-end application; the configuration service provides storage of unified dynamic configuration information under multiple environments for the micro-service, and synchronizes the updated information to the micro-service application in a push-pull mode under each operating environment.

Images