CN107733986A - Support the protection of integrated deployment and monitoring operation big data support platform - Google Patents
Support the protection of integrated deployment and monitoring operation big data support platform Download PDFInfo
- Publication number
- CN107733986A CN107733986A CN201710831332.8A CN201710831332A CN107733986A CN 107733986 A CN107733986 A CN 107733986A CN 201710831332 A CN201710831332 A CN 201710831332A CN 107733986 A CN107733986 A CN 107733986A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- big data
- protection
- monitoring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
- H04L67/1008—Server selection for load balancing based on parameters of servers, e.g. available memory or workload
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses the protection operation big data support platform for supporting integrated deployment and monitoring, including：Multi-source heterogeneous protection service data integration module, distributed big data memory module, distributed big data analysis module, resource management module and monitoring module；Multi-source heterogeneous protection service data integration module; protection operation big data is obtained from external system and protection operational outfit; data pick-up simultaneously converts data to required data structure storage to distributed big data memory module; the analysis result of distributed big data analysis module is written back into distributed big data memory module, and realizes and externally provides data retrieval by unified interface；Distributed big data memory module, store the protection obtained from multi-source heterogeneous protection service data integration module and run big data, the analysis business for distributed big data analysis module provides data basis；Monitoring module, the running situation of cluster and operation is intuitively shown for user, the optimization that performance is calculated for analysis provides support.
Description
Technical field
The present invention relates to big data field, more particularly to support the protection of integrated deployment and monitoring operation big data support
Platform.
Background technology
As examination and repair system, measurement and control integration system are moved in intelligent substation, intelligent electric meter, real-time monitoring system, scene,
And large quantities of construction and application for serving each professional information management system, it is huge quantity will to be produced during Operation of Electric Systems
Greatly, increase quick, various structures, the mass data of source complexity, be typical big data.Therefore recognized by big data technology
Fragile link in electric power networks, power network disaster early warning is carried out in time, the reliability to improving power system, reduction stops on a large scale
The probability of happening of electrification is significant.Management, the operation level of power network itself can be not only lifted, and can be portion of government
Door, industrial quarters and users provide more preferably services, and expanding value-added service for Utilities Electric Co. provides condition.
In protection operation field, the application of existing various big data analysis methods, but systematization, the big data of scale should
With also deploying far away.Particularly, the extensive space-time multidimensional data for monitoring or recording in processing secondary device or system, seeks
When power equipment, facility, the state change rule of system and its mutual association or affecting laws, the processing of mass data
The protection operation of storage, the knowledge excavation of mass data and data-driven needs with the big data application of analysis method etc.
Ask, be also met far away.
In the application process of reality, there is also part urgent problem：It is that environmental structure can only be manually first
Complete, workload is big, and maloperation easily occurs；Next to that in the development process of reality, most need of work code
Complete, write complexity；Meanwhile in cluster running, integration is not accomplished into monitoring, therewith it is caused be to cluster with
The supervision of operation is inadequate, and investigation problem is more difficult, it is impossible to real-time early warning；In addition, current big data application, without fully profit
With cluster resource, that cluster pressure is excessive or the wasting of resources many times be present.Therefore need to establish to support integrated portion
The protection of administration and monitoring operation big data support platform, the solution party that integration is provided is calculated for protection operation big data analysis
Case.
The content of the invention
The purpose of the present invention is exactly to solve the above problems, there is provided supports that the protection of integrated deployment and monitoring operation is big
Data supporting platform, possess multiple and distributing sources is uniformly accessed into ability, includes the distribution for possessing dynamic load leveling ability
Formula data storage, support off-line data to calculate the mixing Computational frame with online task computation adaptive scheduling, realize and possess
Flexibly, the run mode configuration adjustment of convenient automatically dispose and cluster configuration, there is provided cluster operation monitoring, load dynamic evaluation
With automatic configuration Optimizing Suggestions, stable, efficient, easily calculating support is provided for the mass data calculating of polytype business
Environment.
To achieve these goals, the present invention adopts the following technical scheme that：
The protection of integrated deployment and monitoring operation big data support platform is supported, including：Multi-source heterogeneous protection operation number
According to integration module, distributed big data memory module, distributed big data analysis module, resource management module and monitoring module；
Multi-source heterogeneous protection service data integration module, for obtaining protection fortune from external system and protection operational outfit
Row big data, data are extracted and convert data to required data structure storage to distributed big data storage mould
Block, the analysis result of distributed big data analysis module is written back into distributed big data memory module, and realizes and pass through system
One interface externally provides data retrieval.
Distributed big data memory module, for storing the protection obtained from multi-source heterogeneous protection service data integration module
Big data is run, the analysis business for distributed big data analysis module provides data basis；
Distributed big data analysis module is distributed big to being stored in for the operation submitted according to resource management module
Protection operation big data in data memory module is analyzed and processed, and mining data value, analysis result is fed back into user,
And by analysis result storage into distributed big data memory module.
Resource management module, for receiving user's request, resource management and job scheduling are carried out, is supported visual with user
Change interaction.
Monitoring module, by being monitored analysis, realization pair to daily record etc. caused by group operation and resource management module
Each link for running operation performs time, the real-time monitoring using resource, and the operation feelings of cluster and operation are intuitively shown for user
Condition, the optimization that performance is calculated for analysis provide support.
The external system and protection operational outfit, including：Transforming plant protecting and servicing unit, internet, communication port
Managing and control system, protect letter system, energy management system, generalized information system；
The protection operation big data, including：The static attribute data of various kinds of equipment or facility, description equipment, facility or
The data of incidence relation between system, all kinds of real time on-line monitoring data and historical record, the operation maintenance of equipment or facility
(record) data, exception or event of failure process record, environmental data；
The multi-source heterogeneous protection service data integration module, including：Data interface adapter, data pick-up engine, number
According to crawling module, data transformation engine, write back data module；
The data interface adapter, the stipulations of transforming plant protecting and servicing unit gathered data are adapted to for realizing,
And to communication port managing and control system, protect letter system, energy management system, generalized information system interface automatic adaptation, support relationship type
The data pick-up of database and non-relational database, interface is provided to create object using factory mode, and pass through interface
Automatic adaptation realizes that unified interface externally provides data retrieval；
The data pick-up engine, for being extracted by data interface adapter from external system and protection operational outfit
Go out data, the extraction mode of data supports Sqoop, Kettle, OGG and self-defined java classes to extract；For data pick-up task,
Configuration, time control are realized, and supports to extract the configuration and processing of time-out；
The data crawl module, for obtaining required meteorological data from internet by web crawlers, for protection
Run big data analysis and required external data basis is provided；
The data transformation engine, the different pieces of information of module acquisition is crawled by data pick-up engine and data for realizing
The conversion of data structure, tables of data between storehouse and uniformly；Data are cleaned according to the switching strategy of formulation, field unified definition is advised
Draw；
The write back data module, deposited for the data after data transformation engine is handled to be write into distributed big data
Module is stored up, and the analysis result of distributed big data analysis module is written back into distributed big data memory module.
The distributed big data memory module, including：Big data management engine, copy mechanism management module, frame sense
Know mechanism management module, load-balancing mechanism management module and data memory module；
The big data management engine, for realizing the management to big data storing process, improve storage performance and inquiry
Efficiency.Big data management engine perceives mechanism management module, load-balancing mechanism management mould from copy mechanism management module, frame
Block obtains the copy mechanism set according to practical business situation, frame perceives mechanism, load-balancing mechanism, is moved according to copy mechanism
Multiple copies are splitted data into state, mechanism are perceived by Replica placement to corresponding frame according to frame, according to load balancing machine
It will store dynamic or query task be distributed to and load low distributed storage from node, realize to protection operation big data
Storage management；
The copy mechanism management module, for managing the copy amount of data and the flowing mode between copy, it is
Big data management engine provides copy mechanism data and supported；Improve the reliability of data.
The frame perceives mechanism management module, for managing data trnascription Placement Strategy, is carried for big data management engine
Mechanism data is perceived for frame to support；Improve the utilization rate of the reliabilities of data, availability and network bandwidth.
The load-balancing mechanism management module, for managing load-balancing mechanism, provided for big data management engine negative
Carry equilibrating mechanism data to support, so as to realize the dynamically distributes of digital independent or store tasks；Improve reading, storage efficiency.
The data memory module, for receive write back data module protection operation big data to be analyzed to be written and
Analysis result, and the management strategy provided according to big data management engine, realize the storage to protection operation big data.
The data memory module, including：HDFS and HBase, the HBase are frequent in calculating process for being stored in
The data of reading, such as file data and configuration parameter；The HDFS deposits the protection service data for calculating.
The data memory module is made up of distributed storage host node and several distributed storages from node, distribution
Formula storage host node is used for the management strategy for receiving big data management engine, is divided according to strategy to distributed storage from node dynamic
With storage and query task；Distributed storage is used for specific storage protection operation big data from node, according to depositing for practical business
Storage scale, increase and decrease from number of nodes, realize dynamic expansion.
The analyzing and processing uses parallel computation mode, based on MapReduce or Spark big data Computational frames, for
Off-line calculation task and in real time calculating carry out parallelization processing respectively, distribution of computation tasks to multiple working nodes are completed, greatly
Width improves data analysis efficiency.
The distributed big data analysis module, including：Big data operation changing engine, operation issue engine, operation are adjusted
Spend engine, fault tolerant mechanism management module, job timeout's mechanism management module and distributed arithmetic framework；
The big data operation changing engine, the data for receiving resource management module transmission calculate operation, by data
It is the MapReduce operations that can be performed parallel needed for Computational frame to calculate operation changing；
Engine is issued in the operation, for according to MapReduce or Spark operations, operation to be published into offline work automatically
Industry Computational frame MapReduce or online assignment Computational frame Spark；
The job scheduling engine, on the basis of being determined in Computational frame, the computing resource situation according to needed for operation
And in distributed arithmetic framework working node resource service condition, realize the specific distribution of operation.According to appearance in assigning process
The fault tolerant mechanism and job timeout's mechanism that wrong mechanism management module and job timeout's mechanism management module provide ensure Activity Calculation
Reliability；
The fault tolerant mechanism management module, for managing fault tolerant mechanism, fault tolerant mechanism data are provided for job scheduling engine,
The task of malfunctioning node is recovered, improves the reliability of whole calculating process；
Job timeout's mechanism management module, for managing job timeout's mechanism, operation is provided for job scheduling engine
Timeout mechanism data, timely processing is carried out to overtime operation, reduces the Job execution time, ensures operation real-time；
The distributed arithmetic framework, for the processing of specific Activity Calculation, job scheduling engine distributes operation to phase
Several working nodes of Computational frame are answered to realize parallel processing, distributed arithmetic framework is deposited according to work data demand from data
Store up and protection operation big data is obtained in module, analysis result is fed back into user, and analysis result is sent to write back data mould
Block, by the storage of write back data module into data memory module.
The resource management module, including：Operation deployment and management module, task scheduling optimization module, cluster building with
Management module, system resource configuration evaluation module, cluster configuration optimization module；
Operation deployment and management module, operation service request is protected for receiving user, data is formed and calculates operation, send
Big data operation changing engine is given, supports dynamically to carry out task scheduling to operation according to cluster load data, reduces with for the moment
Between produce several wait tasks, avoid cluster due to task accumulate occur it is abnormal；
Task scheduling optimization module, for realizing the timing orderly management in distributed arithmetic framework to calculating task, carry
For the load balance process strategy to calculating task, there is provided unsuccessfully retry management strategy；
Cluster building and management module, for guiding user to realize building and managing for cluster, support the flat of full interface
Platform integrated deployment.Using guide mounting means, the semi-automatic installation of cluster environment is realized, while installation is supported in installation process
Preceding environment measuring, the self-defined installation of selection component, the configuration of cluster important parameter and the configuration of log path；With clustering logic framework
Each module information is shown, while supports the start and stop to component in cluster, supports the dynamic adjustment of collection swarm parameter.
System resource configures evaluation module, for obtaining the resource consumption situation of each operation by test jobs operation,
The required minimalist configuration of whole system operation, allocation optimum are calculated, or cluster loading functional is assessed according to formulating to configure, convenient use
Family is adjusted according to their needs.
Cluster configuration optimization module, for dynamically analyzing current cluster state according to monitoring data, and to beyond valve
The place of value or unreasonable allocation is recorded, while is provided dynamic configuration scheme and fed back to user, assists user more
Intelligence optimizes configuration to cluster.
The monitoring module, including：Cluster resource monitoring module, server load monitoring module, task execution resource prison
Control module, task resource consumption evaluation module, customize task report tool and diagrammatic representation instrument；
The cluster resource monitoring module, for using cluster CPU, internal memory, container number situations monitoring, together
When the monitoring of all kinds of resource consumption situations of server disposed to cluster is provided.
The server load monitoring module, for using CPU, internal memory, network, disk I/O to every server in cluster
Etc. being acquired and monitor.
The task execution resource monitoring module, for the run time of single big rank operation in cluster, use money
Source carries out data acquisition, there is provided real time execution operation uses the monitoring of resource situation.
The task resource consumes evaluation module, for assessing single big rank operation resource consumption situation, feeds back to work
Industry scheduling engine is used to optimize job scheduling.
The customization task report tool, for providing customizable task form, user is set to can configure what needs were shown
Monitoring information.
The diagrammatic representation instrument, for intuitively showing the running situation of cluster and operation for user, based on modularization
Application framework is visualized, a variety of exhibition methods of data is realized, passes through rod figure, curve, word forms mode or multiple combinations side
Formula shows the statistic analysis result of historical data, and shows model data by line chart mode.
The beneficial effects of the invention are as follows：
(1) the platform configuration management at full interface is supported.Using guide mounting means, support environment measuring, component apolegamy,
The flexible definition of parameter configuration, and the run mode adjustment and control of supporting assembly.
(2) access of multi-source heterogeneous data is supported, the adaptation access of mainstream data storage is integrated, possesses perfect data and take out
Take daily record to monitor, and can by possess customize, the adapter growth data access style of heat deployment characteristic.
(3) using the mixing computing architecture for supporting that off-line operation is calculated and online assignment is handled.Platform can be according to operation
Calculating requires type, starts distribution operation to most suitable Activity Calculation framework, different computing architectures can be used unification, can
The algorithms library of extension, meet diversified calculating demand.
(4) the adaptive dynamic load leveling under mass data environment is supported.Build the number based on load balancing between node
According to Placement Strategy and the Data Migration adjustable strategies loaded based on target burst, load condition is classified, realizes task and resource
Reasonable equilibrium.
(5) intelligent cluster configuration optimization is supported.Based on the collection of the resource consumption situation to operation, dynamic analysis is worked as
Preceding cluster state, support according to configuration qualitative assessment cluster load capacity, and dynamic minimum or allocation optimum scheme can be provided
Feedback.
(6) factory mode instead of using factory mode and be generated according to class Class equivalent to the new for creating instance objects
Instance objects, bigger scalability and as far as possible few modification amount can be brought to system.
(7) data are cleaned according to the switching strategy of formulation, field unified definition is planned, realize the correctness, only of data
One property, availability.
Brief description of the drawings
Fig. 1 is the protection operation big data support platform of support integrated deployment proposed by the present invention and monitoring.
Embodiment
The invention will be further described with embodiment below in conjunction with the accompanying drawings.
Fig. 1, it is that the protection for the support integrated deployment and monitoring realized based on the present invention runs big data support platform, should
Platform includes：Multi-source heterogeneous protection service data integration module 1, distributed big data memory module 2, the analysis of distributed big data
Module 3, resource management module 4, monitoring module 5；
The multi-source heterogeneous protection service data integration module 1, including：Data interface adapter 101, data pick-up engine
102nd, data crawl module 103, data transformation engine 104, write back data module 105；
The distributed big data memory module 2, including：Big data supervisor engine 2 01, frame perceive mechanism management module
202nd, copy mechanism management module 203, load-balancing mechanism management module 204, data memory module 205, data memory module
205 include distributed storage host node 2051 and multiple distributed storages from node 2052；
The distributed big data analysis module 3, including：Big data operation changing engine 301, operation issue engine 302,
Job scheduling engine 303, fault tolerant mechanism management module 304, job timeout's mechanism management module 305, distributed arithmetic framework
306；
The resource management module 4, including：Operation deployment and management module 401, task scheduling optimization module 402, cluster
Build and management module 403, system resource configuration evaluation module 404, cluster configuration optimization module 405；
The monitoring module 5, including：Cluster resource monitoring module 501, server load monitoring module 502, tasks carrying
Monitoring resource module 503, task resource consumption evaluation module 504, customize task report tool 505, diagrammatic representation instrument
506。
Multi-source heterogeneous protection service data integration module 1, for being set by platform interface from external system and protection operation
Standby middle obtain protects service data, data is extracted to distributed big data memory module 2, distributed big data is analyzed
The analysis result of module 3 is written back into distributed big data memory module 2, and realizes that externally providing data by unified interface examines
Rope.
Multi-source heterogeneous protection service data integration module 1 by the technologies such as data acquisition, web crawlers, ETL from etc. power transformation
Stand protection and servicing unit, internet, communication port managing and control system, protect letter system, energy management system, generalized information system etc. and obtain
Operation big data is fetched protection, the protection operation big data being related to includes：The static attribute data of various kinds of equipment or facility, description are set
The data of incidence relation between standby, facility or system, all kinds of real time on-line monitoring data and its historical record, equipment or facility
Operation maintenance (record) data, exception or event of failure process record, environmental data etc., and convert data to required number
Distributed big data memory module 2 is arrived according to structure storage.
Multi-source heterogeneous protection service data integration module 1, including：Data interface adapter 101, data pick-up engine 102,
Data crawl module 103, data transformation engine 104, write back data module 105.
The data interface adapter 101, the stipulations of transforming plant protecting and servicing unit gathered data are fitted for realizing
Match somebody with somebody, and the different external information systems such as communication port managing and control system, guarantor's letter system, energy management system, generalized information system are connect
The automatic adaptation of mouth, relevant database and the data pick-up of non-relational database are supported, the use of factory mode is establishment pair
As providing interface, and it can realize that unified interface externally provides data retrieval by the automatic adaptation of interface.Factory mode equivalent to
The new of instance objects is created, instead of using factory mode and instance objects are generated according to class Class, can be brought to system bigger
Scalability and as far as possible few modification amount.
The data pick-up engine 102, for extracting data from external system by data interface adapter 101,
The extraction mode of data supports Sqoop, Kettle, OGG and self-defined java classes to extract.For data pick-up task, realization is matched somebody with somebody
Put, time control, and support to extract the configuration and processing of time-out.
The data crawl module 103, for obtaining the rings such as required weather information from internet by web crawlers
Border data, required external data basis is provided for protection operation big data analysis.
The data transformation engine 104, for realizing that crawling module 103 by data pick-up engine 102 and data obtains
Disparate databases between data structure, tables of data conversion and uniformly.Data are cleaned according to the switching strategy of formulation, to field
Unified definition is planned, realizes the correctness, uniqueness, availability of data.
The write back data module 105 is distributed big for the data after data transformation engine 104 is handled to be write
Data memory module 2, and the analysis result of distributed big data analysis module 3 are written back into distributed big data memory module
In.
Distributed big data memory module 2, treated for storing from what multi-source heterogeneous protection service data integration module obtained
The basic data and analysis result of analysis, the analysis business for distributed big data analysis module provide data basis.
Distributed big data memory module 2, including：Big data management engine, copy mechanism management module, frame perceptron
Management module, load-balancing mechanism management module and data memory module processed.
Big data supervisor engine 2 01, for realizing the management to big data storing process, improve storage performance and inquiry effect
Rate.Big data supervisor engine 2 01 perceives mechanism management module 203, load balancing machine from copy mechanism management module 202, frame
Management module 204 processed obtains the copy mechanism set according to practical business situation, frame perceives mechanism, load-balancing mechanism, root
Multiple copies are dynamically splitted data into according to copy mechanism, mechanism is perceived by Replica placement to corresponding frame, root according to frame
To dynamically be stored according to load-balancing mechanism, the task such as inquiry is distributed to and loads relatively low distributed storage from node, so as to real
The storage management of big data is now run to protection.
The copy mechanism management module 202, for managing the copy amount of data and the flowing mode between copy,
Big data supervisor engine 2 01 is acted on, improves the reliability of data.
The frame perceives mechanism management module 203, for managing data trnascription Placement Strategy, acts on big data management
Engine 201, improve the utilization rate of the reliabilities of data, availability and network bandwidth.
The load-balancing mechanism management module 204, for managing load-balancing mechanism, act on big data management engine
201, so as to the dynamically distributes realized digital independent, stored, improve reading, storage efficiency.
The data memory module 205, for receive the basic data to be analyzed to be written of write back data module 105 and
Analysis result, and the management strategy provided according to big data supervisor engine 2 01, realize the storage to protection operation big data.By
HDFS, HBase are formed, and the data for being written infrequently and taking in calculating process are stored in HBase, such as file data and configuration parameter
Deng；The protection service data for calculating is deposited in HDFS.
Data memory module 205 is by distributed storage host node 2051 and multiple distributed storages from 2052 groups of node
It is used for the management strategy for receiving big data management engine into, distributed storage host node 2051, according to strategy to distributed storage
From the storage of the dynamically distributes of node 2052 and query task；Distributed storage is used for the big number of specific storage protection operation from node 2052
According to can increase and decrease from number of nodes according to the storage size of practical business, realize dynamic expansion.
Distributed big data analysis module 3, for the operation submitted according to resource management module 4, to being stored in distribution
Basic data in big data memory module 2 is analyzed and processed, and mining data value, analysis result is fed back into user, and
Analysis result is stored into distributed big data memory module 2 by write back data module 105.It is extra large using supporting to analyze work
The parallel computing of information real time processing is measured, based on the big data Computational frame such as MapReduce, Spark, for off-line calculation
Task and in real time calculating carry out parallelization processing respectively, and distribution of computation tasks to multiple working nodes is completed, greatly improves number
According to analysis efficiency.
Distributed big data analysis module 3, including big data operation changing engine 301, operation issue engine 302, operation
Scheduling engine 303, fault tolerant mechanism management module 304, job timeout's mechanism management module 305, distributed arithmetic framework 306.
The big data operation changing engine 301, calculated for receiving the data that operation deployment and management module 401 are sent
Operation, it is converted into the computation model needed for Computational frame.
Engine 302 is issued in the operation, and for requiring type according to the calculating of operation, operation is published into distribution automatically
Off-line operation Computational frame MapReduce or online assignment Computational frame Spark in computing framework 306 etc..
The job scheduling engine 303, on the basis of being determined in Computational frame, the computing resource feelings according to needed for operation
The resource service condition of working node in condition and distributed arithmetic framework 306, realized using the operation group scheduling model based on SLA
The specific distribution of operation.There is provided in assigning process according to fault tolerant mechanism management module 304 and job timeout's mechanism management module 305
Fault tolerant mechanism and job timeout mechanism ensure the reliability of Activity Calculation.
The fault tolerant mechanism management module 304, for managing fault tolerant mechanism, job scheduling engine 303 is acted on, to failure
The task of node is recovered, and improves the reliability of whole calculating process.
Job timeout's mechanism management module 305, for managing job timeout's mechanism, acts on job scheduling engine
303, timely processing is carried out to overtime operation, reduces the Job execution time, ensures operation real-time.
The distributed arithmetic framework 306, for the processing of specific Activity Calculation, job scheduling engine 303 divides operation
The multiple working nodes for being assigned to corresponding Computational frame realize parallel processing, and distributed arithmetic framework 306 is according to work data demand
Basic data is obtained from data memory module 205, analysis result is fed back into user, and analysis result is sent to data and returned
Writing module 105, stored by write back data module 105 in data memory module 205.
Resource management module 4, protection operation big data support platform is carried out for receiving user's request resource management and
Job scheduling, support and the virtual interactive interface of user.
Resource management module 4, including operation deployment and management module 401, task scheduling optimization module 402, cluster building
With management module 403, system resource configuration evaluation module 404, cluster configuration optimization module 405.
Operation deployment and management module 401, operation service request is protected for receiving user, data is formed and calculates operation,
Big data operation changing engine 301 is sent to, supports dynamically to carry out task scheduling to operation according to cluster load data, reduces
The same time produces excessive wait task, avoids cluster abnormal due to task accumulation appearance.
Task scheduling optimization module 402, for realizing that the timing in distributed arithmetic framework 306 to calculating task is managed in order
Reason, there is provided to the load balance process strategy of calculating task, there is provided unsuccessfully retry management strategy.
Cluster building and management module 403, for guiding user to realize building and managing for cluster, support full interface
Platform integrated deployment.Using guide mounting means, the semi-automatic installation of cluster environment is realized, while peace is supported in installation process
Environment measuring, self-defined selection component installation, the configuration of cluster important parameter, the configuration of log path etc. before dress；With clustering logic
Framework shows each module information, while supports the start and stop to component in cluster, supports the dynamic adjustment of collection swarm parameter.
System resource configures evaluation module 404, for obtaining the resource consumption feelings of each operation by test jobs operation
Condition, the required minimalist configuration of whole system operation, allocation optimum are calculated, or cluster loading functional is assessed according to formulating to configure, side
Just user is adjusted according to their needs.
Cluster configuration optimization module 405, for analyzing current cluster state according to monitoring data come dynamic, and to beyond
The place of threshold values or unreasonable allocation is recorded, while is provided dynamic configuration scheme and fed back to user, assists user
More intelligent optimizes configuration to cluster.
Monitoring module 5, by being monitored analysis to daily record caused by group operation and resource management module, realize to fortune
Each link of row operation performs time, the real-time monitoring using resource, based on visualization technique be user intuitively show cluster and
The running situation of operation, the optimization that performance can be calculated for analysis provide support.
Monitoring module 5, including cluster resource monitoring module 501, server load monitoring module 502, task execution resource
Monitoring module 503, task resource consumption evaluation module 504, customize task report tool 505, diagrammatic representation instrument 506.
The cluster resource monitoring module 501, for platform cluster using CPU, internal memory, container numbers situations such as
Monitoring, while provide the monitoring for all kinds of resource consumption situations of server disposed to cluster.
The server load monitoring module 502, for using CPU, internal memory, net to every server in platform cluster
Network, disk I/O etc. are acquired and monitored.
The task execution resource monitoring module 503, for the run time of single big rank operation, use in cluster
Resource carries out data acquisition, there is provided real time execution operation uses the monitoring of resource situation.
The task resource consumes evaluation module 504, for assessing single big rank operation resource consumption situation, feeds back to
Job scheduling engine 303 is used to optimize job scheduling.
The customization task report tool 505, for providing customizable task form, make user is configurable to need to show
Monitoring information.
The diagrammatic representation instrument 506, for intuitively showing the running situation of cluster and operation for user, based on modularization
Visualization application framework, realize a variety of exhibition methods of data, such as a variety of sides such as rod figure, curve, word form can be passed through
Formula or multiple combinations mode show the statistic analysis result of historical data, and show model data by modes such as line charts.
Although above-mentioned the embodiment of the present invention is described with reference to accompanying drawing, model not is protected to the present invention
The limitation enclosed, one of ordinary skill in the art should be understood that on the basis of technical scheme those skilled in the art are not
Need to pay various modifications or deformation that creative work can make still within protection scope of the present invention.
Claims (10)
1. the protection of integrated deployment and monitoring operation big data support platform is supported, it is characterized in that, including：Multi-source heterogeneous protection
Service data integration module, distributed big data memory module, distributed big data analysis module, resource management module and monitoring
Module；
Multi-source heterogeneous protection service data integration module is big for obtaining protection operation from external system and protection operational outfit
Data, data are extracted and convert data to required data structure storage to distributed big data memory module, will
The analysis result of distributed big data analysis module is written back into distributed big data memory module, and is realized and passed through unified interface
Data retrieval is externally provided；
Distributed big data memory module, for storing the protection operation obtained from multi-source heterogeneous protection service data integration module
Big data, the analysis business for distributed big data analysis module provide data basis；
Distributed big data analysis module, for the operation submitted according to resource management module, to being stored in distributed big data
Protection operation big data in memory module is analyzed and processed, mining data value, and analysis result is fed back into user, and will
Analysis result is stored into distributed big data memory module；
Resource management module, for receiving user's request, resource management and job scheduling are carried out, support and the visualization of user are handed over
Mutually；
Monitoring module, by being monitored analysis to daily record etc. caused by group operation and resource management module, realize to operation
Each link of operation performs time, the real-time monitoring using resource, and the running situation of cluster and operation is intuitively shown for user, is
The optimization that analysis calculates performance provides support.
2. the protection of integrated deployment and monitoring operation big data support platform is supported as claimed in claim 1, it is characterized in that,
The external system and protection operational outfit, including：Transforming plant protecting and servicing unit, internet, communication port managing and control system,
Protect letter system, energy management system, generalized information system.
3. the protection of integrated deployment and monitoring operation big data support platform is supported as claimed in claim 1, it is characterized in that,
The protection operation big data, including：The static attribute data of various kinds of equipment or facility, description equipment, facility or system
Between incidence relation data, all kinds of real time on-line monitoring data and historical record, the operation maintenance data of equipment or facility,
Exception or event of failure process record, environmental data.
4. the protection of integrated deployment and monitoring operation big data support platform is supported as claimed in claim 1, it is characterized in that,
The multi-source heterogeneous protection service data integration module, including：Data interface adapter, data pick-up engine, data crawl mould
Block, data transformation engine, write back data module；
The data interface adapter, the stipulations of transforming plant protecting and servicing unit gathered data are adapted to for realizing, and
To communication port managing and control system, protect letter system, energy management system, generalized information system interface automatic adaptation, support relational data
Storehouse and the data pick-up of non-relational database, interface is provided to create object using factory mode, and pass through the automatic of interface
Adaptation realizes that unified interface externally provides data retrieval；
The data pick-up engine, for extracting number from external system and protection operational outfit by data interface adapter
According to the extraction mode of data supports Sqoop, Kettle, OGG and self-defined java classes to extract；For data pick-up task, realize
Configuration, time control, and support to extract the configuration and processing of time-out；
The data crawl module, for obtaining required meteorological data from internet by web crawlers, are run for protection
Big data analysis provides required external data basis；
The data transformation engine, for realize by data pick-up engine and data crawl module acquisition disparate databases between
Data structure, tables of data conversion and uniformly；Data are cleaned according to the switching strategy of formulation, field unified definition is planned；
The write back data module, mould is stored for the data after data transformation engine is handled to be write into distributed big data
Block, and the analysis result of distributed big data analysis module are written back into distributed big data memory module.
5. the protection of integrated deployment and monitoring operation big data support platform is supported as claimed in claim 1, it is characterized in that,
The distributed big data memory module, including：Big data management engine, copy mechanism management module, frame perceptron tubulation
Manage module, load-balancing mechanism management module and data memory module；
The big data management engine, for realizing the management to big data storing process, improve storage performance and search efficiency；
Big data management engine is from copy mechanism management module, frame perceives mechanism management module, load-balancing mechanism management module obtains
The copy mechanism set according to practical business situation, frame is taken to perceive mechanism, load-balancing mechanism, according to copy mechanism dynamically
Multiple copies are splitted data into, mechanism is perceived by Replica placement to corresponding frame according to frame, moved according to load-balancing mechanism
It will store to state or query task be distributed to and load low distributed storage from node, realize the storage to protection operation big data
Management；
The copy mechanism management module, for managing the copy amount of data and the flowing mode between copy, for big number
Copy mechanism data is provided according to management engine to support；
The frame perceives mechanism management module, and for managing data trnascription Placement Strategy, machine is provided for big data management engine
Frame perceives mechanism data and supported；
The load-balancing mechanism management module, for managing load-balancing mechanism, it is equal to provide load for big data management engine
The mechanism data that weighs is supported, so as to realize the dynamically distributes of digital independent or store tasks；
The data memory module, for receiving write back data module protection operation big data to be analyzed to be written and analysis
As a result, the management strategy and according to big data management engine provided, realize the storage to protection operation big data.
6. the protection of integrated deployment and monitoring operation big data support platform is supported as claimed in claim 5, it is characterized in that,
The data memory module, including：HDFS and HBase, the HBase are used to be stored in the number for being written infrequently and taking in calculating process
According to such as file data and configuration parameter；The HDFS deposits the protection service data for calculating；
The data memory module is made up of distributed storage host node and several distributed storages from node, and distribution is deposited
Storage host node is used for the management strategy for receiving big data management engine, is deposited according to strategy to distributed storage from node dynamically distributes
Storage and query task；Distributed storage is used for specific storage protection operation big data from node, is advised according to the storage of practical business
Mould, increase and decrease from number of nodes, realize dynamic expansion；
The analyzing and processing uses parallel computation mode, based on MapReduce or Spark big data Computational frames, for offline
Calculating task and in real time calculating carry out parallelization processing respectively, and distribution of computation tasks to multiple working nodes is completed.
7. the protection of integrated deployment and monitoring operation big data support platform is supported as claimed in claim 1, it is characterized in that,
The distributed big data analysis module, including：Big data operation changing engine, operation issue engine, job scheduling engine, appearance
Wrong mechanism management module, job timeout's mechanism management module and distributed arithmetic framework；
The big data operation changing engine, the data for receiving resource management module transmission calculate operation, data are calculated
Operation changing is MapReduce operations；
Engine is issued in the operation, for according to MapReduce or Spark operations, operation to be published into off-line operation meter automatically
Calculate framework MapReduce or online assignment Computational frame Spark；
The job scheduling engine, on the basis of being determined in Computational frame, according to needed for operation computing resource situation and point
The resource service condition of working node in cloth computing framework, realizes the specific distribution of operation；According to fault-tolerant machine in assigning process
What the fault tolerant mechanism and job timeout's mechanism that management module processed and job timeout mechanism management module provide ensured Activity Calculation can
By property；
The fault tolerant mechanism management module, for managing fault tolerant mechanism, fault tolerant mechanism data are provided for job scheduling engine, pair event
The task of barrier node is recovered, and improves the reliability of whole calculating process；
Job timeout's mechanism management module, for managing job timeout's mechanism, job timeout is provided for job scheduling engine
Mechanism data, timely processing is carried out to overtime operation, reduces the Job execution time, ensures operation real-time；
The distributed arithmetic framework, for the processing of specific Activity Calculation, job scheduling engine distributes operation to mutually accrued
Several working nodes for calculating framework realize parallel processing, and distributed arithmetic framework is according to work data demand from data storage mould
Protection operation big data is obtained in block, analysis result is fed back into user, and analysis result is sent to write back data module, is led to
The storage of write back data module is crossed into data memory module.
8. the protection of integrated deployment and monitoring operation big data support platform is supported as claimed in claim 1, it is characterized in that,
The resource management module, including：Operation deployment and management module, task scheduling optimization module, cluster building and management module,
System resource configuration evaluation module, cluster configuration optimization module；
Operation deployment and management module, operation service request is protected for receiving user, data is formed and calculates operation, be sent to big
Data operation transform engine, support dynamically to carry out task scheduling to operation according to cluster load data, reduce same time production
Raw several wait task, avoid cluster from being accumulated due to task and exception occur；
Task scheduling optimization module, for realizing the timing orderly management in distributed arithmetic framework to calculating task, there is provided right
The load balance process strategy of calculating task, there is provided unsuccessfully retry management strategy；
Cluster building and management module, for guiding user to realize building and managing for cluster, support the platform one at full interface
Bodyization is disposed；Using guide mounting means, the semi-automatic installation of cluster environment is realized, while installation front ring is supported in installation process
Border detection, the self-defined installation of selection component, the configuration of cluster important parameter and the configuration of log path；Shown with clustering logic framework
Each module information, while the start and stop to component in cluster are supported, support the dynamic adjustment of collection swarm parameter；
System resource configures evaluation module, for obtaining the resource consumption situation of each operation by test jobs operation, calculates
Minimalist configuration, allocation optimum needed for whole system operation, or cluster loading functional is assessed according to formulating to configure, facilitate user's root
Need to be adjusted according to itself；
Cluster configuration optimization module, for analyzing current cluster state according to monitoring data come dynamic, and to beyond threshold values or
The place of unreasonable allocation is recorded, while is provided dynamic configuration scheme and fed back to user, assists user more intelligent
Configuration is optimized to cluster.
9. the protection of integrated deployment and monitoring operation big data support platform is supported as claimed in claim 1, it is characterized in that,
The monitoring module, including：Cluster resource monitoring module, server load monitoring module, task execution resource monitoring module, appoint
Business resource consumption evaluation module, customize task report tool and diagrammatic representation instrument；
The cluster resource monitoring module, for using cluster CPU, internal memory, container number situations monitoring, carry simultaneously
For the monitoring for all kinds of resource consumption situations of server disposed to cluster；
The server load monitoring module, for being entered to every server in cluster using CPU, internal memory, network, disk I/O etc.
Row collection and monitoring；
The task execution resource monitoring module, for entering to the run time of single big rank operation in cluster, using resource
Row data acquisition, there is provided real time execution operation uses the monitoring of resource situation；
The task resource consumes evaluation module, for assessing single big rank operation resource consumption situation, feeds back to operation tune
Degree engine is used to optimize job scheduling；
The customization task report tool, for providing customizable task form, user is set to can configure the monitoring that needs are shown
Information.
10. the protection of integrated deployment and monitoring operation big data support platform, its feature are supported as claimed in claim 9
It is,
The diagrammatic representation instrument, for intuitively showing the running situation of cluster and operation for user, based on the visual of modularization
Change application framework, realize a variety of exhibition methods of data, pass through rod figure, curve, word forms mode or multiple combinations mode exhibition
The statistic analysis result of existing historical data, and model data is shown by line chart mode.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710831332.8A CN107733986B (en) | 2017-09-15 | 2017-09-15 | Protection operation big data supporting platform supporting integrated deployment and monitoring |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710831332.8A CN107733986B (en) | 2017-09-15 | 2017-09-15 | Protection operation big data supporting platform supporting integrated deployment and monitoring |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107733986A true CN107733986A (en) | 2018-02-23 |
| CN107733986B CN107733986B (en) | 2021-01-26 |
Family
ID=61207192
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710831332.8A Expired - Fee Related CN107733986B (en) | 2017-09-15 | 2017-09-15 | Protection operation big data supporting platform supporting integrated deployment and monitoring |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107733986B (en) |
Cited By (37)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108446335A (en) * | 2018-02-27 | 2018-08-24 | 广州港数据科技有限公司 | Heterogeneous system data pick-up based on database and unified outbound data exchange method |
| CN108647265A (en) * | 2018-04-28 | 2018-10-12 | 新疆熙菱信息技术股份有限公司 | Based on multiple platform data interactive formula system |
| CN108846076A (en) * | 2018-06-08 | 2018-11-20 | 山大地纬软件股份有限公司 | The massive multi-source ETL process method and system of supporting interface adaptation |
| CN108985981A (en) * | 2018-06-28 | 2018-12-11 | 北京奇虎科技有限公司 | Data processing system and method |
| CN109117285A (en) * | 2018-07-27 | 2019-01-01 | 高新兴科技集团股份有限公司 | Support the distributed memory computing cluster system of high concurrent |
| CN109254989A (en) * | 2018-08-27 | 2019-01-22 | 北京东软望海科技有限公司 | A kind of method and device of the elastic ETL architecture design based on metadata driven |
| CN109656993A (en) * | 2018-12-05 | 2019-04-19 | 贵州电网有限责任公司 | Mass data storage and computing system under a kind of distributed environment |
| CN109688068A (en) * | 2019-02-03 | 2019-04-26 | 辽宁邮电规划设计院有限公司 | Network load balancing method and device based on big data analysis |
| CN109739925A (en) * | 2019-01-07 | 2019-05-10 | 北京云基数技术有限公司 | A kind of data processing system and method based on big data |
| CN110187869A (en) * | 2019-05-14 | 2019-08-30 | 上海直真君智科技有限公司 | Unified inter-operation system and method between a kind of big data isomery storage computation model |
| CN110533436A (en) * | 2019-09-05 | 2019-12-03 | 深圳市携众通科技有限公司 | A kind of method of multisystem work order data fusion |
| CN110690984A (en) * | 2018-07-05 | 2020-01-14 | 上海宝信软件股份有限公司 | Spark-based big data weblog acquisition, analysis and early warning method and system |
| CN110764747A (en) * | 2019-10-22 | 2020-02-07 | 南方电网科学研究院有限责任公司 | Data calculation scheduling method based on Airflow |
| CN111078396A (en) * | 2019-11-22 | 2020-04-28 | 厦门安胜网络科技有限公司 | Distributed data access method and system based on multitask instances |
| CN111143651A (en) * | 2019-12-23 | 2020-05-12 | 安徽海豚新媒体产业发展有限公司 | New media integration operation data acquisition analysis system for management |
| CN111177205A (en) * | 2019-12-31 | 2020-05-19 | 重庆中电自能科技有限公司 | New energy station data sharing method and system |
| CN111309549A (en) * | 2020-02-03 | 2020-06-19 | 北京字节跳动网络技术有限公司 | Monitoring method, system, readable medium and electronic device |
| CN111435344A (en) * | 2019-01-15 | 2020-07-21 | 中国石油集团川庆钻探工程有限公司长庆钻井总公司 | Big data-based drilling acceleration influence factor analysis model |
| CN111596950A (en) * | 2020-05-15 | 2020-08-28 | 博易智软(北京)技术有限公司 | Distributed data development engine system |
| CN111696589A (en) * | 2020-06-04 | 2020-09-22 | 华夏吉泰(北京)科技有限公司 | Mass seismic data storage equipment and seismic professional data management system |
| CN111797297A (en) * | 2020-09-09 | 2020-10-20 | 平安国际智慧城市科技股份有限公司 | Page data processing method and device, computer equipment and storage medium |
| CN111866038A (en) * | 2019-04-25 | 2020-10-30 | 华东计算技术研究所（中国电子科技集团公司第三十二研究所） | Distributed storage dynamic defense system and method based on heterogeneous multiple copies |
| CN112104751A (en) * | 2020-11-10 | 2020-12-18 | 中国电力科学研究院有限公司 | Method, device and system for processing regulation and control cloud data |
| CN112148718A (en) * | 2020-10-28 | 2020-12-29 | 云赛智联股份有限公司 | Big data support management system for city-level data middling station |
| CN112261108A (en) * | 2020-10-16 | 2021-01-22 | 江苏奥工信息技术有限公司 | Cluster management platform based on big data sharing service |
| CN112307396A (en) * | 2020-10-21 | 2021-02-02 | 五凌电力有限公司 | Platform architecture based on multi-engine data modeling calculation analysis and processing method thereof |
| CN112333036A (en) * | 2021-01-06 | 2021-02-05 | 广州技象科技有限公司 | Multi-storage-node-based power Internet of things configuration data backup method and device |
| CN112381501A (en) * | 2020-11-05 | 2021-02-19 | 上海汇付数据服务有限公司 | Product operation platform system |
| CN112693502A (en) * | 2019-10-23 | 2021-04-23 | 上海宝信软件股份有限公司 | Urban rail transit monitoring system and method based on big data architecture |
| CN112818045A (en) * | 2021-01-22 | 2021-05-18 | 辽宁长江智能科技股份有限公司 | Data access unified management platform for big data |
| CN112860776A (en) * | 2021-01-20 | 2021-05-28 | 山东众阳健康科技集团有限公司 | Method and system for extracting and scheduling various data |
| CN112905732A (en) * | 2019-11-19 | 2021-06-04 | 成都长城开发科技有限公司 | Method and device for acquiring reading success rate of electric meter |
| CN113347170A (en) * | 2021-05-27 | 2021-09-03 | 北京计算机技术及应用研究所 | Intelligent analysis platform design method based on big data framework |
| CN113535157A (en) * | 2021-09-16 | 2021-10-22 | 中国电子科技集团公司第十五研究所 | Heterogeneous big data resource encapsulation integration system and method capable of being plugged and unplugged during operation |
| CN113986907A (en) * | 2021-12-24 | 2022-01-28 | 深圳市明源云科技有限公司 | Graphic maintenance method, device and equipment for abnormal data and storage medium |
| CN116089518A (en) * | 2023-04-07 | 2023-05-09 | 广州思迈特软件有限公司 | Data model extraction method and system, terminal and medium |
| CN112381501B (en) * | 2020-11-05 | 2024-06-07 | 上海汇付支付有限公司 | Product operation platform system |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104391989A (en) * | 2014-12-16 | 2015-03-04 | 浪潮电子信息产业股份有限公司 | Distributed ETL all-in-one machine system |
-
2017
- 2017-09-15 CN CN201710831332.8A patent/CN107733986B/en not_active Expired - Fee Related
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104391989A (en) * | 2014-12-16 | 2015-03-04 | 浪潮电子信息产业股份有限公司 | Distributed ETL all-in-one machine system |
Cited By (45)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108446335A (en) * | 2018-02-27 | 2018-08-24 | 广州港数据科技有限公司 | Heterogeneous system data pick-up based on database and unified outbound data exchange method |
| CN108647265A (en) * | 2018-04-28 | 2018-10-12 | 新疆熙菱信息技术股份有限公司 | Based on multiple platform data interactive formula system |
| CN108846076A (en) * | 2018-06-08 | 2018-11-20 | 山大地纬软件股份有限公司 | The massive multi-source ETL process method and system of supporting interface adaptation |
| CN108985981A (en) * | 2018-06-28 | 2018-12-11 | 北京奇虎科技有限公司 | Data processing system and method |
| CN108985981B (en) * | 2018-06-28 | 2021-04-23 | 北京奇虎科技有限公司 | Data processing system and method |
| CN110690984A (en) * | 2018-07-05 | 2020-01-14 | 上海宝信软件股份有限公司 | Spark-based big data weblog acquisition, analysis and early warning method and system |
| CN109117285A (en) * | 2018-07-27 | 2019-01-01 | 高新兴科技集团股份有限公司 | Support the distributed memory computing cluster system of high concurrent |
| CN109254989A (en) * | 2018-08-27 | 2019-01-22 | 北京东软望海科技有限公司 | A kind of method and device of the elastic ETL architecture design based on metadata driven |
| CN109656993A (en) * | 2018-12-05 | 2019-04-19 | 贵州电网有限责任公司 | Mass data storage and computing system under a kind of distributed environment |
| CN109739925A (en) * | 2019-01-07 | 2019-05-10 | 北京云基数技术有限公司 | A kind of data processing system and method based on big data |
| CN111435344A (en) * | 2019-01-15 | 2020-07-21 | 中国石油集团川庆钻探工程有限公司长庆钻井总公司 | Big data-based drilling acceleration influence factor analysis model |
| CN109688068A (en) * | 2019-02-03 | 2019-04-26 | 辽宁邮电规划设计院有限公司 | Network load balancing method and device based on big data analysis |
| CN111866038A (en) * | 2019-04-25 | 2020-10-30 | 华东计算技术研究所（中国电子科技集团公司第三十二研究所） | Distributed storage dynamic defense system and method based on heterogeneous multiple copies |
| CN110187869A (en) * | 2019-05-14 | 2019-08-30 | 上海直真君智科技有限公司 | Unified inter-operation system and method between a kind of big data isomery storage computation model |
| CN110533436A (en) * | 2019-09-05 | 2019-12-03 | 深圳市携众通科技有限公司 | A kind of method of multisystem work order data fusion |
| CN110764747A (en) * | 2019-10-22 | 2020-02-07 | 南方电网科学研究院有限责任公司 | Data calculation scheduling method based on Airflow |
| CN112693502A (en) * | 2019-10-23 | 2021-04-23 | 上海宝信软件股份有限公司 | Urban rail transit monitoring system and method based on big data architecture |
| CN112905732A (en) * | 2019-11-19 | 2021-06-04 | 成都长城开发科技有限公司 | Method and device for acquiring reading success rate of electric meter |
| CN111078396A (en) * | 2019-11-22 | 2020-04-28 | 厦门安胜网络科技有限公司 | Distributed data access method and system based on multitask instances |
| CN111078396B (en) * | 2019-11-22 | 2023-12-19 | 厦门安胜网络科技有限公司 | Distributed data access method and system based on multitasking examples |
| CN111143651A (en) * | 2019-12-23 | 2020-05-12 | 安徽海豚新媒体产业发展有限公司 | New media integration operation data acquisition analysis system for management |
| CN111143651B (en) * | 2019-12-23 | 2023-11-17 | 安徽海豚新媒体产业发展有限公司 | Data acquisition and analysis system for new media integrated operation management |
| CN111177205B (en) * | 2019-12-31 | 2023-04-21 | 重庆中电自能科技有限公司 | New energy station data sharing method and system |
| CN111177205A (en) * | 2019-12-31 | 2020-05-19 | 重庆中电自能科技有限公司 | New energy station data sharing method and system |
| CN111309549A (en) * | 2020-02-03 | 2020-06-19 | 北京字节跳动网络技术有限公司 | Monitoring method, system, readable medium and electronic device |
| CN111309549B (en) * | 2020-02-03 | 2023-04-21 | 北京字节跳动网络技术有限公司 | Monitoring method, monitoring system, readable medium and electronic equipment |
| CN111596950A (en) * | 2020-05-15 | 2020-08-28 | 博易智软(北京)技术有限公司 | Distributed data development engine system |
| CN111696589A (en) * | 2020-06-04 | 2020-09-22 | 华夏吉泰(北京)科技有限公司 | Mass seismic data storage equipment and seismic professional data management system |
| CN111797297B (en) * | 2020-09-09 | 2020-12-15 | 平安国际智慧城市科技股份有限公司 | Page data processing method and device, computer equipment and storage medium |
| CN111797297A (en) * | 2020-09-09 | 2020-10-20 | 平安国际智慧城市科技股份有限公司 | Page data processing method and device, computer equipment and storage medium |
| CN112261108A (en) * | 2020-10-16 | 2021-01-22 | 江苏奥工信息技术有限公司 | Cluster management platform based on big data sharing service |
| CN112307396A (en) * | 2020-10-21 | 2021-02-02 | 五凌电力有限公司 | Platform architecture based on multi-engine data modeling calculation analysis and processing method thereof |
| CN112148718A (en) * | 2020-10-28 | 2020-12-29 | 云赛智联股份有限公司 | Big data support management system for city-level data middling station |
| CN112381501B (en) * | 2020-11-05 | 2024-06-07 | 上海汇付支付有限公司 | Product operation platform system |
| CN112381501A (en) * | 2020-11-05 | 2021-02-19 | 上海汇付数据服务有限公司 | Product operation platform system |
| CN112104751B (en) * | 2020-11-10 | 2021-02-12 | 中国电力科学研究院有限公司 | Method, device and system for processing regulation and control cloud data |
| CN112104751A (en) * | 2020-11-10 | 2020-12-18 | 中国电力科学研究院有限公司 | Method, device and system for processing regulation and control cloud data |
| CN112333036A (en) * | 2021-01-06 | 2021-02-05 | 广州技象科技有限公司 | Multi-storage-node-based power Internet of things configuration data backup method and device |
| CN112860776A (en) * | 2021-01-20 | 2021-05-28 | 山东众阳健康科技集团有限公司 | Method and system for extracting and scheduling various data |
| CN112860776B (en) * | 2021-01-20 | 2022-12-06 | 众阳健康科技集团有限公司 | Method and system for extracting and scheduling various data |
| CN112818045A (en) * | 2021-01-22 | 2021-05-18 | 辽宁长江智能科技股份有限公司 | Data access unified management platform for big data |
| CN113347170A (en) * | 2021-05-27 | 2021-09-03 | 北京计算机技术及应用研究所 | Intelligent analysis platform design method based on big data framework |
| CN113535157A (en) * | 2021-09-16 | 2021-10-22 | 中国电子科技集团公司第十五研究所 | Heterogeneous big data resource encapsulation integration system and method capable of being plugged and unplugged during operation |
| CN113986907A (en) * | 2021-12-24 | 2022-01-28 | 深圳市明源云科技有限公司 | Graphic maintenance method, device and equipment for abnormal data and storage medium |
| CN116089518A (en) * | 2023-04-07 | 2023-05-09 | 广州思迈特软件有限公司 | Data model extraction method and system, terminal and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107733986B (en) | 2021-01-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107733986A (en) | Support the protection of integrated deployment and monitoring operation big data support platform | |
| CN105005570B (en) | Magnanimity intelligent power data digging method and device based on cloud computing | |
| CN104281130B (en) | Hydroelectric equipment monitoring and fault diagnosis system based on big data technology | |
| CN102882969B (en) | A kind of safety production cloud service platform of industrial and mining enterprises | |
| CN111077870A (en) | Intelligent OPC data real-time acquisition and monitoring system and method based on stream calculation | |
| CN110047014A (en) | A kind of user's electricity data restorative procedure based on load curve and history electricity | |
| CN104050042B (en) | The resource allocation methods and device of ETL operations | |
| CN102917032B (en) | A kind of safety production cloud service platform of industrial and mining enterprises | |
| CN106502772A (en) | Electric quantity data batch high speed processing method and system based on distributed off-line technology | |
| CN104915793A (en) | Public information intelligent analysis platform based on big data analysis and mining | |
| CN107103064B (en) | Data statistical method and device | |
| CN105843182A (en) | Power dispatching accident handling scheme preparing system and power dispatching accident handling scheme preparing method based on OMS | |
| CN114416855A (en) | Visualization platform and method based on electric power big data | |
| KR20150112357A (en) | Sensor data processing system and method thereof | |
| CN102903011A (en) | Mass data processing system used for safety production cloud service platform facing industrial and mining enterprises | |
| CN102880802A (en) | Fatal danger fountainhead analysis and evaluation method for safety production cloud service platform system facing industrial and mining enterprises | |
| CN109739919A (en) | A kind of front end processor and acquisition system for electric system | |
| CN102903010A (en) | Support vector machine-based abnormal judgment method for safety production cloud service platform orientating industrial and mining enterprises | |
| CN101615176B (en) | Data monitoring system and method for realizing same | |
| CN106202566A (en) | A kind of magnanimity electricity consumption data mixing based on big data storage system and method | |
| CN102930372A (en) | Data analysis method for association rule of cloud service platform system orienting to safe production of industrial and mining enterprises | |
| CN109977125A (en) | A kind of big data safety analysis plateform system based on network security | |
| EP1993016B1 (en) | Embedded historians with data aggregator | |
| CN112884452A (en) | Intelligent operation and maintenance multi-source data acquisition visualization analysis system | |
| CN101196901B (en) | Computer system and method for database query |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20210126 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |