CN109344199A - The system and method for big data quantity distributed treatment is realized in cloud computing platform - Google Patents
The system and method for big data quantity distributed treatment is realized in cloud computing platform Download PDFInfo
- Publication number
- CN109344199A CN109344199A CN201811101541.8A CN201811101541A CN109344199A CN 109344199 A CN109344199 A CN 109344199A CN 201811101541 A CN201811101541 A CN 201811101541A CN 109344199 A CN109344199 A CN 109344199A
- Authority
- CN
- China
- Prior art keywords
- data
- engine subsystem
- module
- monitoring
- cloud computing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The present invention relates to the systems that big data quantity distributed treatment is realized in a kind of cloud computing platform, wherein the system includes data screening processing engine subsystem, data relation analysis engine subsystem and data monitoring O&M engine subsystem, the data screening processing engine subsystem is connected with the data relation analysis engine subsystem and data monitoring O&M engine subsystem respectively, and the data relation analysis engine subsystem is connected with data monitoring O&M engine subsystem.Using this kind of structure, business development personnel are when realizing related needs of the enterprise for big data quantity, data processing speed can be effectively promoted with the method in the present invention, and it can be according to enterprise demand, not only flexibly formulate the verification rule of data, distributed deployment can also be carried out at any time according to portfolio simultaneously, optimize the processing engine of data, improve the practicability of system.
Description
Technical field
The present invention relates to computer application field, in particular to the analysis classes system regions of high-throughput stream data, tools
Body refers to the system and method that big data quantity distributed treatment is realized in a kind of cloud computing platform.
Background technique
With the innovation and application popularization of the generation information technologies such as cloud computing, mobile Internet and Internet of Things, Ren Leizao
Big data era has been entered, more and more enterprises need to assist enterprise to make decisions using the analysis for big data,
And it is different from the past only just with experience and intuition.Therefore, enterprise has the analysis processing of big data quantity very big
Demand.
Distributed data processing system (Distributed Data Stream Management System, DDSMS) is
The system that distributed treatment can be carried out to data.DDSMS can greatly shorten data processing time, improve response speed,
There is extremely extensive purposes in real life.In general, a DDSMS has function below:
(1) due to the requirement of the limitation of amount of physical memory and treatment effeciency, when carrying out online processing to data stream, generally
Scan data one time；
(2) within the regular hour, data can be ranked up, makes unordered to become orderly；
(3) for a user, the programming of traditional DDSMS has user well in real time to the inquiry of data
Property；
(4) traditional DDSMS processing data in, when the data volume for encountering data flow is huge be more than system carrying energy
When power, some data are removed at random or selectively to alleviate the expansion of system data；
(5) traditional DDSMS also proposed some requirements to the processing of abnormal data, first have to rapidly, while will be in accordance with
It requires in real time；
(6) timely the interface of data user can provide convenient data information inquiry for user.
Big data united analysis processing method based on cloud computing, can be by magnanimity structuring, unstructured and half structure
Change the processing of data distribution formula, the query analysis request of isomeric data can be parsed, according to the data object position of query analysis
It dispatches data processing to calculate, Data Analysis Services calculating is distributed on each data memory node, realizes the parallel of isomeric data
Analysis processing, providing universal data access interface has the complexity and challenge for overcoming big data analysis processing, meets big
The scale of data processing constantly increases and the advantages that real-time demand.
, cannot will effectively will wherein however, although the data of different structure can be carried out high speed processing by the technology
Some related data associate, while high speed processing mass data, if can will wherein related data correlation
Get up, can more meet the needs of enterprise.
Summary of the invention
The purpose of the present invention is overcoming the above-mentioned prior art, provide it is a kind of processing mass data energy according to
The business demand of enterprise is associated data, system architecture is simple, system compatibility is relatively strong, stable and reliable working performance, fits
With the system and method for realizing big data quantity distributed treatment in the relatively broad cloud computing platform of range.
To achieve the goals above, system and the side of big data quantity distributed treatment are realized in cloud computing platform of the invention
Method is as follows:
The system that big data quantity distributed treatment is realized in the cloud computing platform, is mainly characterized by, the system packet
Include data screening processing engine subsystem, data relation analysis engine subsystem and data monitoring O&M engine subsystem, the number
It is equal with the data relation analysis engine subsystem and data monitoring O&M engine subsystem respectively according to Screening Treatment engine subsystem
It is connected, and the data relation analysis engine subsystem is connected with data monitoring O&M engine subsystem；
The data screening handles engine subsystem
Data validation correction verification module is connected with the data monitoring O&M engine subsystem, and for the number to input
According to progress validity checking；
Data persistence processing module is connected with the data monitoring O&M engine subsystem, and for that will pass through the number
Relevant business object is converted to according to the data of legitimacy verifies module validity checking；
Data classification processing module, respectively with the data relation analysis engine subsystem and data monitoring O&M engine subsystem
System is connected, and the business object for that will be generated by the business object of the data persistence processing module according to data into
Row classification, is separately sent to data relation analysis engine subsystem for different business objects；
Data backup module is connected, for drawing the data relation analysis with the data monitoring O&M engine subsystem
The data that subsystem receives are held up to be backed up.
The data relation analysis engine subsystem includes:
Data service model checking module handles engine subsystem and data monitoring O&M engine with the data screening respectively
Subsystem is connected, and verifies for that will handle the received data of engine subsystem from data screening, filters out and meet pass
The data of bracing part；
Data service relating module is connected, for that will pass through data service with the data monitoring O&M engine subsystem
The data of model checking module verification are handled.
The data monitoring O&M engine subsystem includes:
Data processing exception processing module handles engine subsystem and data relation analysis engine with the data screening respectively
Subsystem is connected, and can not handle for the data screening to be handled engine subsystem and data relation analysis engine subsystem
Data backup, and record processing abnormal conditions；
Log processing module handles engine subsystem with the data screening respectively and data relation analysis engine subsystem is equal
It is connected, for monitoring the log of data screening processing engine subsystem and data relation analysis engine subsystem operational process
Information；
Data feedback monitoring module handles engine subsystem and data relation analysis engine subsystem with the data screening respectively
System is connected, and reminds for showing to pass through the data after the data service relating module is associated with and generate early warning.
The method for realizing big data quantity distributed treatment control in cloud computing platform based on above-mentioned system, it is main special
Point be, the method the following steps are included:
(1) the data validation correction verification module, data persistence processing module and data classification processing module to data into
Row Screening Treatment, comprising the following steps:
(1.1) the data validation correction verification module configuration data legitimacy verifies rule, and rule verification is carried out to data；
(1.2) the data persistence processing module is by the data conversion verified by data validation at business object；
(1.3) business object is sent to data relation analysis engine subsystem by the data classification processing module；
(2) the data service model checking module and data business association module are to the data relation analysis after screening；
(3) the selection result and analysis result are fed back to enterprise by the data monitoring O&M engine subsystem.
Preferably, further comprising the steps of after the step (1.3):
(1.4) the reception record sheet of the system creation different data format, and the data after screening are backuped into database
In.
Preferably, further comprising the steps of after the step (1.4):
(1.5) the data processing exception processing module judges whether each module is faulty, if it is, the data processing is different
The data that normal processing module is handling malfunctioning module are converted into xml document and save, then timing to data re-transmitting until
Data processing success；If it is not, then continuing step (1.1).
Preferably, the data validation correction verification module configuration data legitimacy verifies rule in the step (1.1), specifically
Are as follows:
The data validation correction verification module passes through xsd file configuration data validation verification rule.
Preferably, the data validation correction verification module carries out rule verification to data in the step (1.1), specifically:
Semi-structured data is converted to java.io.ByteArrayInputStream by the data validation correction verification module,
Data validation correction verification module simultaneously utilizes javax.xml.validation.Schema and javax.xml.validation.Sc
HemaFactory carries out rule verification to data.
Preferably, the data persistence processing module turns the data verified by data validation in the step (1.2)
Change business object into, specifically:
The data persistence processing module is by the data verified by data validation using JAXB tool according to schema
Entity class is generated, and calls MessageHandleImpl tool-class, generation entity class is converted into the business object of persistence.
Preferably, the business object is sent to data relation analysis by the data classification processing module in the step (1.3)
Engine subsystem, specifically:
The business object is sent to according to the type code of business object by http agreement by data classification processing module
Data relation analysis engine subsystem.
Preferably, the reception record sheet includes the unique id of data and creation time field in the step (1.4).
Preferably, the step (2) specifically includes the following steps:
(2.1) the data service model checking module formulates business rule, and the data received are carried out correlation
Business model verification；
(2.2) business object is associated by the data service relating module；
Preferably, the data service model checking module is related by the data received progress in the step (2.1)
Property business model verification, specifically:
The data service model checking module judges whether business model is permanent business model, if it is, all
Data into data relation analysis engine subsystem require to verify, and then proceed to step (2.2)；Otherwise continue step
(2.2)。
The data service relating module will be different by the unique id field of data received in record sheet in the step (2.2)
Business object according to business association major key be associated with.
Preferably, between the step (2.1) and (2.2) the following steps are included:
(2.1.1) the data service model checking module carries out secondary verification to the data verified by business model, and
Generate feedback result.
Preferably, including data feedback monitoring module in the data monitoring O&M engine subsystem, which is specifically wrapped
Include following steps:
(3.1) system and creates data feedback record sheet by result back services distributed deployment to different server；
(3.2) the selection result is notified enterprise by the data feedback monitoring module, and data feedback is recorded in feedback content
In record sheet.
Preferably, the data feedback record sheet includes the unique id of data, feedback content and feedback time in the step (3.1)
Field.
Preferably, further include log monitoring processing operation in the method, specifically:
The log processing module records the normal log of system when data screening processing engine subsystem work, and records
System exception log when the data processing exception processing module works.
Using the system and method for realizing big data quantity distributed treatment in the cloud computing platform of the invention, business development
Personnel can effectively promote data processing speed when realizing related needs of the enterprise for big data quantity with the method in the present invention
Degree, and the verification rule of data according to enterprise demand, not only can be flexibly formulated, while can also be divided at any time according to portfolio
Cloth deployment, optimize the processing engine of data, improve the practicability of system, simultaneity factor framework is simple, system compatibility compared with
By force, stable and reliable working performance, the scope of application are relatively broad.
Detailed description of the invention
Fig. 1 is the general frame figure that the system of big data quantity distributed treatment is realized in cloud computing platform of the invention.
Fig. 2 is in the method for realize in cloud computing platform of the invention big data quantity distributed treatment about data screening
Handle the flow diagram of engine.
Fig. 3 is in the method for realize in cloud computing platform of the invention big data quantity distributed treatment about data correlation
The flow diagram of analysis engine.
Specific embodiment
It is further to carry out combined with specific embodiments below in order to more clearly describe technology contents of the invention
Description.
The system that big data quantity distributed treatment is realized in the cloud computing platform of the invention, at data screening
Engine subsystem, data relation analysis engine subsystem and data monitoring O&M engine subsystem are managed, at the data screening
Reason engine subsystem is connected with the data relation analysis engine subsystem and data monitoring O&M engine subsystem respectively
It connects, and the data relation analysis engine subsystem is connected with data monitoring O&M engine subsystem；
The data screening handles engine subsystem
Data validation correction verification module is connected with the data monitoring O&M engine subsystem, and for input
Data carry out validity checking；
Data persistence processing module is connected with the data monitoring O&M engine subsystem, and for that will pass through
The data of the data validation correction verification module validity checking are converted to relevant business object；
Data classification processing module, respectively with the data relation analysis engine subsystem and data monitoring O&M engine
Subsystem is connected, and the industry for will be generated by the business object of the data persistence processing module according to data
Business object is classified, and different business objects is separately sent to data relation analysis engine subsystem；
Data backup module is connected with the data monitoring O&M engine subsystem, for closing the data
The data that connection analysis engine subsystem receives are backed up.
The data relation analysis engine subsystem includes:
Data service model checking module handles engine subsystem and data monitoring O&M with the data screening respectively
Engine subsystem is connected, and verifies for that will handle the received data of engine subsystem from data screening, filters out symbol
Close the data of Correlation Criteria；
Data service relating module is connected, for that will pass through data with the data monitoring O&M engine subsystem
The data of business model correction verification module verification are handled.
The data monitoring O&M engine subsystem includes:
Data processing exception processing module handles engine subsystem and data relation analysis with the data screening respectively
Engine subsystem is connected, for the data screening to be handled engine subsystem and data relation analysis engine subsystem
The data backup that can not be handled, and record processing abnormal conditions；
Log processing module handles engine subsystem and data relation analysis engine subsystem with the data screening respectively
System is connected, and the data screening processing engine subsystem and data relation analysis engine subsystem for monitoring described were run
The log information of journey；
Data feedback monitoring module handles engine subsystem and data relation analysis engine with the data screening respectively
Subsystem is connected, and reminds for showing to pass through the data after the data service relating module association and generate early warning.
In the present invention, the side of big data quantity distributed treatment control in cloud computing platform should be realized based on above-mentioned system
Method, comprising the following steps:
(1) data validation correction verification module, data persistence processing module and data classification processing module logarithm described in
According to progress Screening Treatment, comprising the following steps:
(1.1) the data validation correction verification module configuration data legitimacy verifies rule described in, and rule is carried out to data
Verifying, specifically:
The data validation correction verification module passes through xsd file configuration data validation verification rule, the data
Semi-structured data is converted to java.io.ByteArrayInputStream, data validation verification by legitimacy verifies module
Module simultaneously utilizes javax.xml.validation.Schema and javax.xml.validation.SchemaFactory logarithm
According to progress rule verification；
(1.2) the data persistence processing module described in is by the data conversion verified by data validation at business pair
As, specifically:
The data persistence processing module by the data verified by data validation using JAXB tool according to
Schema generates entity class, and calls MessageHandleImpl tool-class, and generation entity class is converted into the business of persistence
Object；
(1.3) business object is sent to data relation analysis engine subsystem by the data classification processing module described in
System, specifically:
Data classification processing module sends out the business object according to the type code of business object by http agreement
It send to data relation analysis engine subsystem；
(1.4) the reception record sheet of the system creation different data format described in, and the data after screening are backuped into number
According in library, wherein the reception record sheet includes the unique id of data and creation time field；
(1.5) the data processing exception processing module described in judges whether each module is faulty, if it is, the number
Xml document is converted into according to the data that processing exception processing module is handling malfunctioning module and is saved, and then timing is to data
It retransmits until data processing success；If it is not, then continuing step (1.1)；
(2) the data service model checking module and data business association module described in are to the data correlation after screening point
Analysis；
(2.1) the data service model checking module described in formulates business rule, and the data received are carried out correlation
Property business model verification, specifically:
The data service model checking module judges whether business model is permanent business model, if it is,
All data into data relation analysis engine subsystem require to verify, and then proceed to step (2.2)；Otherwise continue step
(2.2)；
Data service model checking module described in (2.1.1) carries out secondary school to the data verified by business model
It tests, and generates feedback result；
(2.2) business object is associated by the data service relating module described in, in which:
The data service relating module is by receiving the unique id field of data in record sheet for different business pair
As being associated with according to business association major key；
(3) the selection result and analysis result are fed back to enterprise by the data monitoring O&M engine subsystem described in；
(3.1) system described in by result back services distributed deployment to different server, and create data feedback note
Record table, wherein the data feedback record sheet includes the unique id of data, feedback content and feedback time field；
(3.2) the selection result is notified enterprise by the data feedback monitoring module described in, and data are recorded in feedback content
In feedback record table.
It in a preferred embodiment of the present invention, further include log monitoring processing operation in the method, specifically
Are as follows:
System normal day when the log processing module record data screening processing engine subsystem works
Will, and record the system exception log when data processing exception processing module work.
In actual use, the present invention is not carry out data correlation processing when handling mass data, propose finger
A kind of System and method for of the big data quantity distributed treatment in cloud computing platform, i.e., while handling mass data, moreover it is possible to
According to the business demand of enterprise, data are associated.
The system and method for big data quantity distributed treatment in cloud computing platform mainly include data screening processing engine
System, data relation analysis engine subsystem, data monitoring O&M engine subsystem, as shown in Figure 1.
Data screening handles engine subsystem
Data validation correction verification module, this module mainly carry out validity checking to the data of input, avoid wrong illegal
Data enter system, filter out legal data and be further processed, reduce the treating capacity of system data.
Data persistence processing module, the main storage model of this module are converted to the data model in memory, i.e., will receive
Data be converted to relevant business object.
Data classification processing module, this module are mainly classified according to the business object that data generate, will be different
Business object is separately sent to data correlation service.
The data received are mainly backed up, are caused to prevent engine subsystem delay machine by data backup module, this module
Loss of data.
Data relation analysis engine subsystem includes:
Data service model checking module, this module mainly will handle the received data root of engine subsystem from data screening
It is verified according to the business model of formulation, filters out the data for meeting Correlation Criteria.
Data service relating module, this module by the data verified by business model according to certain business correlation into
Row filtering, duplicate removal/change, generate association key assignments, in group association, intercorrelation, multi-party fractionation, follow-up link trigger etc. it is a series of
Processing.
Data monitoring O&M engine subsystem includes:
Data processing exception processing module, the data that this module mainly can not be handled each engine subsystem carry out standby
Part, and processing abnormal conditions are recorded, facilitate operation maintenance personnel to check failure cause, and timing pushes abnormal data again, to draw
Data are handled again after holding up subsystem fault reparation.
Log processing module, this module are used for the log information of supervisor engine subsystem operational process, facilitate examining for later period
It looks into and misarrangement.Log information will be generated on each node of engine subsystem in the process of running, it will by log processing module
Log information recording is into file.
Data feedback monitoring module, for this module for the data after showing association, enterprise can be to the business datum after association
Real-time query is carried out, and the data after system meeting analyzing and associating generate relevant early warning and remind according to different business models,
Enterprise is facilitated to supervise data source.
Data screening handle engine subsystem process the following steps are included:
By the data validation correction verification module, data persistence processing module, data classification processing module, data backup
Module, data processing exception processing module, log processing module are integrated to data screening processing engine subsystem, and data are sieved
Choosing processing engine subsystem distributed deployment initializes running environment into each server；
In the data validation correction verification module, configuration data legitimacy verifies rule is verified by xsd file configuration and is advised
Then, semi-structured data is converted into java.io.ByteArrayInputStream and utilized
Javax.xml.validation.Schema and javax.xml.validation.SchemaFactory carries out rule verification；
In the data persistence processing module, by the data verified by data validation using JAXB tool according to
Schema generates entity class, and calls MessageHandleImpl tool-class, converts generation entity class to the business of persistence
Object；
It is logical according to the type code of business object by the business object after persistence in the data classification processing module
It crosses http agreement and is sent to data relation analysis engine subsystem；
In the data backup module, after the data described in the above-mentioned steps are sent successfully, data backup module can will
It sends successful data and is converted into xml document, and be saved in and formulate under catalogue；
In the data processing exception processing module, the module is mainly in the treatment process of above-mentioned steps, Ke Nengfa
Raw fault condition carries out abnormality processing.Treatment mechanism is as follows, when data processing exception processing module monitors that some module goes out
After existing failure, the data that data processing exception processing module can handled malfunctioning module are converted into xml document and save different
Regular data storing directory, data processing exception processing module have independent thread timing and go processing abnormal data storing directory
Xml document, until abnormal data all complete by processing；
In the log processing module, in above-mentioned steps, log information can be generated, the deployment of this module is nested in each
In operation system, effect is the normal log of record system and the log that system is abnormal.It needs when record according to regulation
Format, the detailed data of normal log, abnormal log need to record context when detailed exception object and abnormal generation
Data
Data relation analysis engine subsystem process the following steps are included:
By the data service model checking module, data service relating module, data processing exception processing module, log
Processing module is integrated to data relation analysis engine subsystem, and by the distributed deployment of data relation analysis engine subsystem to respectively
In a server, and initialize running environment；
The data service model checking module, the number that this module will be received from data screening processing engine subsystem
According to progress business model inspection, business model is divided into permanent business model and temporary traffic model, permanent business model
It for the business model permanently to come into force, can not generally modify, all data into engine subsystem require to verify；Provisional industry
Business model is the business model temporarily to come into force, allows to modify and supports heat deployment；Business model verification will continue to refinement business pair
As type, the state of business object is screened out, and gives data service relating module and carries out subsequent processing；
The data service relating module, which is filtered according to certain business correlation, duplicate removal/change, life
At a series of processing such as association, intercorrelation, multi-party fractionation, follow-up link triggering in association key assignments, group.
In order to be more clearly understood that technology contents of the invention, spy lifts following embodiment and is described in detail.
Shown in please referring to Fig.1 to Fig.3, the System and method for of big data quantity distributed treatment in the cloud computing platform, specifically
Embodiment comprises the steps of:
1, data screening is handled
The present invention first screens the data of a large amount of isomeries received, at the process meeting distributed parallel of screening
Reason, so as to improve data processing speed, the data for finally meeting format will circulate into next link, not meet the data of format
It will be removed.The link includes data loading, backup and abnormality processing operation, and detailed process is as follows:
By data screening service distributed deployment into different server；
Satisfactory data format is formulated, the data received are verified；
By the data conversion after verification at business object, the business object that data generate is sorted and is sent to data
Association service；
Create the reception record sheet of different data format, comprising: the fields such as the unique id of data, creation time will pass through sieve
The data of choosing backup in database；
If data processing is abnormal, system can be by this part of data landing at XML file, and then data are retransmitted in timing,
Until data processing success；
2, the association analysis and monitoring of data
This link will be associated analysis to the data of a upper link, will wherein related data associate point
Analysis, can monitor analysis result in real time.The link includes the operation such as data correlation, analysis, backup and abnormality processing, and detailed process is such as
Under:
It creates data and analyzes record sheet, comprising: the fields such as the unique id of data, analysis link, analysis time, analysis result；
Business rule is formulated, the data received carry out the verification of business correlation, and generate relevant feedback result；
By the different unique id received in record sheet, different business objects is associated with according to business association major key；
Settable specific rule verifies the data verified by service logic again and generates relevant feedback knot
Fruit；
Analysis result can be reflected in real time on the front end monitoring page；
3, result is fed back
This link by the first two link the selection result and analysis result feed back to enterprise.The link includes feedback note
The operation such as record, abnormality processing, detailed process is as follows:
It will be in result back services distributed deployment to different server；
Create data feedback record sheet, comprising: the fields such as the unique id of data, feedback content, feedback time；
The selection result is notified into enterprise, and feedback content is recorded in data feedback record sheet.
Using the system and method for realizing big data quantity distributed treatment in the cloud computing platform of the invention, business development
Personnel can effectively promote data processing speed when realizing related needs of the enterprise for big data quantity with the method in the present invention
Degree, and the verification rule of data according to enterprise demand, not only can be flexibly formulated, while can also be divided at any time according to portfolio
Cloth deployment, optimizes the processing engine of data, improves the practicability of system.
In this description, the present invention is described with reference to its specific embodiment.But it is clear that can still make
Various modifications and alterations are without departing from the spirit and scope of the invention.Therefore, the description and the appended drawings should be considered as illustrative
And not restrictive.
Claims (18)
1. realizing the system of big data quantity distributed treatment in a kind of cloud computing platform, which is characterized in that the system includes
Data screening handles engine subsystem, data relation analysis engine subsystem and data monitoring O&M engine subsystem, described
It is sub with the data relation analysis engine subsystem and data monitoring O&M engine respectively that data screening handles engine subsystem
System is connected, and the data relation analysis engine subsystem is connected with data monitoring O&M engine subsystem；
The data screening handles engine subsystem
Data validation correction verification module is connected with the data monitoring O&M engine subsystem, and for the number to input
According to progress validity checking；
Data persistence processing module is connected with the data monitoring O&M engine subsystem, and being used for will be by described
The data of data validation correction verification module validity checking be converted to relevant business object；
Data classification processing module, respectively with the data relation analysis engine subsystem and data monitoring O&M engine subsystem
System is connected, and the business pair for will be generated by the business object of the data persistence processing module according to data
As classifying, different business objects is separately sent to data relation analysis engine subsystem；
Data backup module is connected with the data monitoring O&M engine subsystem, for dividing the data correlation
The data that analysis engine subsystem receives are backed up.
2. the system of big data quantity distributed treatment in cloud computing platform according to claim 1, which is characterized in that described
Data relation analysis engine subsystem include:
Data service model checking module handles engine subsystem and data monitoring O&M engine with the data screening respectively
Subsystem is connected, and verifies for that will handle the received data of engine subsystem from data screening, filters out and meet pass
The data of bracing part；
Data service relating module is connected, for that will pass through data service with the data monitoring O&M engine subsystem
The data of model checking module verification are handled.
3. the system of big data quantity distributed treatment in cloud computing platform according to claim 1, which is characterized in that described
Data monitoring O&M engine subsystem include:
Data processing exception processing module handles engine subsystem and data relation analysis engine with the data screening respectively
Subsystem is connected, for can not by the described data screening processing engine subsystem and data relation analysis engine subsystem
The data backup of processing, and record processing abnormal conditions；
Log processing module, it is equal with the data screening processing engine subsystem and data relation analysis engine subsystem respectively
It is connected, for monitoring the data screening processing engine subsystem and data relation analysis engine subsystem operational process
Log information；
Data feedback monitoring module handles engine subsystem and data relation analysis engine subsystem with the data screening respectively
System is connected, and reminds for showing to pass through the data after the data service relating module association and generate early warning.
4. a kind of side for realizing big data quantity distributed treatment control in cloud computing platform based on system described in claim 1
Method, which is characterized in that the method the following steps are included:
(1) data validation correction verification module, data persistence processing module and data classification processing module described in data into
Row Screening Treatment, comprising the following steps:
(1.1) the data validation correction verification module configuration data legitimacy verifies rule described in, and rule verification is carried out to data；
(1.2) the data persistence processing module described in is by the data conversion verified by data validation at business object；
(1.3) business object is sent to data relation analysis engine subsystem by the data classification processing module described in；
(2) the data service model checking module and data business association module described in are to the data relation analysis after screening；
(3) the selection result and analysis result are fed back to enterprise by the data monitoring O&M engine subsystem described in.
5. the method according to claim 4 for realizing big data quantity distributed treatment in cloud computing platform, the data
Monitoring has data processing exception processing module in O&M engine subsystem, which is characterized in that after the step (1.3) also
The following steps are included:
(1.4) the reception record sheet of the system creation different data format described in, and the data after screening are backuped into database
In.
6. the method according to claim 5 for realizing big data quantity distributed treatment in cloud computing platform, the data
Monitoring has data processing exception processing module in O&M engine subsystem, which is characterized in that after the step (1.4) also
The following steps are included:
(1.5) the data processing exception processing module described in judges whether each module is faulty, if it is, at the data
The data that reason exception processing module is handling malfunctioning module are converted into xml document and save, and then timing is to data re-transmitting
Until data processing success；If it is not, then continuing step (1.1).
7. the method according to claim 4 for realizing big data quantity distributed treatment in cloud computing platform, which is characterized in that
The rule of data validation correction verification module configuration data legitimacy verifies described in the step (1.1), specifically:
The data validation correction verification module passes through xsd file configuration data validation verification rule.
8. the method according to claim 4 for realizing big data quantity distributed treatment in cloud computing platform, which is characterized in that
Data validation correction verification module described in the step (1.1) carries out rule verification to data, specifically:
Semi-structured data is converted to java.io.ByteArrayInputStream by the data validation correction verification module,
Data validation correction verification module simultaneously utilizes javax.xml.validation.Schema and javax.xml.validation.Sc
HemaFactory carries out rule verification to data.
9. the method according to claim 4 for realizing big data quantity distributed treatment in cloud computing platform, which is characterized in that
The data conversion that data persistence processing module described in the step (1.2) will be verified by data validation is at business
Object, specifically:
The data persistence processing module is by the data verified by data validation using JAXB tool according to schema
Entity class is generated, and calls MessageHandleImpl tool-class, generation entity class is converted into the business object of persistence.
10. the method according to claim 4 for realizing big data quantity distributed treatment in cloud computing platform, feature exist
In the business object is sent to data relation analysis by data classification processing module described in the step (1.3)
Engine subsystem, specifically:
The business object is sent to according to the type code of business object by http agreement by data classification processing module
Data relation analysis engine subsystem.
11. the method according to claim 6 for realizing big data quantity distributed treatment in cloud computing platform, feature exist
In reception record sheet described in the step (1.4) includes the unique id of data and creation time field.
12. the method according to claim 4 for realizing big data quantity distributed treatment in cloud computing platform, the data
It include data service model checking module and data business association module, the data monitoring in association analysis engine subsystem
It include data processing exception processing module in O&M engine subsystem, which is characterized in that the step (2) specifically includes following
Step:
(2.1) the data service model checking module described in formulates business rule, and the data received are carried out correlation
Business model verification；
(2.2) business object is associated by the data service relating module described in.
13. the method according to claim 12 for realizing big data quantity distributed treatment in cloud computing platform, the industry
Business model includes permanent business model and temporary traffic model, which is characterized in that number described in the step (2.1)
The business model that the data received carry out correlation is verified according to business model correction verification module, specifically:
The data service model checking module judges whether business model is permanent business model, if it is, all
Data into data relation analysis engine subsystem require to verify, and then proceed to step (2.2)；Otherwise continue step
(2.2)。
14. the method according to claim 12 for realizing big data quantity distributed treatment in cloud computing platform, feature exist
In data service relating module described in the step (2.2) will by the unique id field of data received in record sheet
Different business objects is associated with according to business association major key.
15. the method according to claim 12 for realizing big data quantity distributed treatment in cloud computing platform, feature exist
In, between the step (2.1) and (2.2) the following steps are included:
Data service model checking module described in (2.1.1) carries out secondary verification to the data verified by business model, and
Generate feedback result.
16. the method according to claim 4 for realizing big data quantity distributed treatment in cloud computing platform, feature exist
In, in the data monitoring O&M engine subsystem include data feedback monitoring module, the step (3) specifically include with
Lower step:
(3.1) result back services distributed deployment to different server, and is created data feedback record sheet by the system described in；
(3.2) the selection result is notified enterprise by the data feedback monitoring module described in, and data feedback is recorded in feedback content
In record sheet.
17. the method according to claim 16 for realizing big data quantity distributed treatment in cloud computing platform, feature exist
In data feedback record sheet described in the step (3.1) includes the unique id of data, feedback content and feedback time word
Section.
18. the method according to claim 4 for realizing big data quantity distributed treatment in cloud computing platform, feature exist
In including log processing module in the data monitoring O&M engine subsystem, further include log monitoring in the method
Processing operation, specifically:
The normal log of system when the log processing module record data screening processing engine subsystem works, and
System exception log when the record data processing exception processing module works.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811101541.8A CN109344199A (en) | 2018-09-20 | 2018-09-20 | The system and method for big data quantity distributed treatment is realized in cloud computing platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811101541.8A CN109344199A (en) | 2018-09-20 | 2018-09-20 | The system and method for big data quantity distributed treatment is realized in cloud computing platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109344199A true CN109344199A (en) | 2019-02-15 |
Family
ID=65305871
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811101541.8A Withdrawn CN109344199A (en) | 2018-09-20 | 2018-09-20 | The system and method for big data quantity distributed treatment is realized in cloud computing platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109344199A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111092931A (en) * | 2019-11-15 | 2020-05-01 | 中国科学院计算技术研究所 | Method and system for rapidly distributing streaming data of online super real-time simulation of power system |
| CN111625535A (en) * | 2020-04-17 | 2020-09-04 | 贝壳技术有限公司 | Method, device and storage medium for realizing business data association |
| CN112243205A (en) * | 2020-10-23 | 2021-01-19 | 中国电子科技集团公司第二十八研究所 | Multi-protocol fusion processing method based on Internet of things |
| CN112527682A (en) * | 2020-12-24 | 2021-03-19 | 四川享宇金信金融科技有限公司 | Model development method for fusing product functionality and user harmony |
| CN112948204A (en) * | 2021-02-07 | 2021-06-11 | 上海汉询软件有限公司 | Data processing system based on DataRobot technology |
| CN113722357A (en) * | 2021-06-03 | 2021-11-30 | 泰康保险集团股份有限公司 | Data file checking method and device, electronic equipment and storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102819609A (en) * | 2012-08-23 | 2012-12-12 | 国电南京自动化股份有限公司 | Modeling method for persisting data model |
| CN103970903A (en) * | 2014-05-27 | 2014-08-06 | 重庆大学 | Large industrial system feedback data real-time processing method and system based on Web |
| CN105323111A (en) * | 2015-11-17 | 2016-02-10 | 南京南瑞集团公司 | Operation and maintenance automation system and method |
| US20160283715A1 (en) * | 2013-03-14 | 2016-09-29 | Sas Institute Inc. | Unauthorized activity detection and classification |
| CN106156350A (en) * | 2016-07-25 | 2016-11-23 | 恒安嘉新(北京)科技有限公司 | The big data analysing method of a kind of visualization and system |
| CN107103094A (en) * | 2017-05-18 | 2017-08-29 | 前海梧桐（深圳）数据有限公司 | Data among enterprises incidence relation method for catching and its system based on mass data |
| CN107357809A (en) * | 2017-05-27 | 2017-11-17 | 国家电网公司 | Magnanimity platform Hisoon real-time database metric data access system |
| CN107506408A (en) * | 2017-08-08 | 2017-12-22 | 北京盛华安信息技术有限公司 | To the method and system of magnanimity event distribution formula association matching |
-
2018
- 2018-09-20 CN CN201811101541.8A patent/CN109344199A/en not_active Withdrawn
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102819609A (en) * | 2012-08-23 | 2012-12-12 | 国电南京自动化股份有限公司 | Modeling method for persisting data model |
| US20160283715A1 (en) * | 2013-03-14 | 2016-09-29 | Sas Institute Inc. | Unauthorized activity detection and classification |
| CN103970903A (en) * | 2014-05-27 | 2014-08-06 | 重庆大学 | Large industrial system feedback data real-time processing method and system based on Web |
| CN105323111A (en) * | 2015-11-17 | 2016-02-10 | 南京南瑞集团公司 | Operation and maintenance automation system and method |
| CN106156350A (en) * | 2016-07-25 | 2016-11-23 | 恒安嘉新(北京)科技有限公司 | The big data analysing method of a kind of visualization and system |
| CN107103094A (en) * | 2017-05-18 | 2017-08-29 | 前海梧桐（深圳）数据有限公司 | Data among enterprises incidence relation method for catching and its system based on mass data |
| CN107357809A (en) * | 2017-05-27 | 2017-11-17 | 国家电网公司 | Magnanimity platform Hisoon real-time database metric data access system |
| CN107506408A (en) * | 2017-08-08 | 2017-12-22 | 北京盛华安信息技术有限公司 | To the method and system of magnanimity event distribution formula association matching |
Non-Patent Citations (1)
| Title |
|---|
| 余成: "基于复杂事件处理的分布式监控***", 《中国优秀硕士学位论文全文数据库 信息科技辑（月刊）》 * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111092931A (en) * | 2019-11-15 | 2020-05-01 | 中国科学院计算技术研究所 | Method and system for rapidly distributing streaming data of online super real-time simulation of power system |
| CN111092931B (en) * | 2019-11-15 | 2021-08-06 | 中国科学院计算技术研究所 | Method and system for rapidly distributing streaming data of online super real-time simulation of power system |
| CN111625535A (en) * | 2020-04-17 | 2020-09-04 | 贝壳技术有限公司 | Method, device and storage medium for realizing business data association |
| CN112243205A (en) * | 2020-10-23 | 2021-01-19 | 中国电子科技集团公司第二十八研究所 | Multi-protocol fusion processing method based on Internet of things |
| CN112527682A (en) * | 2020-12-24 | 2021-03-19 | 四川享宇金信金融科技有限公司 | Model development method for fusing product functionality and user harmony |
| CN112527682B (en) * | 2020-12-24 | 2023-10-27 | 四川享宇金信金融科技有限公司 | Model development method for fusing product functionality and user coordination |
| CN112948204A (en) * | 2021-02-07 | 2021-06-11 | 上海汉询软件有限公司 | Data processing system based on DataRobot technology |
| CN113722357A (en) * | 2021-06-03 | 2021-11-30 | 泰康保险集团股份有限公司 | Data file checking method and device, electronic equipment and storage medium |
| CN113722357B (en) * | 2021-06-03 | 2024-04-09 | 泰康保险集团股份有限公司 | Data file verification method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109344199A (en) | The system and method for big data quantity distributed treatment is realized in cloud computing platform | |
| CN101784988B (en) | Transaction aggregation to increase transaction processing throughput | |
| EP2541413A1 (en) | Systems and Methods for Data Integrity Checking | |
| US10121071B2 (en) | Document verification system | |
| US20080065588A1 (en) | Selectively Logging Query Data Based On Cost | |
| CN105528275B (en) | Database security inspection method | |
| CN1604040A (en) | Dynamic transaction control within a host transaction processing system | |
| US20220129816A1 (en) | Methods and arrangements to manage requirements and controls, and data at the intersection thereof | |
| CN107656858A (en) | A kind of method and system of automatic O＆M monitoring oracle database | |
| CN113791586A (en) | Novel industrial APP and identification registration analysis integration method | |
| CN111930726B (en) | Off-line form-based grade protection evaluation data acquisition and analysis method and system | |
| CN115982012A (en) | Evaluation model and method for interface management capability maturity | |
| Shehzadi et al. | A novel framework for change requirement management (CRM) in agile software development (ASD) | |
| CN110825641A (en) | Micro-service application test tool set based on simulation data generator | |
| CN107423035B (en) | Product data management system in software development process | |
| Lytvyn et al. | Development of Intellectual System for Data De-Duplication and Distribution in Cloud Storage. | |
| CN117541195A (en) | Intelligent management method and system for audit project | |
| El Akkaoui et al. | A Quality-based ETL Design Evaluation Framework. | |
| CN116841980A (en) | Bank data processing system | |
| CN114817171A (en) | Buried point data quality control method | |
| Tadano et al. | Automatic synthesis of SRN models from system operation templates for availability analysis | |
| CN115168297A (en) | Bypassing log auditing method and device | |
| KR102676519B1 (en) | Apparatus and Method for Providing Intelligent Task Automation Service | |
| CN112596675A (en) | Multi-storage method and device of data, electronic equipment and computer storage medium | |
| CN110262927A (en) | Treating method and apparatus applied to data backup |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20190215 |
|
| WW01 | Invention patent application withdrawn after publication |