CN103955510A - Massive electricity marketing data integration method uploaded by ETL cloud platform - Google Patents
Massive electricity marketing data integration method uploaded by ETL cloud platform Download PDFInfo
- Publication number
- CN103955510A CN103955510A CN201410180132.7A CN201410180132A CN103955510A CN 103955510 A CN103955510 A CN 103955510A CN 201410180132 A CN201410180132 A CN 201410180132A CN 103955510 A CN103955510 A CN 103955510A
- Authority
- CN
- China
- Prior art keywords
- data
- cloud platform
- uploaded
- module
- etl
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 43
- 230000010354 integration Effects 0.000 title claims abstract description 20
- 230000005611 electricity Effects 0.000 title abstract 3
- 238000012544 monitoring process Methods 0.000 claims abstract description 26
- 230000008569 process Effects 0.000 claims abstract description 22
- 238000012545 processing Methods 0.000 claims description 26
- 238000013500 data storage Methods 0.000 claims description 13
- 238000007726 management method Methods 0.000 claims description 11
- 230000002159 abnormal effect Effects 0.000 claims description 10
- 230000005856 abnormality Effects 0.000 claims description 9
- 238000003860 storage Methods 0.000 claims description 6
- 230000005540 biological transmission Effects 0.000 claims description 5
- 238000001514 detection method Methods 0.000 claims description 4
- 230000006870 function Effects 0.000 claims description 4
- 238000005201 scrubbing Methods 0.000 claims description 4
- 230000000694 effects Effects 0.000 claims description 3
- 230000000977 initiatory effect Effects 0.000 claims description 3
- 230000008602 contraction Effects 0.000 claims description 2
- 230000007613 environmental effect Effects 0.000 claims description 2
- 238000004422 calculation algorithm Methods 0.000 abstract description 4
- 238000004140 cleaning Methods 0.000 abstract description 4
- 238000000605 extraction Methods 0.000 description 10
- 239000000284 extract Substances 0.000 description 7
- 238000006243 chemical reaction Methods 0.000 description 6
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000002354 daily effect Effects 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000004069 differentiation Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 230000004907 flux Effects 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 239000013642 negative control Substances 0.000 description 1
- 238000013515 script Methods 0.000 description 1
- 230000033772 system development Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/06—Energy or water supply
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Economics (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Water Supply & Treatment (AREA)
- General Health & Medical Sciences (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- Marketing (AREA)
- General Business, Economics & Management (AREA)
- Human Resources & Organizations (AREA)
- Primary Health Care (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Public Health (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a massive electricity marketing data integration method uploaded by an ETL cloud platform. The whole process module of the method comprises a business relevant data module, a monitoring module, an ETL operation process, a protocol stack module, a middle server data memory module, a protocol stack uploading process, and a cloud platform data memory system module. The method utilizes an ETL tool to develop series 'operation' relevant to the business, the business data of various platforms is processed by a series of steps such as cleaning, extracting, converting and 'protocol stack' rules to generate text files on a middle server, and the text files are model index text files which can be directly applied to the latter cloud platform algorithm modeling. Finally, model index text files on the middle server are uploaded to the cloud platform according to the /protocol stack' rules, the uploading from the business data of a database to the index data of the cloud platform is completed, and the rapid integration for the massive electricity marketing data is realized.
Description
Technical field
The present invention relates to the technical field of extraction, conversion and the integration of source data, be specifically related to magnanimity power marketing data automatically to upload to cloud platform data storage system based on ETL cloud platform, realize the quick integration of mass data.
Technical background
At present, power supply administration of districts and cities system has realized plant stand electric flux terminal, negative control terminal, distribution transformer terminals, the integration of low-voltage collecting meter reading terminal four systems is used, realize plant stand metering, specially become user, distribute-electricity transformer district, the table of residential quarter counts collection, each power supply administration plant stand coverage rate, load management terminal coverage rate, distribution transformer monitoring and metering terminal coverage rate reaches 100% substantially, low pressure resident's collection is copied coverage rate and is reached more than 25%, metering automation system main website gathers mass data every day, carry out data sharing by Marketing Management Information System development interface and realize centralized automatic meter-reading, the monitoring of opposing electricity-stealing, the work such as metering fault anomaly analysis.
Along with the informatization of electric power enterprise, also impel data to increase in large quantities, according to statistics, every 2～3 years of data volume will be doubled and redoubled.Power supply administration of prefectures and cities construction scale data, electric quantity data, load data, line loss data, service operation data, power supply quality data etc., these data volumes rise to EB(1 EB=1 000 PB from PB very soon) mass data, growth rate is more and more faster, data volume is very big; Electric power data numerous types, traditional electrical production is mainly taking structural data as main, relate to measurement, the record of flow process and the management of assets etc. of all kinds of electric weight, the unstructured data such as video, audio frequency, text increased rapidly in recent years, the very fast ultrastructure data of its quantity, become the chief component of the large data of electric power gradually.
Because power industry is produced and meets society need as target taking electric power safety, the data that relate to electrical production, metering and billing, power marketing etc. must be accurate, and requirement of real-time is also high.So how enterprise is by various technological means, and data are converted to real-time information, knowledge, have become to improve the Main Bottleneck of electric power enterprise core competitiveness.ETL is a main technological means.
The present invention is based on the automatic uploading file of ETL cloud platform, and magnanimity power marketing data upload, to cloud platform data storage system, is realized to the quick integration of mass data.First utilize ETL instrument, develop the series " operation " with traffic aided.Then by this series of " operation " the business datum of various platforms through cleaning, extract, the series of steps such as conversion and " protocol family " (the one group of agreement being associated with each other) rule, generate the text above intermediate server MS, these texts are model index texts, directly can be used in later stage cloud platform algorithm modeling.Finally the model index text above intermediate server MS is uploaded to cloud platform according to " protocol family " rule, finishing service data are uploading to cloud platform achievement data by database.
Summary of the invention
The object of the invention is, in order to solve large this problem of magnanimity power marketing Data Integration difficulty, provides a kind of magnanimity power marketing data integration method of uploading based on ETL cloud platform.
To achieve these goals, technical scheme of the present invention is as follows:
A magnanimity power marketing data integration method of uploading based on ETL cloud platform, it comprises the magnanimity power marketing platform for data arrangement that some station servers form, and uses ETL instrument, characterized by further comprising:
Service related data module, monitoring module, ETL work flow, protocol family module, intermediate server data memory module, protocol family are uploaded flow process and cloud platform data storage system modules, wherein
Realizing magnanimity power marketing Data Integration comprises the following steps:
1. data pick-up, will be stored in data pick-up in service related data storehouse out;
2. data cleansing, utilizes ETL instrument to unify the service related data of isomery by writing corresponding operation Job processing procedure, removes different data layouts;
3. data are returned, and carry out corresponding processing and obtain unified data and return, and are the achievement data that the modelling phase uses;
4. data-switching, carries out service related data and uploads data to the transmission of intermediate server, adopts the storage of txt form;
5. data upload, the fixing catalogue data of scanning intermediate server uploads to cloud platform data storage system automatically.
This method is passed through this series of " operation " series of steps such as the process extraction in the service related data of heterogeneous platform, cleaning, conversions, generate the text above intermediate server MS, these texts are model index texts, directly can be used in later stage cloud platform algorithm modeling.Then the model index text above intermediate server MS is uploaded to cloud platform according to self-defined " rule ", complete the uploading to cloud platform achievement data by service related data.Its beneficial effect bringing is beneficial effect: realized the quick integration to magnanimity power marketing data, data are done to standardization definition, realized unified coding, unified classification and tissue.The content of standardization definition comprises: standard code is unified, business terms is unified.
brief description of the drawings:
Fig. 1 is structural representation of the present invention;
Fig. 2 is ETL operation process chart;
Fig. 3 is that MS cloud platform is uploaded operation schematic diagram.
Embodiment
Below in conjunction with accompanying drawing, technical scheme of the present invention is described in further detail.
Fig. 1 is structural representation of the present invention, it comprises the magnanimity power marketing platform for data arrangement that 8 station servers form, wherein 6 calculation servers, 1 database server and 1 intermediate server MS, that is to say WEB server, use ETL(Extraction-Transformation-Loading, extract, change and load) instrument, characterized by further comprising: service related data module, monitoring module, ETL work flow, protocol family module, intermediate server data memory module, protocol family are uploaded flow process and cloud platform data storage system modules, wherein
Described service related data module is each province and city database server modules;
Described monitoring module is mainly made up of monitoring operation and Platform Server monitoring;
Described ETL work flow refers to carries out corresponding Job processing procedure;
Described protocol family module comprises data scrubbing rule, data upload constraint, data processing rule, operation logic rules;
Described intermediate server data memory module refers to the operation of ETL series is uploaded to intermediate server MS by service related data, and adopts the storage of txt form;
Described protocol family is uploaded flow process and is referred to that the model index text above intermediate server MS uploads to cloud platform according to " protocol family " rule;
Described cloud platform data storage system modules mainly by HDFS file system and thereon build Hive and HBase database.
Realize Data Integration further comprising the steps of:
1, data pick-up, will be stored in data pick-up in service related data storehouse out;
2, data cleansing, utilizes ETL instrument to unify the service related data of isomery by writing corresponding operation Job processing procedure, removes different data layouts;
3, data are returned, and carry out corresponding processing and obtain unified data and return, and are the achievement data that the modelling phase uses;
4, data-switching, carries out service related data and uploads data to the transmission of intermediate server, adopts the storage of txt form;
5, data upload, the fixing catalogue data of scanning intermediate server uploads to cloud platform data storage system automatically.
Described monitoring module is uploaded flow process to the operation of ETL series and protocol family and is carried out monitoring management.
To ETL series, operation has effect of contraction to described protocol family module, and protocol family is uploaded to flow process directive function.
Described cloud platform data storage system modules refers to and stores the data that are stored in intermediate server MS into cloud platform data storage system.
Described ETL is the abbreviation of Extraction-Transformation-Loading, is the instrument that extracts, changes and load.
Described protocol family is data scrubbing rule, data upload constraint, data processing rule, operation logic rules etc.
Workflow explanation in Fig. 1: first by each province and city data, as Nanning load data, North Sea load data etc., upload to intermediate server.This upload procedure completes by exploitation ETL operation, and the ETL operation of exploitation needs constraint, the constraint of data scrubbing and the constraint of operation logic control of the data upload in protocol compliant bunch.Simultaneously carrying out in ETL work data uploads, what system monitoring module can monitoring task carries out state; Administration module can retrain to carry out task readjustment, task status according to the operation logic rules in protocol family operation is set etc.The data of intermediate server are the magnanimity power marketing data after integration.Intermediate server data upload data in cloud platform data storage system by the custom protocol in protocol family, use multilink to reach object parallel, efficient transfer data.Custom protocol in protocol family instructs the process of uploading data, and each link of monitoring module monitoring simultaneously uploaded state, and administration module manages according to uploading operation service logic.
The present invention, by using ETL instrument, develops the series " operation " (being Job in ETL) with traffic aided.By the business datum of various platforms through extraction, clean, conversion and " protocol family " rule etc. series of steps, generate the text above intermediate server MS, these texts are model index texts, directly can be used in later stage cloud platform algorithm modeling.Wherein, ETL data pick-up refers to according to loading data and extracts the degree of affecting of result to target database (as data warehouse), be divided into the data pick-up mode that incrementally updating and full dose are upgraded: it is to extract from database intact the data of the table in data source or view that full dose extracts, and the form that can identify of the ETL instrument that converts oneself to, increment extraction is data that only extract newly-increased in the table that will extract in database or amendment since extracting last time; ETL data cleansing refers to filters those undesirable data, and undesirable data are mainly to have the data of incomplete data, mistake and the data three major types of repetition; ETL data-switching refers to the conversion of carrying out inconsistent data-switching, data granularity; The loading of ETL data refers to pretreated data is loaded in target data warehouse, can adopt SQL statement to complete loading.Then the model index text above intermediate server MS is uploaded to cloud platform according to " protocol family " rule, finishing service data are uploading to cloud platform achievement data by database.
In addition,, in the time that source data is carried out to pre-service, these 3 the preposition work of cluster management, load balancing and the ETL based on concurrent working mode to server have been comprised.
Aspect the cluster management and load balancing of server, the present invention adopts the parallel processing of Oracle RAC parallel cluster technology fulfillment database system, ensures the scalability of the system that realizes, and makes full use of computational resource.Oracle 10g is an architecture software that calculates specially exploitation for corporate grid, and its real application cluster (Real Application Cluster, RAC) is the assembly of Oracle10g, is also the core that its gridding technique is realized.It has Database cache fusion, shared disk, clear applications switching three large Core Features.In RAC architecture, each node moves a database instance.Between node, connect to carry out exchanges data by private network, and be connected with shared disk storage respectively.Node and application layer are in same dedicated network, although each node has different physical IP address, application client still can connect in the configuration of a virtual data base Service name, and system realizes load balancing automatically.Load balancing can adapt to the change of fast-changing business demand and thing followed working load automatically, by dynamically redistributing database resource, can between node, communicate by letter to optimize use cluster system resource with minimizing magnetic disc i/o with low delay.
Aspect the ETL based on concurrent working mode, the present invention can adopt multithreading to realize multi-task parallel scheduling, each task can be both thread independently, and the precedence of thread execution also can be set according to relation of interdependence, adopted the mode of serial and parallel combination to carry out data acquisition.ETL based on parallel processing technique can improve the execution efficiency in data Layer processing in ETL process, and a large amount of query scripts is distributed on multiple nodes and is carried out simultaneously.
Fig. 2 has provided ETL work flow, and the groundwork of ETL series operation is to be model achievement data for specific business datum according to certain logical transition, and work flow contains several necessary conditions below:
1, the operation of every ETL series all can generate in this locality one or more index files through extraction, cleaning, conversion and " protocol family " constraint, the operation of ETL series refers to the stack of multiple ETL tasks, and the multiple tasks that comprise such as the large arrow (dotted line) on the left side, Fig. 2 middle part can become a serial operation;
2, between serial ETL operation, there is no restricting relation, but the operation of the ETL in same stage series there is an agreement to be exactly: all ETL series operation in a stage all could continue the next stage after operation.Part in Fig. 2 above " protocol family " constraint is the serial ETL operation in same stage, there is no restricting relation between them.They belong to the same stage, after all task runs in this stage, just can carry out the next one stage, upload operation;
3, the ETL task in serial operation has two kinds of relations: rely on arranged side by side.Such as, in transformer series operation in Fig. 2, " per day load factor task " just depends on " transformer capacity extraction task " and " transformer load extraction task ", and is coordination between " transformer capacity extraction task " and " transformer load extraction task ";
4, the subsidiary monitoring of every ETL series operation, monitoring has two effects: 1. log; 2. in the time that appearance is abnormal, subsidiary corresponding " exception code ", to task processing module, unification is processed.
So-called " exception code " refers to different abnormal differentiation codings.
5, the task of uploading of " protocol family " constraint and task management managed together intermediate server (MS).
Shown in Fig. 2, work flow is explained as follows:
After the job initiation of ELT series, can start the serial ETL operation with concrete service logic equity number.Such as when 3 service logics of model data protection, so just have and start the operation of 3 ETL series.If but each service logic includes 4 indexs, just there are so altogether 12 indexs, in operation, will start 12 subtasks and upload uploading.
Each serial operation moves alone, and serial monitoring operation module is correspondingly carried out corresponding record.Such as after " transformer load extraction task " in the operation of transformer series is successfully completed, can record a daily record.If occur in carrying out " load factor standard deviation task ", extremely, the Mission Monitor module of transformer series operation will record an abnormal log so, sends a message that contains " exception code " to task management/abnormality processing module simultaneously.Task management/abnormality processing module receives after message, can create corresponding environment record (environment record refer to establishment be which task of which serial task).Then send one and restart " load factor standard deviation task ", " load factor standard deviation task " re-executes.Run succeeded, continue; Otherwise, the recording exceptional again of monitoring module correspondingly, then task processing module is confirmed follow-up processing according to current environment record.If certain task start of ETL series 2 times afterwards or not success, will start so the task of its dependence, after dependence task success, continue to carry out current abnormal task.If abnormal task or failure, will exit the operation of ETL series so.Any one serial operation occurs that above-mentioned situation all can exit the operation of ETL series.
Task arranged side by side in series operation is also same phased mission, defers to the agreement of condition (2).
After each serial Job execution completes, Mission Monitor correspondingly will send one and complete message code to task management/abnormality processing module.All serial operations have all sent and have completed after message code, will start the task in next stage, upload serial operation.
Uploading serial operation, is exactly special ETL series operation in fact.Meet flow process (2), (3), (4).
Fig. 3 has provided the operation of MS-cloud platform and has uploaded flow process, and data upload, to intermediate server (MS) rear (intermediate server (MS) that is to say WEB server, is subordinate to same PC), will start executing data and upload to cloud platform task.Detailed process is as follows:
(1) MS-cloud platform is uploaded after job initiation, first scan the All Files of the fixing catalogue of intermediate server, total file number FileCounts is write in cloud platform metadata, global profile of initialization number variable FileCount in cloud platform metadata simultaneously, initial value is 0;
(2) the every class index file in intermediate server (MS) is uploaded at every turn and is taken a uploading channel, under normal circumstances, and when uploading, uploading the corresponding information that records upload file in metadata at every turn.Such as: upload file name, upload size, the total size of file etc.After having uploaded, a global profile number variable of uploading in metadata increases 1 certainly;
(3) before each file is uploaded, all can pass through checking module, checking module uses the rule of " protocol family " to check file.The general flow of detection module: first relatively the total number FileCounts of file and the global profile number variable FileCount in cloud platform metadata compares, if equated, exits MS-cloud platform and uploads operation.Otherwise, in cloud platform metadata, search the file that will upload, see if there is corresponding information.If no, can distribute a uploading channel, start file and upload.In the process that file is uploaded, can carry out Mission Monitor, the implementation status of Mission Monitor module monitors task writes daily record simultaneously;
(4) if detecting in file upload procedure, detection module occurs extremely, will carrying out abnormality processing.Abnormality processing operates in monitoring module, except recording necessary log information, also can store corresponding " current environment " simultaneously, for recovering abnormal front environmental information.Abnormality processing can not modified to the data in current cloud platform metadata.Such as, certain file has been uploaded a part, has then occurred interruption, namely abnormal.In cloud platform metadata, will store so the partial document information (as above passing the information such as interruption position) of having uploaded can not operate again to it.Can again send the message of a upload file that contains exception code simultaneously;
(5) each file is uploaded, and do not know that whether this article part is to upload to interrupt or do not uploaded (can't occurring the situation that file has been uploaded, because the instruction that only just meeting Transmit message is uploaded in starting first MS-cloud platform to upload operation and abnormal restarting), so in step (3) if in documentary information detected, will start so the function of breakpoint transmission, continue to upload the task of uploading of interruption;
(6) operation of MS-cloud platform is exited mouth in (3) step.
Claims (5)
1. a magnanimity power marketing data integration method of uploading based on ETL cloud platform, it comprises the magnanimity power marketing platform for data arrangement that some station servers form, and uses ETL instrument, characterized by further comprising following steps:
(1) data pick-up, will be stored in data pick-up in service related data storehouse out;
(2) data cleansing, utilizes ETL instrument to unify the service related data of isomery by writing corresponding operation Job processing procedure, removes different data layouts;
(3) data are returned, and carry out corresponding processing and obtain unified data and return, and are the achievement data that the modelling phase uses;
(4) data-switching, carries out service related data and uploads data to the transmission of intermediate server, adopts the storage of txt form;
(5) data upload, the fixing catalogue data of scanning intermediate server uploads to cloud platform data storage system automatically.
2. integration method according to claim 1, characterized by further comprising:
Service related data module, monitoring module, ETL work flow, protocol family module, intermediate server data memory module, protocol family are uploaded flow process and cloud platform data storage system modules, wherein
Described service related data module is each province and city database server modules;
Described monitoring module is mainly made up of monitoring operation and Platform Server monitoring;
Described ETL work flow refers to carries out corresponding Job processing procedure;
Described protocol family module comprises data scrubbing rule, data upload constraint, data processing rule, operation logic rules;
Described intermediate server data memory module refers to the operation of ETL series is uploaded to intermediate server MS by service related data, and adopts the storage of txt form;
Described protocol family is uploaded flow process and is referred to that the model index text above intermediate server MS uploads to cloud platform according to " protocol family " rule;
Described cloud platform data storage system modules mainly by HDFS file system and thereon build Hive and HBase database.
3. as follows according to its concrete steps of the integration method described in claim 1 or 2:
(1) MS-cloud platform is uploaded after job initiation, first scan the All Files of the fixing catalogue of intermediate server, total file number FileCounts is write in cloud platform metadata, global profile of initialization number variable FileCount in cloud platform metadata simultaneously, initial value is 0;
(2) the every class index file in intermediate server MS is uploaded at every turn and is taken a uploading channel, under normal circumstances, and when uploading, uploading the corresponding information that records upload file in metadata at every turn; After having uploaded, a global profile number variable of uploading in metadata increases 1 certainly;
(3) before each file is uploaded, all can pass through checking module, checking module uses the rule of " protocol family " to check file;
The general flow of detection module: first relatively the total number FileCounts of file and the global profile number variable FileCount in cloud platform metadata compares, if equated, exits MS-cloud platform and uploads operation; Otherwise, in cloud platform metadata, search the file that will upload, see if there is corresponding information; If no, can distribute a uploading channel, start file and upload; In the process that file is uploaded, can carry out Mission Monitor, the implementation status of Mission Monitor module monitors task writes daily record simultaneously;
(4) if detecting in file upload procedure, detection module occurs extremely, will carrying out abnormality processing; Abnormality processing operates in monitoring module, except recording necessary log information, also can store corresponding " current environment " simultaneously, for recovering abnormal front environmental information; Abnormality processing can not modified to the data in current cloud platform metadata; In cloud platform metadata, will store the partial document information of having uploaded so, can again not operate it; Can again send the message of a upload file that contains exception code simultaneously;
(5) each file is uploaded, and do not know that whether this article part is to upload to interrupt or do not uploaded, can't there is the situation that file has been uploaded, because the instruction that only just meeting Transmit message is uploaded in starting first MS-cloud platform to upload operation and abnormal restarting, so in step (3) if in documentary information detected, will start so the function of breakpoint transmission, continue to upload the task of uploading of interruption;
(6) operation of MS-cloud platform is exited mouth in (3) step.
4. integration system according to claim 1, is characterized in that drawing together:
Described monitoring module is uploaded flow process to the operation of ETL series and protocol family and is carried out monitoring management.
5. according to the integration system described in claim 1, it is characterized in that:
To ETL series, operation has effect of contraction to described protocol family module, and protocol family is uploaded to flow process directive function.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410180132.7A CN103955510A (en) | 2014-04-30 | 2014-04-30 | Massive electricity marketing data integration method uploaded by ETL cloud platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410180132.7A CN103955510A (en) | 2014-04-30 | 2014-04-30 | Massive electricity marketing data integration method uploaded by ETL cloud platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103955510A true CN103955510A (en) | 2014-07-30 |
Family
ID=51332785
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410180132.7A Pending CN103955510A (en) | 2014-04-30 | 2014-04-30 | Massive electricity marketing data integration method uploaded by ETL cloud platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103955510A (en) |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104408561A (en) * | 2014-11-25 | 2015-03-11 | 同程网络科技股份有限公司 | Job-station based order distributing method |
| CN104601723A (en) * | 2015-02-03 | 2015-05-06 | 中国南方电网有限责任公司 | Power marketing management system SOA framework based on internal service bus |
| CN104750861A (en) * | 2015-04-16 | 2015-07-01 | 中国电力科学研究院 | Method and system for cleaning mass data of energy storage power station |
| CN106599116A (en) * | 2016-11-30 | 2017-04-26 | 中国南方电网有限责任公司 | Cloud platform data integration management system and method |
| CN106777180A (en) * | 2016-12-22 | 2017-05-31 | 北京京东金融科技控股有限公司 | The method of high-performance distributed data conversion, apparatus and system |
| CN108280532A (en) * | 2017-09-02 | 2018-07-13 | 国网辽宁省电力有限公司 | A kind of improved power equipment asset management system and method |
| CN108733758A (en) * | 2018-04-11 | 2018-11-02 | 北京三快在线科技有限公司 | Hotel's static data method for pushing, device, electronic equipment and readable storage medium storing program for executing |
| CN109347874A (en) * | 2018-11-29 | 2019-02-15 | 杭州电力设备制造有限公司 | Electric network data method for uploading, device, system and storage medium based on cloud storage |
| CN110069297A (en) * | 2019-03-28 | 2019-07-30 | 平安科技（深圳）有限公司 | Abnormality eliminating method, device, computer equipment and storage medium based on Spring MVC |
| CN110515548A (en) * | 2019-08-15 | 2019-11-29 | 浙江万朋教育科技股份有限公司 | A method of avoiding waste third party cloud memory space |
| CN110750384A (en) * | 2019-10-15 | 2020-02-04 | 浙江众鑫空间科技有限公司 | Big data management system |
| CN112015799A (en) * | 2020-10-20 | 2020-12-01 | 平安国际智慧城市科技股份有限公司 | ETL task execution method and device, computer equipment and storage medium |
| CN116361389A (en) * | 2023-03-17 | 2023-06-30 | 国网江苏省电力有限公司营销服务中心 | Data synchronization link method and system based on national network marketing acquisition system |
| CN117933432A (en) * | 2024-02-05 | 2024-04-26 | 无锡市政公用新能源科技有限公司 | Intelligent new energy charging operation service management system based on cloud platform |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102646228A (en) * | 2012-02-23 | 2012-08-22 | 天津市电力公司 | Multi-service real-time data integration processing system and method of intelligent power grid |
| CN102930393A (en) * | 2012-10-25 | 2013-02-13 | 海南电网公司 | Comprehensive power grid information display visualization system |
-
2014
- 2014-04-30 CN CN201410180132.7A patent/CN103955510A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102646228A (en) * | 2012-02-23 | 2012-08-22 | 天津市电力公司 | Multi-service real-time data integration processing system and method of intelligent power grid |
| CN102930393A (en) * | 2012-10-25 | 2013-02-13 | 海南电网公司 | Comprehensive power grid information display visualization system |
Non-Patent Citations (3)
| Title |
|---|
| 朱重吉等: "ETL 技术在监测中心数据集成中的应用", 《广西电力》 * |
| 李文婧等: "广西电网数据中心配套ETL过程实现", 《广西电力》 * |
| 李文婧等: "数据中心广西电网配套ETL 过程实现", 《广西电业》 * |
Cited By (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104408561A (en) * | 2014-11-25 | 2015-03-11 | 同程网络科技股份有限公司 | Job-station based order distributing method |
| CN104601723A (en) * | 2015-02-03 | 2015-05-06 | 中国南方电网有限责任公司 | Power marketing management system SOA framework based on internal service bus |
| CN104601723B (en) * | 2015-02-03 | 2018-04-27 | 中国南方电网有限责任公司 | Power Marketing Management System SOA framework based on internal services bus |
| CN104750861A (en) * | 2015-04-16 | 2015-07-01 | 中国电力科学研究院 | Method and system for cleaning mass data of energy storage power station |
| WO2016165378A1 (en) * | 2015-04-16 | 2016-10-20 | 国网新源张家口风光储示范电站有限公司 | Energy storage power station mass data cleaning method and system |
| CN106599116A (en) * | 2016-11-30 | 2017-04-26 | 中国南方电网有限责任公司 | Cloud platform data integration management system and method |
| CN106777180B (en) * | 2016-12-22 | 2020-09-01 | 北京京东金融科技控股有限公司 | Method, device and system for high-performance distributed data conversion |
| CN106777180A (en) * | 2016-12-22 | 2017-05-31 | 北京京东金融科技控股有限公司 | The method of high-performance distributed data conversion, apparatus and system |
| CN108280532A (en) * | 2017-09-02 | 2018-07-13 | 国网辽宁省电力有限公司 | A kind of improved power equipment asset management system and method |
| CN108733758A (en) * | 2018-04-11 | 2018-11-02 | 北京三快在线科技有限公司 | Hotel's static data method for pushing, device, electronic equipment and readable storage medium storing program for executing |
| CN108733758B (en) * | 2018-04-11 | 2022-04-05 | 北京三快在线科技有限公司 | Hotel static data pushing method and device, electronic equipment and readable storage medium |
| CN109347874A (en) * | 2018-11-29 | 2019-02-15 | 杭州电力设备制造有限公司 | Electric network data method for uploading, device, system and storage medium based on cloud storage |
| CN110069297A (en) * | 2019-03-28 | 2019-07-30 | 平安科技（深圳）有限公司 | Abnormality eliminating method, device, computer equipment and storage medium based on Spring MVC |
| WO2020192134A1 (en) * | 2019-03-28 | 2020-10-01 | 平安科技（深圳）有限公司 | Exception handling method and apparatus based on spring mvc, and computer device and storage medium |
| CN110515548B (en) * | 2019-08-15 | 2021-04-06 | 浙江万朋教育科技股份有限公司 | Method for avoiding waste of third-party cloud storage space |
| CN110515548A (en) * | 2019-08-15 | 2019-11-29 | 浙江万朋教育科技股份有限公司 | A method of avoiding waste third party cloud memory space |
| CN110750384A (en) * | 2019-10-15 | 2020-02-04 | 浙江众鑫空间科技有限公司 | Big data management system |
| CN112015799A (en) * | 2020-10-20 | 2020-12-01 | 平安国际智慧城市科技股份有限公司 | ETL task execution method and device, computer equipment and storage medium |
| CN116361389A (en) * | 2023-03-17 | 2023-06-30 | 国网江苏省电力有限公司营销服务中心 | Data synchronization link method and system based on national network marketing acquisition system |
| CN116361389B (en) * | 2023-03-17 | 2024-03-08 | 国网江苏省电力有限公司营销服务中心 | Data synchronization link method and system based on national network marketing acquisition system |
| CN117933432A (en) * | 2024-02-05 | 2024-04-26 | 无锡市政公用新能源科技有限公司 | Intelligent new energy charging operation service management system based on cloud platform |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103955510A (en) | Massive electricity marketing data integration method uploaded by ETL cloud platform | |
| US9652723B2 (en) | Electrical transformer failure prediction | |
| CN114041112A (en) | Virtual storage system architecture | |
| CN107341205A (en) | A kind of intelligent distribution system based on big data platform | |
| CN104156810A (en) | Power dispatching production management system based on cloud computing and realization method of power dispatching production management system | |
| Zhang et al. | Research on hadoop-based enterprise file cloud storage system | |
| CN103067525A (en) | Cloud storage data backup method based on characteristic codes | |
| Zainab et al. | Big data management in smart grids: Technologies and challenges | |
| CN104660633A (en) | New media public service platform | |
| Gibadullin et al. | Service-oriented distributed energy data management using big data technologies | |
| CN109446230A (en) | A kind of big data analysis system and method for photovoltaic power generation influence factor | |
| Mohamed et al. | A review on big data management and decision-making in smart grid | |
| CN107818106B (en) | Big data offline calculation data quality verification method and device | |
| CN106056322A (en) | Smart grid scheduling system based on cloud computing | |
| Wu et al. | An Auxiliary Decision‐Making System for Electric Power Intelligent Customer Service Based on Hadoop | |
| Lee et al. | A big data management system for energy consumption prediction models | |
| Wang et al. | Storage and query of condition monitoring data in smart grid based on Hadoop | |
| Wang et al. | Block storage optimization and parallel data processing and analysis of product big data based on the hadoop platform | |
| Benhaddou et al. | Big data processing for smart grids | |
| Chen et al. | Big data storage architecture design in cloud computing | |
| Pan et al. | An open sharing pattern design of massive power big data | |
| Sun et al. | Cloud-based data analysis of user side in smart grid | |
| Zheng et al. | Speeding up processing data from millions of smart meters | |
| Lu et al. | A novel mass data processing framework based on Hadoop for electrical power monitoring system | |
| Alalawi et al. | A survey on cloud-based distributed computing system frameworks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20140730 |