CN113064866B - Power business data integration system - Google Patents
Power business data integration system Download PDFInfo
- Publication number
- CN113064866B CN113064866B CN202110226550.5A CN202110226550A CN113064866B CN 113064866 B CN113064866 B CN 113064866B CN 202110226550 A CN202110226550 A CN 202110226550A CN 113064866 B CN113064866 B CN 113064866B
- Authority
- CN
- China
- Prior art keywords
- data
- module
- common analysis
- information
- electricity
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000010354 integration Effects 0.000 title claims abstract description 28
- 238000004458 analytical method Methods 0.000 claims abstract description 75
- 230000005611 electricity Effects 0.000 claims abstract description 69
- 230000002159 abnormal effect Effects 0.000 claims abstract description 15
- 238000013179 statistical model Methods 0.000 claims abstract description 14
- 238000007781 pre-processing Methods 0.000 claims abstract description 4
- 238000006243 chemical reaction Methods 0.000 claims description 23
- 238000000034 method Methods 0.000 claims description 19
- 238000004140 cleaning Methods 0.000 claims description 12
- 230000008569 process Effects 0.000 claims description 11
- 238000001514 detection method Methods 0.000 claims description 6
- 238000011084 recovery Methods 0.000 claims description 5
- 230000004308 accommodation Effects 0.000 claims description 3
- 230000008859 change Effects 0.000 claims description 3
- 238000013499 data model Methods 0.000 claims description 3
- 238000013500 data storage Methods 0.000 claims 2
- 238000013480 data collection Methods 0.000 claims 1
- 238000012545 processing Methods 0.000 abstract description 4
- 238000005192 partition Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 241000533950 Leucojum Species 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/116—Details of conversion of file system types or formats
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/162—Delete operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/35—Clustering; Classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0631—Resource planning, allocation, distributing or scheduling for enterprises or organisations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/06—Energy or water supply
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Human Resources & Organizations (AREA)
- Economics (AREA)
- Strategic Management (AREA)
- General Business, Economics & Management (AREA)
- Marketing (AREA)
- Health & Medical Sciences (AREA)
- Entrepreneurship & Innovation (AREA)
- Tourism & Hospitality (AREA)
- Computer Security & Cryptography (AREA)
- Development Economics (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Game Theory and Decision Science (AREA)
- Human Computer Interaction (AREA)
- Educational Administration (AREA)
- Public Health (AREA)
- Water Supply & Treatment (AREA)
- General Health & Medical Sciences (AREA)
- Primary Health Care (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides an electric power business data integration system, which comprises a data acquisition module, a data processing module and a data processing module, wherein the data acquisition module is used for acquiring business data, daily meter reading data, meter reading abnormal data and customer service data of a user; the data integration module is used for calling the common analysis objects, integrating the common analysis objects according to a preset integration rule, integrating the data of the bottom library table related to each common analysis object in a cross-domain manner, and determining the association relation of the common analysis objects; the data summarizing module is used for summarizing the multidimensional according to a pre-stored preprocessing strategy and the association relation of each common analysis object to generate a common analysis object multidimensional statistical model; and the client full-dimension information module is used for determining the association relation of all relevant information of the electricity client in the accounting period according to the multi-dimensional statistical model of the common analysis object and generating a client full-dimension information model. The invention opens all islands related to the electricity fee accounting service and realizes full-dimension and full-direction display based on electricity customers.
Description
Technical Field
The invention relates to the technical field of power, in particular to a power business data integration system.
Background
At present, when an electric charge accounting service personnel performs accounting service, because a marketing system model is huge and complex, service data are scattered in different fields or system functions, when all information of a customer needs to be acquired, a plurality of functional pages need to be checked back and forth, even different service systems (such as the marketing system only has monthly electric quantity, and daily electric quantity data need to be checked from a metering automation system), the integration degree of the electric power service data is low, the data cannot be counted integrally, and therefore abnormal positions cannot be accurately positioned when electric charge checking is performed, and the electric charge checking work efficiency is greatly influenced. Therefore, an effective technical means is necessary to integrate and uniformly model data scattered in different fields or modules to form an information model based on the full dimension of the client.
Disclosure of Invention
The invention aims to provide an electric power business data integration system which solves the technical problems that the integration degree of electric power business data is low, the data cannot be effectively counted as a whole, and the electric charge checking work efficiency is affected in the existing method.
To achieve the above object, an embodiment of the present invention provides a power business data integration system, including:
the data acquisition module is used for acquiring business data, daily meter reading data, meter reading abnormal data and customer service data of a user from a plurality of data sources through the data interface;
the data integration module is used for calling common analysis objects from business data, daily meter reading data, meter reading abnormal data and customer service data, integrating the common analysis objects according to preset integration rules, and integrating the data of the bottom library table related to each common analysis object in a cross-domain manner to obtain a large-width table of each common analysis object; determining the association relation of the common analysis objects according to the large-width table of the common analysis objects; the common analysis objects at least comprise electricity customers, electricity consumption capacity, electricity sales capacity, electricity charge and electricity price, customer service work orders, customer service agents, telephone traffic records, industry expansion work orders and industry expansion matched items;
the data summarizing module is used for summarizing the multidimensional according to the pre-stored preprocessing strategy and the association relation of each common analysis object and outputting basic statistical indexes and service topics; generating a multi-dimensional statistical model of the common analysis object according to the basic statistical index and the business theme;
and the client full-dimension information module is used for determining the association relation of all relevant information of the electricity client in the accounting period according to the multi-dimensional statistical model of the common analysis object and generating a client full-dimension information model.
Preferably, the system further comprises a cache module for storing business data, daily meter reading data, meter reading abnormal data and customer service data of the collected users from a plurality of data sources; storing a multidimensional statistical model of the common analysis object output by the data summarization module; and storing the customer full-dimension information model generated by the customer full-dimension information module.
Preferably, the service data of the user at least includes: user files, business expansion work list basic information, meter changing information, meter reading information, metering point relation, transformer information, accounting transformer change information, accounting transformer compensation accommodation amount information, access electric quantity record, access electric quantity detail, metering point transformer relation, metering point electric quantity, metering price detail and rechecking work list information.
Preferably, the data acquisition module is further configured to analyze the service data, the daily meter reading data, the meter reading abnormal data and the customer service data, and store the service data according to a preset storage rule.
Preferably, the data summarization module further comprises:
the data cleaning module is used for identifying the same entity data record of each common analysis object from different data sources according to a preset target data conversion rule and detecting consistency of the common analysis objects which have the same entity data but come from different data sources;
the code conversion module is used for detecting a related data table and a field value which relate to code definition and code value in a common analysis object according to the data dimension and a preset code value definition standard; determining the association relation between the common analysis object record and the code according to the detection result;
the multi-table merging module is used for associating a plurality of service tables according to the association field, and realizing random interchange of rows and columns through merging of the plurality of service tables; and splitting the associated fields to generate stretching relation information of the wide table.
Preferably, the data cleaning module is further configured to generate difference information between the same common analysis objects in different data sources according to the consistency detection result, and generate data quality information;
and performing type conversion or value conversion on the common analysis object according to the preset conversion rule and the data quality information to form intermediate data which can be spliced and automatically mapped.
Preferably, the code conversion module is further configured to detect whether a standard code data value is within a preset reasonable value range, and determine that an illegal code value occurs when detecting that the standard code data value is not within the preset reasonable value range, so as to generate a data problem list and a corresponding scheme; and identifying a new added value in the dynamic code, and updating the corresponding code definition when the new added dynamic code value is identified.
Preferably, the client full-dimension information module further comprises:
the CUB cube module is used for generating core data cube data according to a preset core data cube and a core data model, scheduling related computing resources to execute cube computing and summarizing statistics; storing the cube data in a temporary cube table, comparing the new cube data with the existing cube data, and generating cube update according to the difference result;
the data loading module is used for loading the data output by the code conversion module, the data cleaning module and the CUB cube module into a temporary target data table, comparing the difference between the temporary target data table and the data in the existing target data table, and generating update information of the existing data table;
and the parallel scheduling module is used for: and the code conversion module, the data cleaning model, the CUB cube model and the data loading module are subjected to parallel scheduling according to task execution time through a preset scheduling rule.
Preferably, the customer full-dimension information module is further used for inputting a service or service application scene required by the electricity user into the customer full-dimension information model, outputting each bill of the electricity user, and establishing all relevant information in the process from electricity collection to electricity charge recovery.
In summary, the embodiment of the invention has the following beneficial effects:
the electric power business data integration system establishes a full data link from electric quantity collection to electric charge recovery for each bill of each electric power consumer, provides clear full life cycle view of the electric charge bill for the meter reading and collecting business personnel, and simultaneously provides complete information support for handling electric quantity and electric charge consumer complaints; and the electricity consumption clients are taken as centers, all islanding related to electricity consumption accounting services is achieved around the information such as client file information, monthly electricity consumption information, business expansion service information, customer service appeal information, electricity consumption rechecking information, meter reading information, real-time electricity consumption loads and the like, and full-dimensional display based on the electricity consumption clients is achieved.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions of the prior art, the drawings which are required in the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the description below are only some embodiments of the invention, and that it is within the scope of the invention to one skilled in the art to obtain other drawings from these drawings without inventive faculty.
Fig. 1 is a schematic structural diagram of an electric power business data integration system according to an embodiment of the invention.
FIG. 2 is a schematic diagram of a client full-dimension information model according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings, for the purpose of making the objects, technical solutions and advantages of the present invention more apparent.
Fig. 1 is a schematic diagram of an embodiment of a power business data integration system according to the present invention. In this embodiment, it includes:
the data acquisition module is used for acquiring business data, daily meter reading data, meter reading abnormal data and customer service data of a user from a plurality of data sources through the data interface; it can be understood that the collection and analysis and warehousing of the text file are realized by utilizing a flash (log collection system) to collect in real time and transmitting through Kfaka. Data is collected from the data sources and the collected data is sent to the designated destination. In order to ensure that the conveying process is successful, the data is cached before the data is sent to the destination, and the cached data is deleted after the data really arrives at the destination.
In a specific embodiment, data of each business system (or other related systems, or databases) such as a marketing management system, a customer comprehensive system, a metering automation system and the like are extracted by means of batch synchronization, incremental update and the like, and all the data are time stamped. For example, the following business data may be obtained from the marketing management system; the method specifically comprises the following steps: user files, business expansion work order basic information, meter changing information, meter reading information, metering point relation, transformer information, accounting transformer change information, accounting transformer compensation accommodation amount information, access electric quantity record, access electric quantity detail, metering point transformer relation, metering point electric quantity, metering price detail, rechecking work order information and the like. Daily meter reading data and meter reading abnormal data of public and private variable users can be obtained from a metering automation system and are used for assisting in electric quantity abnormal analysis; data such as customer appeal, customer service, etc. can be obtained from the customer's omnidirectional system.
Specifically, after data acquisition, the service data, the daily meter reading data and the meter reading abnormal data are required to be analyzed, the data are transmitted through a Kfaka technology to realize acquisition, analysis and warehousing of text files, specifically, a producer (a data source) sends a message to a Kafka cluster, and the message is classified (Topic), for example, two producers, one of which sends a message classified as Topic1 and the other sends a message of Topic2, and Topic is a theme, and the message can be classified by specifying the theme to the message, so that consumers can only pay attention to the message in the Topic required by themselves; the consumer constantly pulls messages from the clusters by establishing long connections with the kafka clusters, and then can process these messages. The customer service data is stored according to a preset storage rule; when a topic is created during storage, the number of partitions can be specified, the more the partitions are, the larger the throughput of the partition is, but the more resources are needed, the message is stored into different partitions according to the balancing strategy after the kafka receives the message sent by the producer. The method realizes the centralization of data resources in enterprises such as marketing, clients omnibearing and metering by batch offline data acquisition, batch real-time data acquisition and real-time stream data acquisition, and prepares for the next data integration; and the data is consistent with the data of each source end service system, and the integration and processing of the data are not performed.
The data integration module is used for calling the common analysis objects from business data, daily meter reading data, meter reading abnormal data and customer service data, integrating the common analysis objects according to a preset integration rule, integrating the data of the bottom library table related to each common analysis object in a cross-domain manner to obtain a large-width table of each common analysis object, wherein the large-width table refers to a database table in which indexes, dimensions and attributes related to business subjects are related together, the efficiency problem during iterative computation in the training process of the data mining model can be greatly improved by placing related fields in the same table, and it can be understood that the data types contained in the common analysis objects are identified, classified and stored through the data acquisition module, so that the indexes, dimensions and attributes related to each classified data can be clearly known, the large-width table for recording each data can be directly called, for example, the information such as the electricity consumption capacity, electricity price and the like of a certain electricity consumer can be obtained by calling the corresponding fields, and the electricity consumption can be obtained by calling the corresponding electricity consumption capacity and the dimensions, and the electricity price can be obtained by filtering the electricity consumption information; the association relation of the common analysis objects can be determined according to the large broad table of the common analysis objects; the common analysis objects at least comprise electricity customers, electricity consumption capacity, electricity sales capacity, electricity charge and electricity price, customer service work orders, customer service agents, telephone traffic records, industry expansion work orders and industry expansion matched items; it can be understood that the method is mainly used for offline storage of massive historical data on the basis of data caching, and meanwhile, the method realizes the concentration of processes of analysis object integration, data standard unification, data quality management and the like. The broad table is stored in the same table because different contents are stored in the broad table, and the broad table does not accord with the model design specification of three modes, so that the broad table has the advantages of improving the query performance and being convenient.
In a specific embodiment, the integration of the common analysis objects is performed according to the theme of the marketing and distribution data application. The data integration layer integrates the data of the bottom library table related to each analysis object in a cross-domain manner according to the common analysis objects to form a large-width table of the analysis objects such as the electricity consumption customer, the electricity consumption capacity, the electricity sales capacity and the like, and the large-width table comprises all common analysis dimension information of the data analysis objects; and reconstructing data according to the subdivision relation of the business main body, the business process and the business object and the association relation.
The data summarizing module is used for summarizing the multidimensional according to the pre-stored preprocessing strategy and the association relation of each common analysis object and outputting basic statistical indexes and service topics; generating a multi-dimensional statistical model of the common analysis object according to the basic statistical index and the business theme; it can be understood that the data integration module obtains the association relationship, so that the collection of the data multidimensional summarization and calculation process can be realized, for example, the electricity users are summarized according to the dimension of the electricity price of the electricity fee (such as the electricity price summarization standard is 1, 2, 3 yuan, etc.), and the electricity users with the electricity price of 1, 2, 3 can be obtained through the association relationship, so as to obtain summarized data taking the electricity price of the electricity fee as the dimension; for another example, the number indexes of the customer service worksheets can be summarized according to the time dimension to the level of 15 minutes, hours, days, weeks, ten days, months, seasons, half a year, years and the like, and the number indexes of the customer service worksheets are summarized in each level; similarly, summary data between any two parameters in each common analysis object, such as electricity consumer-electricity consumption capacity, electricity consumer-traffic records, etc., may be obtained. Forming a concentrated basic statistical index and a topic system (any parameter is a topic, and the corresponding parameter can form the basic statistical index; the statistical data corresponding to all the parameters form a data system); based on data cleaning and converted data detail, a statistical model of analysis objects such as electricity sales, electricity metering, customer service work orders, electricity customers and the like is built by a modeling mode of multidimensional models such as a star model, for example, any one of topics (one parameter in common analysis objects) is selected by determining the statistical data, other common analysis objects related to the statistical model are formed into a statistical model (the star model is the most direct embodiment, the star model takes the determined one parameter as the center, other parameters are in a dispersed shape around the star model, and the related forms can be expressed as other models such as the snowflake model, constellation model and the like in the same way), and summary statistics of all common dimensions is carried out for the data analysis topics. When the work orders are inquired or analyzed according to different time and units, the work order quantity indexes corresponding to the time granularity for summarizing can be inquired directly, data re-filtering screening and summarizing calculation are not needed, and response efficiency is improved.
Specifically, the data summarization module further includes: the data cleaning module is used for identifying the same entity data record of each common analysis object from different data sources according to a preset target data conversion rule and detecting consistency of the common analysis objects which have the same entity data but come from different data sources; the method is also used for generating difference information between the same common analysis objects in different data sources according to the consistency detection result to generate data quality information; and performing type conversion or value conversion on the common analysis object according to the preset conversion rule and the data quality information to form intermediate data which can be spliced and automatically mapped.
The code conversion module is used for detecting a related data table and a field value which relate to code definition and code value in a common analysis object according to the data dimension and a preset code value definition standard; determining the association relation between the common analysis object record and the code according to the detection result; the method is also used for detecting whether the standard coded data value is in a preset reasonable value range, and judging that an illegal coded value appears when the standard coded data value is not in the preset reasonable value range, so as to generate a data problem list and a corresponding scheme; and identifying a new added value in the dynamic code, and updating the corresponding code definition when the new added dynamic code value is identified.
The multi-table merging module is used for associating a plurality of service tables according to the association field, realizing random interchange of rows and columns through merging the plurality of service tables, for example, 2 or more than 2 tables are associated according to the association field in the forms of left connection, right connection, full connection, internal connection and the like by taking service as a drive, and realizing random interchange of rows and columns; and splitting the associated fields to generate stretching relation information of the wide table, so as to realize the stretching function of the wide table.
And the client full-dimension information module is used for determining the association relation of all relevant information of the electricity client in the accounting period according to the multi-dimensional statistical model of the common analysis object and generating a client full-dimension information model, as shown in fig. 2. It can be understood that data fusion and sharing are performed for business and application scenes, so that data value integration is realized, and data service capability opening and business service capability opening of marketing and distribution data marts are completed.
In a specific embodiment, the customer full-dimension information module is further configured to input a service or service application scenario required by the electricity user into the customer full-dimension information model, and output each bill of the electricity user to establish all relevant information in the process from electricity collection to electricity charge recovery; it can be appreciated that from the analysis perspective, the user usually only needs to pay attention to the service domain topic analysis service provided by the data mart, and from the application perspective, the user can select different types of data products in the same topic domain according to specific application scenarios.
Specifically, the client full-dimension information module further includes: the CUB cube module is used for generating core data cube data according to a preset core data cube and a core data model, scheduling related computing resources to execute cube computing and summarizing statistics; storing the cube data in a temporary cube table, comparing the new cube data with the existing cube data, and generating cube update according to the difference result; and generating a cube updating script according to the difference analysis result, and scheduling related computing resources to execute a cube updating task.
The data loading module is used for loading the data output by the code conversion module, the data cleaning module and the CUB cube module into a temporary target data table, comparing the difference between the temporary target data table and the data in the existing target data table, and generating update information of the existing data table; and taking the target model as a drive, loading the source data subjected to dimension conversion and data cleaning and the CUB cube into a temporary target data table, comparing the difference between the temporary target data table and the data in the existing target data table, and automatically generating an execution script for updating the existing data table according to the difference comparison result. The system dispatch related computing resource distributes script execution tasks to different processing units for execution, records the execution result and processes various exceptions in the execution process.
And the parallel scheduling module is used for: the method comprises the steps of performing parallel scheduling on a code conversion module, a data cleaning model, a CUB cube model and a data loading module according to task execution time length through a preset scheduling rule; optimizing according to the task execution time length, and scheduling in a parallel mode to shorten the overall operation period of the scheduled task.
The caching module is used for storing business data, daily meter reading data, meter reading abnormal data and customer service data of a user acquired from a plurality of data sources; storing a multidimensional statistical model of the common analysis object output by the data summarization module; and storing the customer full-dimension information model generated by the customer full-dimension information module.
In summary, the embodiment of the invention has the following beneficial effects:
the electric power business data integration system provided by the invention establishes a full data link from electric quantity collection to electric charge recovery for each bill of each electricity consumer, provides clear full life cycle view of the electric charge bill for the meter reading and checking business personnel, and simultaneously provides complete information support for handling electric quantity and electric charge customer complaints.
And the electricity consumption clients are taken as centers, all islanding related to electricity consumption accounting services is achieved around the information such as client file information, monthly electricity consumption information, business expansion service information, customer service appeal information, electricity consumption rechecking information, meter reading information, real-time electricity consumption loads and the like, and full-dimensional display based on the electricity consumption clients is achieved.
The foregoing disclosure is illustrative of the present invention and is not to be construed as limiting the scope of the invention, which is defined by the appended claims.
Claims (5)
1. An electrical power business data integration system, comprising:
the data acquisition module is used for acquiring business data, daily meter reading data, meter reading abnormal data and customer service data of a user from a plurality of data sources through the data interface;
the data integration module is used for calling common analysis objects from business data, daily meter reading data, meter reading abnormal data and customer service data, integrating the common analysis objects according to preset integration rules, and integrating the data of the bottom library table related to each common analysis object in a cross-domain manner to obtain a large-width table of each common analysis object; determining the association relation of the common analysis objects according to the large-width table of the common analysis objects; the common analysis objects at least comprise electricity customers, electricity consumption capacity, electricity sales capacity, electricity charge and electricity price, customer service work orders, customer service agents, telephone traffic records, industry expansion work orders and industry expansion matched items;
the data summarizing module is used for summarizing the multidimensional according to the pre-stored preprocessing strategy and the association relation of each common analysis object and outputting basic statistical indexes and service topics; generating a multi-dimensional statistical model of the common analysis object according to the basic statistical index and the business theme;
the client full-dimension information module is used for determining the association relation of all relevant information of the electricity client in the accounting period according to the multi-dimensional statistical model of the common analysis object and generating a client full-dimension information model; the system comprises a data source, a cache module, a data storage module and a data storage module, wherein the data source is used for acquiring business data, daily meter reading data, meter reading abnormal data and customer service data of a user; storing a multidimensional statistical model of the common analysis object output by the data summarization module; storing a customer full-dimension information model generated by a customer full-dimension information module;
wherein, the business data of the user at least comprises: user files, basic information of business expansion work orders, meter changing information, meter reading information, metering point relation, transformer information, accounting transformer change information, accounting transformer compensation accommodation amount information, access electric quantity record, access electric quantity detail, metering point transformer relation, metering point electric quantity, metering price detail and rechecking work order information;
the data summarization module further comprises:
the data cleaning module is used for identifying the same entity data record of each common analysis object from different data sources according to a preset target data conversion rule and detecting consistency of the common analysis objects which have the same entity data but come from different data sources;
the code conversion module is used for detecting a related data table and a field value which relate to code definition and code value in a common analysis object according to the data dimension and a preset code value definition standard; determining the association relation between the common analysis object record and the code according to the detection result;
the multi-table merging module is used for associating a plurality of service tables according to the association field, and realizing random interchange of rows and columns through merging of the plurality of service tables; splitting the associated field to generate stretching relation information of the wide table;
the client full-dimension information module further comprises:
the CUB cube module is used for generating core data cube data according to a preset core data cube and a core data model, scheduling related computing resources to execute cube computing and summarizing statistics; storing the cube data in a temporary cube table, comparing the new cube data with the existing cube data, and generating cube update according to the difference result;
the data loading module is used for loading the data output by the code conversion module, the data cleaning module and the CUB cube module into a temporary target data table, comparing the difference between the temporary target data table and the data in the existing target data table, and generating update information of the existing data table;
the parallel scheduling module is used for carrying out parallel scheduling on the code conversion module, the data cleaning model, the CUB cube model and the data loading module according to the task execution time length through a preset scheduling rule.
2. The system of claim 1, wherein the data collection module is further configured to parse the business data, the daily meter reading data, the meter reading anomaly data, and the customer service data and store the business data, the daily meter reading data, the meter reading anomaly data, and the customer service data according to a preset storage rule.
3. The system of claim 2, wherein the data cleansing module is further configured to generate difference information between the same common analysis object in different data sources according to the consistency detection result, and generate data quality information;
and performing type conversion or value conversion on the common analysis object according to the preset conversion rule and the data quality information to form intermediate data which can be spliced and automatically mapped.
4. The system of claim 3, wherein the transcoding module is further configured to detect whether the standard encoded data value is within a preset reasonable value range, and determine that an illegal encoded value occurs when the standard encoded data value is detected not to be within the preset reasonable value range, so as to generate a data problem list and a corresponding scheme; and identifying a new added value in the dynamic code, and updating the corresponding code definition when the new added dynamic code value is identified.
5. The system of claim 4, wherein the customer full-dimension information module is further configured to input a service or service application scenario required by the electricity consumer into the customer full-dimension information model, and output each bill of the electricity consumer to establish all relevant information in the process from electricity collection to electricity fee recovery.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110226550.5A CN113064866B (en) | 2021-03-02 | 2021-03-02 | Power business data integration system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110226550.5A CN113064866B (en) | 2021-03-02 | 2021-03-02 | Power business data integration system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113064866A CN113064866A (en) | 2021-07-02 |
| CN113064866B true CN113064866B (en) | 2024-03-01 |
Family
ID=76559464
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110226550.5A Active CN113064866B (en) | 2021-03-02 | 2021-03-02 | Power business data integration system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113064866B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113342798B (en) * | 2021-07-07 | 2022-10-21 | 广东电网有限责任公司 | Data management system based on data fusion |
| CN113627821A (en) * | 2021-08-25 | 2021-11-09 | 深圳供电局有限公司 | Method and system for identifying abnormal electricity utilization based on electricity utilization behavior characteristics |
| CN113743894A (en) * | 2021-08-30 | 2021-12-03 | 深圳供电局有限公司 | Method and system for establishing rechecking rule model for rechecking electric bill |
| CN113761036A (en) * | 2021-09-07 | 2021-12-07 | 国网福建省电力有限公司经济技术研究院 | Power grid statistics professional index multidimensional association table query display method |
| CN114267124A (en) * | 2021-12-22 | 2022-04-01 | 深圳供电局有限公司 | Charging information processing method and system |
| CN115858372B (en) * | 2022-12-19 | 2024-04-26 | 上海浦东发展银行股份有限公司 | Batch data construction and automatic verification method and system based on OLAP system |
| CN116450717B (en) * | 2023-03-14 | 2023-10-20 | 深圳市英捷软件有限公司 | Data integration method and information management system for cross-service modules |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104915764A (en) * | 2015-05-29 | 2015-09-16 | 中国南方电网有限责任公司 | Electric power marketing management system based on client-user-account model and application thereof |
| CN106777244A (en) * | 2016-12-27 | 2017-05-31 | 国网浙江象山县供电公司 | A kind of power customer electricity consumption behavior analysis method and system |
| CN109669934A (en) * | 2018-12-11 | 2019-04-23 | 江苏瑞中数据股份有限公司 | A kind of data warehouse and its construction method suiting electric power customer service |
| CN110163621A (en) * | 2018-02-10 | 2019-08-23 | 广州供电局有限公司 | A kind of electric power customer service big data DSS |
| CN112395361A (en) * | 2020-12-04 | 2021-02-23 | 华南理工大学 | Electric power engineering construction block chain management system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020202189A1 (en) * | 2019-03-29 | 2020-10-08 | Bert Labs Private Limited | Self-service artificial intelligence platform leveraging data-based and physics-based models for providing real-time controls and recommendations |
-
2021
- 2021-03-02 CN CN202110226550.5A patent/CN113064866B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104915764A (en) * | 2015-05-29 | 2015-09-16 | 中国南方电网有限责任公司 | Electric power marketing management system based on client-user-account model and application thereof |
| CN106777244A (en) * | 2016-12-27 | 2017-05-31 | 国网浙江象山县供电公司 | A kind of power customer electricity consumption behavior analysis method and system |
| CN110163621A (en) * | 2018-02-10 | 2019-08-23 | 广州供电局有限公司 | A kind of electric power customer service big data DSS |
| CN109669934A (en) * | 2018-12-11 | 2019-04-23 | 江苏瑞中数据股份有限公司 | A kind of data warehouse and its construction method suiting electric power customer service |
| CN112395361A (en) * | 2020-12-04 | 2021-02-23 | 华南理工大学 | Electric power engineering construction block chain management system |
Non-Patent Citations (1)
| Title |
|---|
| "大数据环境下的电力客户服务数据分析***";周文琼;《计算机***应用》;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113064866A (en) | 2021-07-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113064866B (en) | Power business data integration system | |
| CN104820670B (en) | A kind of acquisition of power information big data and storage method | |
| CN109582667A (en) | A kind of multiple database mixing storage method and system based on power regulation big data | |
| CN108470228A (en) | Financial data auditing method and audit system | |
| CN109582717A (en) | A kind of database unified platform and its read method towards electric power big data | |
| CN115934680B (en) | One-stop big data analysis processing system | |
| CN111552813A (en) | Power knowledge graph construction method based on power grid full-service data | |
| CN113822547B (en) | Comprehensive information platform system for refined demand side management and management method | |
| CN111159180A (en) | Data processing method and system based on data resource directory construction | |
| CN112347071B (en) | Power distribution network cloud platform data fusion method and power distribution network cloud platform | |
| CN112131203A (en) | Method and system for building data warehouse | |
| CN111753034A (en) | One-stop type geographical big data platform | |
| Li | Research on big data analysis data acquisition and data analysis | |
| CN111127099A (en) | E-commerce user analysis system based on big data and analysis method thereof | |
| CN111737325A (en) | Power data analysis method and device based on big data technology | |
| CN111126852A (en) | BI application system based on big data modeling | |
| CN112817958A (en) | Electric power planning data acquisition method and device and intelligent terminal | |
| Dai | Designing an accounting information management system using big data and cloud technology | |
| CN116402187A (en) | Enterprise pollution discharge prediction method based on power big data | |
| Hu | E-commerce big data computing platform system based on distributed computing logistics information | |
| CN117573646A (en) | Data management method and system based on dimension modeling | |
| CN112148261A (en) | Data center platform design method of intelligent shipyard digital service platform | |
| CN116644136A (en) | Data acquisition method, device, equipment and medium for increment and full data | |
| CN116795816A (en) | Stream processing-based multi-bin construction method and system | |
| Dong et al. | Research on Architecture of Power Big Data High-Speed Storage System for Energy Interconnection |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |