CN115689788A - Financial data analysis method - Google Patents
Financial data analysis method Download PDFInfo
- Publication number
- CN115689788A CN115689788A CN202211377219.4A CN202211377219A CN115689788A CN 115689788 A CN115689788 A CN 115689788A CN 202211377219 A CN202211377219 A CN 202211377219A CN 115689788 A CN115689788 A CN 115689788A
- Authority
- CN
- China
- Prior art keywords
- data
- financial
- module
- management
- platform
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a financial data analysis method, which comprises a key index billboard module, a professional management center module, a basic service center module, a financial team management module, a rds database, a logstack data collection engine, a flink processing engine, a binlog, a topic message set, an odps data processing platform, a datahub metadata platform and a display terminal, wherein the key index billboard module is connected with the professional management center module; the invention has the beneficial effects that: the unified management of the daily business indexes of the company level and the basic level is realized, the management report is issued in time, the risk is effectively avoided by financial management personnel, and the analysis and decision requirements of the company financial management personnel are met; the construction of four modules of a key index billboard, a professional management center, a basic service center and a financial team management is developed, so that financial staff can develop auxiliary decision-making work such as business process management and control, index analysis, report analysis and the like on one platform, the barrier of extracting data for analysis across multiple systems is broken, and the working efficiency is greatly improved.
Description
Technical Field
The invention belongs to the technical field of financial data analysis, and particularly relates to a financial data analysis method.
Background
The method comprises the following steps that most of the existing analysis scenes are based on off-line warehouse counting, source end data are extracted every day at regular time, and correlation analysis is performed on data from each system at regular time in the warehouse; data is extracted by adopting datax, distributed storage is realized by adopting hdfs, most calculation engines adopt mapreduce, and analysis and calculation capacity on a large-scale data set is provided by adopting first division and second division, then reduction and combination.
Based on the requirements of various operation indexes of the current state network, the internal management of a company is improved, the business standardization management and control are realized, effective management means is needed for supporting and making decisions, the indexes and the business management and control are preposed, problems are found in time through the building of a platform, and the indexes and the management improvement requirements are met in advance.
When obtaining data, current financial staff need log in a plurality of systems to inquire and export, so that the workload of the financial staff is greatly increased, and the accuracy and the real-time performance of the data cannot be guaranteed.
Management reports and data analysis means independently developed in systems such as financial ERP (enterprise resource planning), financial management and control and the like are limited by factors such as performance and business volume of a source system, and for engineering financial management reports, the management reports cannot be timely presented due to large data volume and complex logic, and results can be queried often within several days.
The establishment of the data middle platform breaks through the original data island, financial staff do not need to log in a plurality of systems to acquire data, and one system meets the requirement of a user for obtaining and exporting data; however, in the calculation engine at the present stage, mapreduce is subjected to task division, so that time consumption in resource allocation is large, and the calculation engine is only suitable for calculating mass off-line and stock data; in some urgent needs, expectation is seen, and when decision adjustment is influenced, the time efficiency of T +1 obviously cannot meet business requirements, so that the method has a promising significance for time efficiency upgrade of a calculation engine and data consumption.
Disclosure of Invention
The invention aims to provide a financial data analysis method, which has urgent requirements on deep mining of data value, enterprise management by data and business driving by information, provides online real-time computing capability, provides quick and accurate data service for financial data application, supports the requirement of financial lean management, and enables a leader decision-making layer to see the influence caused by adjusting parameters in time.
In order to achieve the purpose, the invention provides the following technical scheme: a financial data analysis method comprises a key index board module, a professional management center module, a basic service center module, a financial team management module, a rds database, a logstack data collection engine, a flink processing engine, a binlog log, a topic message set, an odps data processing platform, a datahub metadata platform and a display terminal, wherein the key index board module comprises a key index board, a professional management center module, a basic service center module, a financial team management module, a rds database, a logstack data collection engine, a flink processing engine, a binlog log, a topic message set, an odps data processing platform, a datahub metadata platform and a display terminal, and the key index board module comprises the following steps:
the company leader can master the company operation index and key index information in time by using the key index billboard module; the professional management center module is applied to each room of the financial department of the provincial company, the professional index management and control capability is improved, the report analysis efficiency is accelerated, the original offline management means is changed by applying the financial team management module, the financial department leader can timely master the information of financial staff in a multi-dimensional manner, and an effective aid decision-making means is provided for the management of the financial staff; the financial staff of the basic unit applies the basic service center module to improve the lean level of business management, promote the information construction progress of the basic unit and improve the information management and control effect of the basic unit;
the rds database is used for realizing the storage of data;
the logstack data collection engine is configured to dynamically unify data from different data sources and normalize the data to a selected destination;
the flink processing engine is used for carrying out state calculation on unbounded and bounded data streams; flink is a distributed processing framework capable of supporting high throughput, low latency, high performance simultaneously; and (3) base stone: checkpoint, state, time, window;
the binlog log is used for recording all updated data or potential updated data; the change of the database is recorded, and the binlog can be used for recovering the data of misoperation, synchronizing a master database and a slave database, and monitoring and distributing the data change; three modes of binlog:
statement-based replication: each sql which can modify data is recorded in the binlog without recording the change of each line, so that the binlog amount is reduced, IO is saved, and the performance is improved;
line-based replication: the context related information of the sql statement is not recorded, only which record is stored and modified is stored, the details of modification of each row of data are clearly recorded, and the problems that the storage process or function and the calling and triggering of trigger cannot be copied correctly under certain specific conditions can not occur;
the mixing mode is as follows: only sql sentences can be recorded in ambiguous sentences generated from the library, and the copying of the ambiguous sentences by using rows can be generated, so that the advantages of the two sentences are taken into consideration;
the topic message set is used for collecting messages; different Topic messages are stored separately, each Topic can have a plurality of producers to send messages to it, or a plurality of consumers to consume the messages;
the odps data processing platform is used for realizing data processing;
the database metadata platform is used for managing metadata; the datahub is open based on Apache License 2, adopts a push-based data collection architecture to support pull, and can continuously collect the changed metadata; the datahub is an end-to-end metadata discovery tool and can help data managers to mine the value of the company data; performing concentrated query search on metadata in data assets such as a database, a data lake, a BI platform, ML feature storage, workflow configuration and the like; the end-to-end journey of the data is easily understood through cross-platform, data set and pipeline blood relationship tracking; rapidly acquiring the context of the related entity through a linear blood relationship graph; acquiring exact information of the accuracy and the correlation of the data set;
the display terminal is used for displaying data;
the analysis method is as follows:
the method comprises the following steps: a user foreground inputs parameters according to service requirements, and the input parameters can be written into an rds database;
step two: the logstack data collection engine collects binlog logs of the rds database in real time, monitors the metadata platform transmitted into the database, and writes target data into a topic message set corresponding to a message queue;
step three: the flink processing engine monitors data in the topic message aggregate stream, reads the tables in the rds database or odps data processing platform, immediately performs correlation calculation on parameters and source table data when the data reach the consumption time point, and writes the data into the rds database in real time for the foreground page display of the display terminal.
As a preferred technical solution of the present invention, the system further includes a parameter modification marking module, and the parameter modification marking module marks the modified parameter.
As a preferred technical solution of the present invention, the datahub metadata platform provides metadata retrieval, data discovery, data monitoring and data supervision.
As a preferred technical scheme of the invention, the financial data security monitoring system further comprises a monitoring terminal, and the financial data security monitoring system can monitor the financial data working condition through the monitoring terminal to ensure the data security.
As a preferable technical scheme, the system further comprises a financial report generating module, and the financial report generating module is used for generating a financial report so as to facilitate the viewing of a user.
As a preferred technical solution of the present invention, the present invention further includes a firewall module, and the data security is further ensured by the firewall module.
As a preferred technical solution of the present invention, the system further includes a scheduling processing module, and the scheduling processing module is configured to perform scheduling processing on the financial data.
As a preferred technical solution of the present invention, the display terminal is an intranet computer.
Compared with the prior art, the invention has the beneficial effects that:
1. by the construction of a financial professional analysis application scene, the unified interface, the unified data and the unified dimension of financial management application are realized, the unified management of the daily business indexes of a company level and a base level is realized, a management report is issued in time, a financial manager effectively avoids risks, and the analysis and decision requirements of the financial manager of a company are met;
2. the construction of four modules of a key index billboard, a professional management center, a basic service center and a financial team management is developed, so that financial staff can develop auxiliary decision-making work such as business process management and control, index analysis, report analysis and the like on one platform, the barrier of extracting data for analysis across a plurality of systems is broken, and the working efficiency is greatly improved;
3. reading source end data in real time, writing data change into a datahub metadata platform in real time, monitoring a consumption topic message set in real time by a flink processing engine, and writing related calculation into an rds database for displaying a foreground page of a display terminal;
4. the timeliness of original data, namely T +1, is improved to be real-time updating, and the offline data counting is upgraded to be real-time data counting; the business capability of 'instant click and instant go' is provided for each specialty, each basic level unit and external partners of the company, and the intelligent operation and new business innovation capability of the company is improved; the centralized convergence of cross-service and cross-unit data is realized, the value analysis and mining capacity is improved, and an omnibearing data support is provided for the quality improvement and efficiency enhancement of a company;
5. the financial data working condition is monitored through the monitoring terminal, and the safety of data is ensured; the financial report is generated through the financial report generating module, so that the user can conveniently check the financial report; the data is further ensured to be safe through the firewall module.
Drawings
FIG. 1 is a data flow diagram of the present invention;
FIG. 2 is a flow chart of the analysis method of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1 and fig. 2, a first embodiment of the present invention provides a financial data analysis method, which includes a key index kanban module, a professional management center module, a basic service center module, a financial team management module, a rds database, a logstack data collection engine, a flink processing engine, a binlog log, a topic message set, an odps data processing platform, a datahub metadata platform, and a display terminal, where:
the company leader can master the company operation index and key index information in time by using the key index billboard module; the professional management center module is applied to each room of the financial department of the provincial company, the professional index management and control capability is improved, the report analysis efficiency is accelerated, the original offline management means is changed by applying the financial team management module, the financial department leader can timely master the information of financial staff in a multi-dimensional manner, and an effective aid decision-making means is provided for the management of the financial staff; the financial staff of the basic unit applies the basic service center module to improve the lean level of business management, promote the information construction progress of the basic unit and improve the information management and control effect of the basic unit;
the rds database is used for realizing the storage of data;
the logstash data collection engine is used to dynamically unify data from different data sources and normalize the data to a selected destination;
the flink processing engine is used for carrying out state calculation on the unbounded data flow and the bounded data flow; flink is a distributed processing framework capable of supporting high throughput, low latency, high performance simultaneously; and (3) base stone: checkpoint, state, time, window;
the binlog log is used for recording all updated data or potential updated data; the change of the database is recorded, and the binlog can be used for recovering the data of misoperation, synchronizing a master database and a slave database, and monitoring and distributing the data change; three modes of binlog:
statement-based replication: each sql which can modify data is recorded in the binlog without recording the change of each line, so that the binlog amount is reduced, IO is saved, and the performance is improved;
line-based replication: the context related information of the sql statement is not recorded, only which record is stored and modified is stored, the details of modification of each row of data are clearly recorded, and the problems that the storage process or function and the calling and triggering of trigger cannot be copied correctly under certain specific conditions can not occur;
the mixing mode is as follows: only sql sentences can be recorded in ambiguous sentences generated from the library, and the copying of the ambiguous sentences by using rows can be generated, so that the advantages of the two sentences are taken into consideration;
the topic message set is used for collecting messages; different Topic messages are stored separately, each Topic can have a plurality of producers to send messages to it, or a plurality of consumers to consume the messages;
the odps data processing platform is used for realizing data processing;
the database metadata platform is used for managing metadata; the method comprises the following steps that (1) the datahub is open-sourced based on Apache License 2, a pull pulling mode is supported by a push-based data collection architecture, and variable metadata can be collected continuously; the datahub is an end-to-end metadata discovery tool and can help data managers to mine the value of the company data; performing concentrated query search on metadata in data assets such as a database, a data lake, a BI platform, ML feature storage, workflow configuration and the like; the end-to-end journey of the data is easily understood through cross-platform, data set and pipeline blood relationship tracking; rapidly acquiring the context of the related entities through a linear blood relation graph; acquiring exact information of the accuracy and the correlation of the data set;
the display terminal is used for displaying data;
the analysis method is as follows:
the method comprises the following steps: the user foreground inputs parameters according to the service requirement, and the input parameters can be written into the rds database;
step two: the logstash data collection engine collects binlog logs of the rds database in real time, monitors and transmits the binlog logs into a database metadata platform, and writes target data into a topic message set corresponding to a message queue;
step three: the flink processing engine monitors data in the topic message aggregate stream, reads the tables in the rds database or odps data processing platform, immediately performs correlation calculation on parameters and source table data when the data reach the consumption time point, and writes the data into the rds database in real time for the foreground page display of the display terminal.
In this embodiment, it is preferable that the apparatus further includes a parameter modification marking module, and the parameter modification marking module marks the modified parameter.
In this embodiment, preferably, the datahub metadata platform provides metadata retrieval, data discovery, data monitoring, and data policing.
In this embodiment, preferably, the system further comprises a monitoring terminal, and the monitoring terminal monitors the financial data working condition to ensure the safety of the data.
In this embodiment, it is preferable that the system further includes a financial report generation module, and the financial report generation module generates a financial report, which is convenient for a user to view.
In this embodiment, it is preferable that the data security system further includes a firewall module, and the firewall module further ensures security of the data.
In this embodiment, preferably, the system further includes a scheduling processing module, and the scheduling processing module is configured to perform scheduling processing on the financial data.
In this embodiment, preferably, the display terminal is an intranet computer.
Although embodiments of the present invention have been shown and described, with particular reference to the foregoing detailed description, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (8)
1. A method of financial data analysis, comprising: including key index kanban module, professional management center module, basic level service center module, financial team management module, rds database, logstack data collection engine, flink processing engine, binlog log, topic message set, odps data processing platform, database metadata platform, display terminal, wherein:
the key index billboard module is used for mastering company operation indexes and key index information;
the professional management center module is used for professional index management and control and report analysis;
the basic service center module is used for information management and control of basic units;
the financial team management module masters financial staff information;
the rds database is used for realizing the storage of data;
the logstack data collection engine is used for dynamically unifying data from different data sources and standardizing the data to a selected destination;
the flink processing engine is used for carrying out state calculation on unbounded and bounded data streams;
the binlog log is used for recording all updated data or potential updated data;
the topic message set is used for collecting messages;
the odps data processing platform is used for realizing data processing;
the database metadata platform is used for managing metadata;
the display terminal is used for displaying data;
the analysis method is as follows:
the method comprises the following steps: the user foreground inputs parameters according to the service requirement, and the input parameters can be written into the rds database;
step two: the logstack data collection engine collects binlog logs of the rds database in real time, monitors the metadata platform transmitted into the database, and writes target data into a topic message set corresponding to a message queue;
step three: the flink processing engine monitors data in the topic message aggregate flow, reads the table in the rds database or the odps data processing platform, immediately performs correlation calculation on parameters and source table data when the data reach consumption time, and writes the data into the rds database in real time for a foreground page of the display terminal to display.
2. A method of financial data analysis according to claim 1 wherein: the system also comprises a parameter modification marking module, and the modified parameters are marked through the parameter modification marking module.
3. A method of financial data analysis according to claim 1 wherein: the datahub metadata platform provides metadata retrieval, data discovery, data monitoring, and data policing.
4. A method of financial data analysis according to claim 1 wherein: still include monitor terminal, monitor the financial data behavior through monitor terminal.
5. A method of financial data analysis according to claim 1 wherein: the system also comprises a financial report generating module, and the financial report is generated through the financial report generating module.
6. A method of financial data analysis according to claim 1 wherein: the system also comprises a firewall module, and the data security is ensured through the firewall module.
7. A method of financial data analysis according to claim 1 wherein: the system also comprises a scheduling processing module which is used for scheduling the financial data.
8. A method of financial data analysis according to claim 1 wherein: the display terminal is an intranet computer.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211377219.4A CN115689788A (en) | 2022-11-04 | 2022-11-04 | Financial data analysis method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211377219.4A CN115689788A (en) | 2022-11-04 | 2022-11-04 | Financial data analysis method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115689788A true CN115689788A (en) | 2023-02-03 |
Family
ID=85050892
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211377219.4A Pending CN115689788A (en) | 2022-11-04 | 2022-11-04 | Financial data analysis method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115689788A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116628722A (en) * | 2023-06-06 | 2023-08-22 | 重庆交通大学 | Financial data safety management processing system |
-
2022
- 2022-11-04 CN CN202211377219.4A patent/CN115689788A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116628722A (en) * | 2023-06-06 | 2023-08-22 | 重庆交通大学 | Financial data safety management processing system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101105793A (en) | Data processing method and system of data library | |
| CN113179173B (en) | Operation and maintenance monitoring system for expressway system | |
| US11681840B1 (en) | System and method for transitioning between executing models | |
| CN109829615B (en) | Target task multistage monitoring device and method based on proprietary cloud | |
| CN112527886A (en) | Data warehouse system based on urban brain | |
| CN111080261A (en) | Visual data asset management system based on big data | |
| CN114218218A (en) | Data processing method, device and equipment based on data warehouse and storage medium | |
| CN111538720B (en) | Method and system for cleaning basic data of power industry | |
| CN113642299A (en) | One-key generation method based on power grid statistical form | |
| CN117290180A (en) | Monitoring method, equipment and medium based on time sequence data analysis equipment operation state | |
| CN106355489A (en) | Data center system and data processing method for management | |
| CN115689788A (en) | Financial data analysis method | |
| CN117149873A (en) | Data lake service platform construction method based on flow batch integration | |
| CN111913933A (en) | Power grid historical data management method and system based on unified support platform | |
| Xu et al. | Cloud computing boosts business intelligence of telecommunication industry | |
| CN108595480A (en) | A kind of big data ETL tool systems and application process based on cloud computing | |
| CN116303814A (en) | Digital bin system, data processing method, device, medium and equipment | |
| Chen et al. | Research and Construction of University Data Governance Platform Based on Smart Campus Environment | |
| CN115237989A (en) | Mine data acquisition system | |
| CN112633621B (en) | Power grid enterprise management decision-making system and method based on PAAS platform | |
| Rahayu et al. | Monitoring system building plan for risk of civil office | |
| CN108121925B (en) | Generation method and generation system of electron microscope data label | |
| CN111369116A (en) | Project early-stage and project early-stage management system based on visualization technology | |
| CN111435466A (en) | Integrated machine room operation and maintenance management system | |
| CN116910062A (en) | Metadata management system based on real-time calculation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |