CN117807971A - Report generation method, device, terminal equipment and storage medium - Google Patents
Report generation method, device, terminal equipment and storage medium Download PDFInfo
- Publication number
- CN117807971A CN117807971A CN202311861783.8A CN202311861783A CN117807971A CN 117807971 A CN117807971 A CN 117807971A CN 202311861783 A CN202311861783 A CN 202311861783A CN 117807971 A CN117807971 A CN 117807971A
- Authority
- CN
- China
- Prior art keywords
- report
- data
- task
- management platform
- generating
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 56
- 238000007726 management method Methods 0.000 claims abstract description 108
- 238000013523 data management Methods 0.000 claims abstract description 77
- 238000012163 sequencing technique Methods 0.000 claims description 21
- 230000002159 abnormal effect Effects 0.000 claims description 18
- 238000010276 construction Methods 0.000 claims description 17
- 238000000586 desensitisation Methods 0.000 claims description 11
- 238000004140 cleaning Methods 0.000 claims description 7
- 238000004458 analytical method Methods 0.000 claims description 6
- 238000012544 monitoring process Methods 0.000 abstract description 11
- 238000012545 processing Methods 0.000 description 16
- 230000003442 weekly effect Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 8
- 238000012552 review Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 7
- 238000011835 investigation Methods 0.000 description 6
- 238000011161 development Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 238000004422 calculation algorithm Methods 0.000 description 4
- 238000013507 mapping Methods 0.000 description 4
- 230000007246 mechanism Effects 0.000 description 4
- 238000012800 visualization Methods 0.000 description 4
- 230000005856 abnormality Effects 0.000 description 3
- 239000003638 chemical reducing agent Substances 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 238000007405 data analysis Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000013144 data compression Methods 0.000 description 2
- 238000013075 data extraction Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 238000000638 solvent extraction Methods 0.000 description 2
- 239000002699 waste material Substances 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 239000006185 dispersion Substances 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 230000001629 suppression Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/166—Editing, e.g. inserting or deleting
- G06F40/177—Editing, e.g. inserting or deleting of tables; using ruled lines
- G06F40/18—Editing, e.g. inserting or deleting of tables; using ruled lines of spreadsheets
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/903—Querying
- G06F16/90335—Query processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/166—Editing, e.g. inserting or deleting
- G06F40/174—Form filling; Merging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/40—Transformation of program code
- G06F8/41—Compilation
- G06F8/42—Syntactic analysis
- G06F8/427—Parsing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Databases & Information Systems (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Audiology, Speech & Language Pathology (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a report generation method, a report generation device, terminal equipment and a storage medium, wherein the report generation method comprises the following steps: receiving a report generation request; inputting the report generation request into a report generation system constructed in advance, and generating a data report by the report generation system according to the report generation request, wherein the report generation system is obtained based on a preset data management platform, a template management platform and a task management platform. According to the invention, the data report is obtained by carrying out report generation through the data management platform, the template management platform and the task management platform in the report generation system which are constructed in advance, so that the generation of the report is realized, the problem of low efficiency caused by no corresponding data management and task monitoring during report generation is solved, and the report generation efficiency is improved.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a report generating method, a report generating device, a terminal device, and a storage medium.
Background
With the expansion and development of the business, the data size and the data consumption demands are increased, and the data is required to be statistically queried and generated under various scenes such as customer complaint problem investigation, business temporary data statistics, supervision data report uploading, weekly report data statistics, value review data and the like, meanwhile, the business data relates to user privacy information, strict access right control is required to be carried out on the data use, and the traditional data report generation scheme is as follows: applying access authority of each data table to write SQL or SQL-like sentences, manually inquiring on a CLI (command-line interface) tool, and exporting a result to generate a corresponding report.
But for the traditional data report generation scheme there are: (1) The lack of data task life cycle management and scheduling mainly comprises the situation that a user needs to manually follow up task execution, influences the working efficiency and does not support task scheduling; (2) poor data task query performance; (3) lack of hierarchical management of data security; (4) The task flow link is complex and mainly comprises the problems of data dispersion and lack of data result visualization.
Therefore, a report generation method is needed to solve the problem of low efficiency caused by the fact that no corresponding data management and task monitoring exist during report generation.
The foregoing is provided merely for the purpose of facilitating understanding of the technical solutions of the present invention and is not intended to represent an admission that the foregoing is prior art.
Disclosure of Invention
The invention mainly aims to provide a report generation method, a report generation device, terminal equipment and a storage medium, and aims to solve the technical problem that the efficiency is low because no corresponding data management and task monitoring exist during report generation.
In order to achieve the above object, the present invention provides a report generation method, including the steps of:
receiving a report generation request;
inputting the report generation request into a report generation system constructed in advance, and generating a data report by the report generation system according to the report generation request, wherein the report generation system is obtained based on a preset data management platform, a template management platform and a task management platform.
Optionally, before the step of inputting the report generating request into the report generating system constructed in advance, the method further includes:
extracting data according to a preset database, acquiring initial metadata, and constructing a platform through the initial metadata to acquire a data management platform;
template configuration is carried out according to the data management platform, a report task template is obtained, template platform construction is carried out through the report task template, and a template management platform is obtained;
and carrying out system construction through the data management platform, the template management platform and a preset task management platform to obtain a report generating system.
Optionally, the step of constructing a platform by using the initial metadata, and the step of obtaining a data management platform includes:
cleaning and type defining the initial metadata to obtain the data type of the initial metadata;
classifying the security of the initial metadata according to preset data characteristics, and acquiring the data security level of the initial metadata;
and constructing a platform through the data type and the data security level to obtain a data management platform.
Optionally, the step of generating, by the report generating system, a data report according to the report generating request includes:
Analyzing the report generation request, acquiring a data query strategy, and carrying out data query through the data management platform according to the data query strategy to acquire report data;
performing task generation through the template management platform according to the report data to obtain a report generation task;
and sending the report generating task to the task management platform for task execution, and acquiring a data report.
Optionally, the step of sending the report generating task to the task management platform for task execution and obtaining a data report includes:
sending the report generating task to an execution queue for queue sequencing through the task management platform, and obtaining a sequencing result;
if the sorting result is that the sorting is successful, the report generating task is sent to a computing cluster to execute the task, and an execution result is obtained;
if the execution result is that the execution is successful, acquiring an initial data report;
and carrying out hierarchical desensitization on the initial data report according to the data security level of the data management platform, obtaining the data report, and generating notification information through a message center and sending the notification information to a corresponding user.
Optionally, the step of sending the report generating task to an execution queue for queue ordering through the task management platform and obtaining an ordering result further includes:
if the sequencing result is that the sequencing fails, acquiring the queue condition of the computing cluster;
and generating abnormal information according to the queue condition, and sending the abnormal information to a corresponding user through a message center.
Optionally, after the step of generating the data report according to the report generation request by the report generation system, the method further includes:
performing scene analysis on the report generation request to obtain a report scene;
according to the report scene, carrying out missing query on the data report through the data management platform to obtain a query result;
and if the query result is data missing, performing task compensation on the data report to acquire a final data report.
The embodiment of the invention also provides a report generating device, which comprises:
the receiving module is used for receiving a report generation request;
the generating module is used for inputting the report generating request into a report generating system which is built in advance, and generating a data report by the report generating system according to the report generating request, wherein the report generating system is obtained based on a preset data management platform, a template management platform and a task management platform.
The embodiment of the invention also provides a terminal device which comprises a memory, a processor and a report generating program which is stored in the memory and can run on the processor, wherein the report generating program realizes the steps of the report generating method when being executed by the processor.
The embodiment of the invention also provides a computer readable storage medium, wherein the computer readable storage medium stores a report generation program, and the report generation program realizes the steps of the report generation method when being executed by a processor.
The embodiment of the invention provides a report generation method, a report generation device, terminal equipment and a storage medium, wherein a report generation request is received; inputting the report generation request into a report generation system constructed in advance, and generating a data report by the report generation system according to the report generation request, wherein the report generation system is obtained based on a preset data management platform, a template management platform and a task management platform. Therefore, the data report is obtained by carrying out report generation through the data management platform, the template management platform and the task management platform in the report generation system which are built in advance, the generation of the report is realized, the problem of low efficiency caused by no corresponding data management and task monitoring during the report generation is solved, and the report generation efficiency is improved.
Drawings
FIG. 1 is a schematic diagram of functional modules of a terminal device to which a report generating device of the present invention belongs;
FIG. 2 is a flowchart illustrating an exemplary embodiment of a report generation method according to the present invention;
FIG. 3 is a schematic diagram of an overall framework of the report generating method of the present invention;
FIG. 4 is a flowchart illustrating another exemplary embodiment of a report generating method according to the present invention;
FIG. 5 is a schematic diagram of report generation method of the present invention relating to report generation task management;
fig. 6 is a schematic diagram of a report generation method according to the present invention, in which abnormal information is transmitted to a user.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
The main solutions of the embodiments of the present invention are: extracting data according to a preset database, acquiring initial metadata, and constructing a platform through the initial metadata to acquire a data management platform; template configuration is carried out according to the data management platform, a report task template is obtained, template platform construction is carried out through the report task template, and a template management platform is obtained; and carrying out system construction through the data management platform, the template management platform and a preset task management platform to obtain a report generating system. Cleaning and type defining the initial metadata to obtain the data type of the initial metadata; classifying the security of the initial metadata according to preset data characteristics, and acquiring the data security level of the initial metadata; and constructing a platform through the data type and the data security level to obtain a data management platform. Analyzing the report generation request, acquiring a data query strategy, and carrying out data query through the data management platform according to the data query strategy to acquire report data; performing task generation through the template management platform according to the report data to obtain a report generation task; and sending the report generating task to the task management platform for task execution, and acquiring a data report. Sending the report generating task to an execution queue for queue sequencing through the task management platform, and obtaining a sequencing result; if the sorting result is that the sorting is successful, the report generating task is sent to a computing cluster to execute the task, and an execution result is obtained; if the execution result is that the execution is successful, acquiring an initial data report; and carrying out hierarchical desensitization on the initial data report according to the data security level of the data management platform, obtaining the data report, and generating notification information through a message center and sending the notification information to a corresponding user. If the sequencing result is that the sequencing fails, acquiring the queue condition of the computing cluster; and generating abnormal information according to the queue condition, and sending the abnormal information to a corresponding user through a message center. Performing scene analysis on the report generation request to obtain a report scene; according to the report scene, carrying out missing query on the data report through the data management platform to obtain a query result; and if the query result is data missing, performing task compensation on the data report to acquire a final data report. Therefore, the problem of low efficiency caused by no corresponding data management and task monitoring during report generation is solved, report generation is realized, and report generation efficiency is improved. Based on the scheme of the invention, the data size and the data consumption requirements are increased in reality, and various scenes such as customer complaint problem investigation, business temporary data statistics, supervision data report uploading, weekly report data statistics, value review data and the like need to be subjected to statistical query and report generation, meanwhile, business data relate to user privacy information, strict access right control is needed for data use, the problem that the life cycle management and scheduling of data tasks and the query performance of the data tasks are poor can not be solved by using the traditional generation scheme, so that the problem of low efficiency is solved.
Technical terms related to the embodiment of the invention:
HDFS: HDFS (Hadoop Distributed File System) is a distributed file system in the Apache Hadoop project, which aims to handle the storage and processing of large-scale data sets, is a reliable, high-throughput distributed file system, is suitable for storing and processing large-scale data sets, and is designed to provide high reliability, high performance and high fault tolerance, so that a user can store, access and analyze large-scale data.
Hive: hive is a data warehouse tool based on Hadoop, and is used for extracting, converting and loading data, which is a mechanism capable of storing, inquiring and analyzing large-scale data stored in Hadoop, the Hive data warehouse tool can map structured data files into a database table and provide SQL inquiring function, can convert SQL sentences into MapReduce tasks to be executed, and has the advantages of low learning cost, and can realize rapid MapReduce statistics through SQL-like sentences, so that MapReduce is simpler without developing special MapReduce application programs. hive is well suited for statistical analysis of data warehouses.
MR: MR (MapReduce) queries are one query style based on the MapReduce framework, which generally includes two phases: mapping (Map) and reduction (Reduce), MR queries can be implemented through a MapReduce programming model, and developers need to write custom mappers and Reducer, implement (Map) and (Reduce) functions to define specific query logic. The input data is usually stored in an HDFS, the Mapper reads the data on the HDFS to perform mapping processing, and then the Reducer performs reduction operation on the mapping result to finally obtain the query result.
SQL: SQL (Structured Query Language) is a standardized language for manipulating relational databases that may be used to query, insert, update, and delete data, and to create and manage database objects such as tables, views, indexes, and the like.
ClickHouse: the ClickHouse is an open-source column type database management system, is specially used for efficiently processing large-scale data analysis, is suitable for the scene of large-scale data analysis and query, has excellent performance, expandability and usability, and is one of the preferred tools for data analysis of many enterprises and organizations.
The embodiment of the invention considers that the problem of low accuracy exists in the mode because the problems of poor data task life cycle management and scheduling and data task query performance cannot be solved when the report is generated in the related technology.
Therefore, the embodiment of the invention designs a report generation method from the problems of increasing data scale and consumption demands in reality, such as customer complaint problem investigation, business temporary data statistics, supervision data report uploading, weekly report data statistics, value review data and the like, needs to carry out statistics query on data and generate a report, meanwhile, business data relates to user privacy information, strict access right control is needed for data use, the problems of poor data task life cycle management and scheduling and data task query performance cannot be solved by using a traditional generation scheme, and therefore, the problem of low efficiency is solved.
Specifically, referring to fig. 1, fig. 1 is a schematic diagram of functional blocks of a terminal device to which the report generating apparatus of the present invention belongs. The report generating device can be independent of the device of the terminal equipment, and can be used for generating the report, and can be carried on the terminal equipment in a form of hardware or software. The terminal equipment can be intelligent mobile equipment with a data processing function such as a mobile phone and a tablet personal computer, and can also be fixed terminal equipment or a server with a data processing function.
In this embodiment, the terminal device to which the report generating apparatus belongs at least includes an output module 110, a processor 120, a memory 130, and a communication module 140.
The memory 130 stores an operating system and a report generation program, and the report generation device may receive a report generation request; inputting the report generation request into a report generation system constructed in advance, and generating a data report by the report generation system according to the report generation request, wherein the report generation system is obtained based on a preset data management platform, a template management platform and a task management platform. Generating a report by the report generating program, and storing the generated result and other information in the memory 130; the output module 110 may be a display screen or the like. The communication module 140 may include a WIFI module, a mobile communication module, a bluetooth module, and the like, and communicates with an external device or a server through the communication module 140.
Wherein the report generator in memory 130 when executed by the processor performs the steps of:
receiving a report generation request;
inputting the report generation request into a report generation system constructed in advance, and generating a data report by the report generation system according to the report generation request, wherein the report generation system is obtained based on a preset data management platform, a template management platform and a task management platform.
Further, the report generator in the memory 130, when executed by the processor, also performs the steps of:
extracting data according to a preset database, acquiring initial metadata, and constructing a platform through the initial metadata to acquire a data management platform;
template configuration is carried out according to the data management platform, a report task template is obtained, template platform construction is carried out through the report task template, and a template management platform is obtained;
and carrying out system construction through the data management platform, the template management platform and a preset task management platform to obtain a report generating system.
Further, the report generator in the memory 130, when executed by the processor, also performs the steps of:
cleaning and type defining the initial metadata to obtain the data type of the initial metadata;
Classifying the security of the initial metadata according to preset data characteristics, and acquiring the data security level of the initial metadata;
and constructing a platform through the data type and the data security level to obtain a data management platform.
Further, the report generator in the memory 130, when executed by the processor, also performs the steps of:
analyzing the report generation request, acquiring a data query strategy, and carrying out data query through the data management platform according to the data query strategy to acquire report data;
performing task generation through the template management platform according to the report data to obtain a report generation task;
and sending the report generating task to the task management platform for task execution, and acquiring a data report.
Further, the report generator in the memory 130, when executed by the processor, also performs the steps of:
sending the report generating task to an execution queue for queue sequencing through the task management platform, and obtaining a sequencing result;
if the sorting result is that the sorting is successful, the report generating task is sent to a computing cluster to execute the task, and an execution result is obtained;
If the execution result is that the execution is successful, acquiring an initial data report;
and carrying out hierarchical desensitization on the initial data report according to the data security level of the data management platform, obtaining the data report, and generating notification information through a message center and sending the notification information to a corresponding user.
Further, the report generator in the memory 130, when executed by the processor, also performs the steps of:
if the sequencing result is that the sequencing fails, acquiring the queue condition of the computing cluster;
and generating abnormal information according to the queue condition, and sending the abnormal information to a corresponding user through a message center.
Further, the report generator in the memory 130, when executed by the processor, also performs the steps of:
performing scene analysis on the report generation request to obtain a report scene;
according to the report scene, carrying out missing query on the data report through the data management platform to obtain a query result;
and if the query result is data missing, performing task compensation on the data report to acquire a final data report.
According to the scheme, the report generation request is received; inputting the report generation request into a report generation system constructed in advance, and generating a data report by the report generation system according to the report generation request, wherein the report generation system is obtained based on a preset data management platform, a template management platform and a task management platform. The report is generated through the data management platform, the template management platform and the task management platform in the report generation system which are built in advance, so that a data report is obtained, and the problem that the efficiency is low because no corresponding data management and task monitoring exist during report generation can be solved. Based on the scheme of the invention, the data size and the data consumption requirements are increased in reality, and various scenes such as customer complaint problem investigation, business temporary data statistics, supervision data report uploading, weekly report data statistics, value review data and the like need to be subjected to statistical query and report generation, meanwhile, business data relate to user privacy information, strict access right control is needed for data use, the problem that the life cycle management and scheduling of data tasks and the query performance of the data tasks are poor can not be solved by using the traditional generation scheme, so that the problem of low efficiency is solved.
The method embodiments of the present invention are presented based on the above-described terminal device architecture but not limited to the above-described framework.
Referring to fig. 2, fig. 2 is a flowchart illustrating an exemplary embodiment of a report generating method according to the present invention. The report generation method comprises the following steps:
step S01, receiving a report generation request;
the execution subject of the method of the embodiment may be a report generating device, or may be a report generating terminal device or a server, and the report generating device is exemplified by the report generating device, and the report generating device may be integrated on a terminal device with a data processing function.
Before describing the report generation method proposed by the embodiment, it should be understood that with the expansion and development of the service, the data scale and the data consumption demands are increased, and statistical query needs to be performed on data and report generation under various scenes such as customer complaint problem investigation, temporary service data statistics, supervision data report uploading, weekly report data statistics, value review data and the like, meanwhile, the service data relates to user privacy information, strict access right control is required for data use, the known traditional data report generation scheme is as follows, the access right of each data table is applied to write SQL or SQL-like sentences, manual query is performed on a self-contained CLI (command-line interface) tool, and a corresponding report is generated by the derived result;
However, for the conventional data report generation scheme, there are problems of lack of data task life cycle management and scheduling, poor data task query performance, lack of data security hierarchical management and complex task flow links, so the method provided by this embodiment is to use a report generation system constructed in advance to generate a data report, and specifically may first receive a report generation request, where the received report generation request may be sent by a user when the user needs to view data, or may also be generated for periodic reports of government systems, enterprises, schools, and the like.
And step S05, inputting the report generation request into a report generation system constructed in advance, and generating a data report by the report generation system according to the report generation request, wherein the report generation system is obtained based on a preset data management platform, a template management platform and a task management platform.
After receiving the report generation request, inputting the report generation request into a pre-constructed report generation system, wherein the report generation system comprises, but is not limited to, a data management platform, a template management platform, a task management platform, a visualization module and the like;
and generating a report by the report generating system according to the report generating request to obtain a data report.
Based on the above-mentioned scheme, it is known that the report generating system mainly depends on generating the data report in this embodiment, so the step S05 further includes, before the step of inputting the report generating request into the report generating system constructed in advance:
step S02, extracting data according to a preset database, obtaining initial metadata, and constructing a platform through the initial metadata to obtain a data management platform;
step S03, carrying out template configuration according to the data management platform to obtain a report task template, and carrying out template platform construction through the report task template to obtain a template management platform;
and S04, performing system construction through the data management platform, the template management platform and a preset task management platform to obtain a report generation system.
The report generating system described in this embodiment is based on a data management platform and a template management platform, so the data management platform should be obtained first, specifically: analyzing the data storage condition, since most of data is mainly distributed in Hive clusters, data warehouses and ClickHouse, data definition and management are needed, so that data extraction can be performed on databases (namely Hive clusters, data warehouses and ClickHouse), metadata are extracted, wherein the metadata comprise but are not limited to business metadata, technical metadata, management metadata and the like, after specific metadata are obtained, in order to facilitate management on subsequently received data, namely platform construction is performed according to the obtained data, a data management platform is obtained, and different query strategies and storage modes which are adopted by the Hive clusters, the data warehouses and the ClickHouse are different may exist, so that the embodiment can be used for collecting and managing multi-source data, and the problem that the data extraction is not smooth can be avoided;
After the data management platform is obtained, the data report generated by the embodiment is generated in a task scheduling mode, so that the corresponding template management platform is required to manage templates of report generation tasks, various types of data such as private templates, public templates and the like are configured for the various types of data due to the fact that the data management platform is provided with corresponding data types, namely business metadata, technical metadata, management metadata and the like, the report task templates are obtained, platform construction is carried out through the report task templates, and the template management platform is obtained, wherein the public templates can be used by all system users, the private templates are used by a single user, after a certain required access template is created by the user, the query task can be directly created by modifying query parameters configured by the templates, the corresponding data can be obtained by the query task, and then the data report is generated;
through the scheme, after the data management platform and the template management platform are obtained, the preset task management platform, the data management platform and the template management platform can be used for carrying out system construction, and the report generation system is obtained.
More specifically, in step S02, the platform construction is performed through the initial metadata, and the step of obtaining the data management platform includes:
Step S021, cleaning and type defining are carried out on the initial metadata, and the data type of the initial metadata is obtained;
step S022, carrying out security classification on the initial metadata according to preset data characteristics, and obtaining the data security level of the initial metadata;
and step S023, carrying out platform construction through the data type and the data security level to obtain a data management platform.
After extracting the database, performing data cleaning and type definition on the initial metadata (namely, the original data) to obtain (1) service metadata: defining information such as business meaning and business rule of data, and enabling a user to quickly data business background, (2) defining technical metadata, and defining data related to development, management and maintenance of a plurality of bins, wherein the technical metadata comprises the following components: data source information; a description of data conversion; rules for data cleaning and data updating, mapping of source data to destination data, etc., (3) management metadata: defining data management related information, including table service responsible person, technical responsible person, etc., it should be clear that, because the objects of action are different, the defined data types may also be different, so the definition of the data should be set according to the actual service requirement, and the embodiment is not limited specifically;
The report generating method in this embodiment is applied to government affairs and enterprise systems, such as banks, so there may be corresponding sensitive data, so a sensitive level marking is needed, in this embodiment, an actual banking transaction system is taken as an example, but it should be understood that the scheme proposed in this embodiment should be more than a banking system, and may also include a government affair system, a corresponding enterprise and institution system, etc., and the adopted data classification policy and measures should be classified according to specific data levels, where the embodiment classifies data according to field dimensions based on "financial data security-data security classification guide", and classifies the data security level from high to low according to the affected object after the data security of the financial institution is damaged and the degree of influence caused, as shown in the following table:
thus, the corresponding data security level is obtained;
and finally, constructing a platform through the data type and the data security level to obtain a data management platform.
Further, as shown in fig. 3, fig. 3 is a schematic diagram of an overall framework of the report generating method of the present invention.
The operation flow of the report generating system in this embodiment is as follows:
Firstly, storing data by a data storage module, wherein the data comprises Mysql and HBase, and the received metadata is derived from Hive, clickHouse, unified logs and the like;
then, core services provided by the report generating system include task processing, task monitoring, message notification, authority control, data processing, sensitive data desensitization, api callback, task compensation and the like;
then, the gateway service corresponding to the report generating system comprises login authentication, route forwarding, flow control, log record and the like;
then, web services corresponding to the report generating system comprise task management, template management, data editing, authority management, result export, task monitoring, field sample, label management and the like;
finally, the operation object provided by the report generating system is a developer and a business person in the embodiment, and based on the service, login authentication, message notification, timing task management, data desensitization, OLAP (online analytical processing), data acquisition and the like are completed.
According to the scheme, the report generation request is received; inputting the report generation request into a report generation system constructed in advance, and generating a data report by the report generation system according to the report generation request, wherein the report generation system is obtained based on a preset data management platform, a template management platform and a task management platform. Therefore, the data report is obtained by carrying out report generation through the data management platform, the template management platform and the task management platform in the report generation system which are built in advance, the generation of the report is realized, the problem of low efficiency caused by no corresponding data management and task monitoring during the report generation is solved, and the report generation efficiency is improved.
Referring to fig. 4, fig. 4 is a flowchart illustrating another exemplary embodiment of the report generating method according to the present invention.
Based on the embodiment shown in fig. 2, the step S05, the step of generating, by the report generating system, a data report according to the report generating request includes:
step S051, analyzing the report generation request, obtaining a data query strategy, and carrying out data query through the data management platform according to the data query strategy to obtain report data;
s052, performing task generation through the template management platform according to the report data to obtain a report generation task;
and step S053, sending the report generation task to the task management platform for task execution, and obtaining a data report.
The embodiment shown in fig. 2 is used to learn that the report generating system has completed construction, so that the received report generating request is parsed at this time, so that the information such as the data type and the like which the user wants to acquire can be known, and the data query policy is generated accordingly. Data query is performed on the data management platform through a preconfigured data processing component Mapper and a query statement, and in this embodiment, the components and the statement of the data query are also optimized, which specifically may be: (1) Because log data is stored in an HDFS in different sizes, performance influence and resource waste caused by too many small files in the MR query process are required to be considered, reasonable parameter configuration is required for task submission, and the system completes parameter tuning by configuring the combined small files, configuring the upper limit of a single Mapper memory and configuring the number of Mapper, reducer reasonably, so that efficient query and cluster resource of the task are effectively utilized, (2) SQL tuning is ensured, and because the data volume processed by Hive is usually in a TB level, SQL tuning is very important, and the system mainly comprises: 1) Improving performance query: by optimizing query plans, index designs, data partitioning and other modes, unnecessary data scanning and calculation are reduced, so that the query speed is increased, and 2) the resource consumption is reduced: by rationally designing the data partitioning and using the index, unnecessary data movement and reading can be reduced, thereby reducing resource consumption, 3) optimizing data storage and compression: for large-scale data sets, data compression techniques can be used to reduce storage space and I/O overhead, and compressed data can be loaded and queried more quickly;
And after report data is obtained through the data processing component and the query statement, generating tasks through the template management platform to obtain report generating tasks, and inputting the report generating tasks into the task management platform to execute the tasks to finally obtain the data report.
Specifically, in the step S053, the step of sending the report generating task to the task management platform for task execution, and the step of obtaining the data report includes:
step S0531, sending the report generating task to an execution queue for queue sequencing through the task management platform, and obtaining a sequencing result;
step S0534, if the sorting result is that the sorting is successful, the report generating task is sent to a computing cluster to execute the task, and an execution result is obtained;
step S0535, if the execution result is that the execution is successful, acquiring an initial data report;
and step S0536, carrying out hierarchical desensitization on the initial data report according to the data security level of the data management platform, obtaining the data report, and generating notification information through a message center and sending the notification information to a corresponding user.
In the task management platform, more report generating tasks may exist, so that the previously generated report generating tasks need to be sent to an execution queue for sorting, the resources of a computing cluster are calculated in the sorting process, if the resources in the computing cluster can execute the current report generating tasks, the report generating tasks are sent to the computing cluster for calculation, and the system is prevented from being damaged, wherein the tasks can be set with priority before sorting, so that when the tasks with more urgency are processed, priority processing (such as resource clearing for the tasks with no importance currently is performed, so as to obtain more resources for processing the tasks) can be performed, and the problem of untimely service feedback is solved;
When the sorting is completed and the task is executed, an initial data report is obtained, but there is a problem of data sensitivity at this time, and in the above embodiment, a multi-level data level has been set, so that the obtained initial data report should be subjected to a desensitization operation, the data desensitization is processed based on data classification, and the classification desensitization rule is as follows:
the higher the data level is, the more stringent the data desensitization algorithm is, for example, the system uses a generalization class algorithm to desensitize addresses and years, uses an encryption class algorithm to desensitize highly sensitive data such as card numbers, certificate numbers and telephone numbers, and uses a suppression class algorithm to desensitize sensitive information such as mailboxes and short messages;
it should be clear that by constructing the completed report generating system, the task life cycle from the submitted to the terminated state is divided into fine granularity, the task list adopts a timing refreshing strategy, the user can intuitively see the stage of the task, determine whether the task is abnormal or not and complete execution, and once the task is successfully or unsuccessfully executed, the system can timely send information to the data user through message pushing to inform the user of checking report results.
More specifically, in the step S0531, the step of sending, by the task management platform, the report generating task to an execution queue for queue sorting, and the step of obtaining the sorting result further includes:
Step S0532, if the sorting result is that the sorting fails, obtaining the queue condition of the computing cluster;
and step S0533, generating abnormal information according to the queue condition, and sending the abnormal information to a corresponding user through a message center.
The task is executed and sequenced, and the task is abnormal and overtime possibly exists, so when the sequencing fails, the queue, the computing cluster and the like are monitored abnormally, and the queue condition is obtained;
generating abnormal information through the queue condition, and sending the abnormal information to a corresponding user through a message center.
Fig. 5 is a schematic diagram of the report generation method according to the present invention, in which abnormal information is transmitted to a user, as shown in fig. 5.
Firstly, submitting tasks to a queue, sequencing the queue if the task is submitted successfully, and performing cluster allocation resource execution after sequencing is completed;
then, if the resource allocation is overtime or the execution of the computing cluster task is overtime, the task abnormality information is sent;
and finally, sending the task abnormality information to the corresponding user through the message center.
FIG. 6 is a schematic diagram of report generation task management according to the report generation method of the present invention, as shown in FIG. 6.
Firstly, submitting report generating tasks to a queue for sorting by a user;
Then, when the task is executed, the task is submitted to a computing cluster for task execution, if more tasks are in the queue, the output queue overflows, and the task cannot enter the queue at the moment;
after the task is executed, the completion of the task execution, the abnormality of the task execution, the overtime and the like can be output, when the task execution is abnormal and overtime, the task is judged to be failed, and the information is fed back to the corresponding user through the message center;
then, after the task execution is completed, the data is sent to be desensitized, and the finally obtained data report and a report generation success notification are sent to the corresponding user;
finally, the embodiment further increases a task delay mechanism, more reliably releases cluster resources occupied by invalid tasks, submits tasks to the cluster through JDBC, and the tasks are received and operated by the cluster for a certain time interval, if the tasks are stopped by using the application Id according to the traditional method, the tasks during the interval cannot be stopped, the tasks are received and operated by the cluster again to cause cluster resource waste, and aiming at the situation, the system increases the task delay stopping mechanism, when the tasks receive a stopping signal but the application Id is not generated, the tasks are placed in a false stopping state, at the moment, the task execution process log is queried, and the task is stopped after the application Id is generated by the tasks, in addition, if the tasks are in a queue and are not submitted to the cluster processing, at the moment, the task is not submitted when the task stopping signal is received again.
Further, based on the embodiment shown in fig. 2, the step S05 further includes, after the step of generating, by the report generating system, a data report according to the report generating request:
s06, performing scene analysis on the report generation request to obtain a report scene;
s07, carrying out missing query on the data report through the data management platform according to the report scene to obtain a query result;
and step S08, if the query result is data missing, performing task compensation on the data report to acquire a final data report.
Because the timing task may fail to execute an instance on a certain day due to various reasons and cause data to be lost, a self-service report needs to be generated regularly for the scenes of development of weekly report, value review, MGM weekly report and the like, therefore, the embodiment increases a task compensation mechanism, and carries out task running compensation again for the instance of abnormal execution of the timing task, thereby reducing complex compensation work, and particularly being capable of:
performing scene analysis on report generation to obtain the scenes of report generation, for example, the scenes of development weekly report, value review, MGM weekly report and the like, and performing missing query on the data management platform;
If the query result is data missing, performing task compensation on the data report table to obtain a final data report, so that the data missing is avoided;
the embodiment also provides visualization processing for the generated data report, the system provides different types of visualization templates such as a dashboard, a line graph and a radar graph, and a user can select different templates to visualize data according to specific requirements.
According to the scheme, the report generation request is analyzed to obtain a data query strategy, and according to the data query strategy, data query is performed through the data management platform to obtain report data; performing task generation through the template management platform according to the report data to obtain a report generation task; and sending the report generating task to the task management platform for task execution, and acquiring a data report. Therefore, the generation of the data report is realized, the problem of low efficiency caused by the fact that no corresponding data management and task monitoring exist during the generation of the report is solved, and the efficiency of the generation of the report is improved.
In addition, the embodiment of the invention also provides a report generating device, which comprises:
The receiving module is used for receiving a report generation request;
the generating module is used for inputting the report generating request into a report generating system which is built in advance, and generating a data report by the report generating system according to the report generating request, wherein the report generating system is obtained based on a preset data management platform, a template management platform and a task management platform.
In addition, the embodiment of the invention also provides a terminal device, which comprises a memory, a processor and a report generating program stored on the memory and capable of running on the processor, wherein the report generating program realizes the steps of the report generating method when being executed by the processor.
Because the report table generating program is executed by the processor, all the technical schemes of all the embodiments are adopted, and therefore, the report table generating program at least has all the beneficial effects brought by all the technical schemes of all the embodiments, and is not described in detail herein.
In addition, the embodiment of the invention also provides a computer readable storage medium, wherein the computer readable storage medium stores a report generation program, and the report generation program realizes the steps of the report generation method when being executed by a processor.
Because the report table generating program is executed by the processor, all the technical schemes of all the embodiments are adopted, and therefore, the report table generating program at least has all the beneficial effects brought by all the technical schemes of all the embodiments, and is not described in detail herein.
Compared with the prior art, the report generation method, the device, the terminal equipment and the storage medium provided by the embodiment of the invention receive the report generation request; inputting the report generation request into a report generation system constructed in advance, and generating a data report by the report generation system according to the report generation request, wherein the report generation system is obtained based on a preset data management platform, a template management platform and a task management platform. Therefore, the problem of low efficiency caused by no corresponding data management and task monitoring during report generation is solved, report generation is realized, and report generation efficiency is improved. Based on the scheme of the invention, the data size and the data consumption requirements are increased in reality, and various scenes such as customer complaint problem investigation, business temporary data statistics, supervision data report uploading, weekly report data statistics, value review data and the like need to be subjected to statistical query and report generation, meanwhile, business data relate to user privacy information, strict access right control is needed for data use, the problem that the life cycle management and scheduling of data tasks and the query performance of the data tasks are poor can not be solved by using the traditional generation scheme, so that the problem of low efficiency is solved.
Compared with the prior art, the embodiment of the invention has the following advantages:
1. task complete life cycle and task scheduling management;
2. task execution performance optimization;
3. data security hierarchical management;
4. and (5) normalizing the report generation flow.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) as above, comprising instructions for causing a terminal device (which may be a mobile phone, a computer, a server, a controlled terminal, or a network device, etc.) to perform the method of each embodiment of the present invention.
The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.
Claims (10)
1. The report generation method is characterized by comprising the following steps of:
receiving a report generation request;
inputting the report generation request into a report generation system constructed in advance, and generating a data report by the report generation system according to the report generation request, wherein the report generation system is obtained based on a preset data management platform, a template management platform and a task management platform.
2. The report generating method as recited in claim 1, wherein said step of inputting said report generating request into a pre-constructed report generating system further comprises:
extracting data according to a preset database, acquiring initial metadata, and constructing a platform through the initial metadata to acquire a data management platform;
template configuration is carried out according to the data management platform, a report task template is obtained, template platform construction is carried out through the report task template, and a template management platform is obtained;
And carrying out system construction through the data management platform, the template management platform and a preset task management platform to obtain a report generating system.
3. The report generating method as set forth in claim 2, wherein the step of obtaining a data management platform by performing platform construction of the initial metadata includes:
cleaning and type defining the initial metadata to obtain the data type of the initial metadata;
classifying the security of the initial metadata according to preset data characteristics, and acquiring the data security level of the initial metadata;
and constructing a platform through the data type and the data security level to obtain a data management platform.
4. The report generating method as recited in claim 2, wherein the step of generating a data report by the report generating system according to the report generating request comprises:
analyzing the report generation request, acquiring a data query strategy, and carrying out data query through the data management platform according to the data query strategy to acquire report data;
performing task generation through the template management platform according to the report data to obtain a report generation task;
And sending the report generating task to the task management platform for task execution, and acquiring a data report.
5. The report generating method as recited in claim 4, wherein the step of sending the report generating task to the task management platform for task execution, and obtaining the data report comprises:
sending the report generating task to an execution queue for queue sequencing through the task management platform, and obtaining a sequencing result;
if the sorting result is that the sorting is successful, the report generating task is sent to a computing cluster to execute the task, and an execution result is obtained;
if the execution result is that the execution is successful, acquiring an initial data report;
and carrying out hierarchical desensitization on the initial data report according to the data security level of the data management platform, obtaining the data report, and generating notification information through a message center and sending the notification information to a corresponding user.
6. The report generating method according to claim 5, wherein the step of sending the report generating task to an execution queue for queue sorting through the task management platform, and obtaining the sorting result further comprises:
If the sequencing result is that the sequencing fails, acquiring the queue condition of the computing cluster;
and generating abnormal information according to the queue condition, and sending the abnormal information to a corresponding user through a message center.
7. The report generating method as recited in claim 2, wherein said step of generating a data report by said report generating system in accordance with said report generating request further comprises:
performing scene analysis on the report generation request to obtain a report scene;
according to the report scene, carrying out missing query on the data report through the data management platform to obtain a query result;
and if the query result is data missing, performing task compensation on the data report to acquire a final data report.
8. A report generating apparatus, characterized in that the report generating apparatus includes:
the receiving module is used for receiving a report generation request;
the generating module is used for inputting the report generating request into a report generating system which is built in advance, and generating a data report by the report generating system according to the report generating request, wherein the report generating system is obtained based on a preset data management platform, a template management platform and a task management platform.
9. A terminal device comprising a memory, a processor and a report generating program stored on the memory and executable on the processor, the report generating program when executed by the processor implementing the steps of the report generating method according to any of claims 1-7.
10. A computer-readable storage medium, wherein a report generation program is stored on the computer-readable storage medium, which when executed by a processor, implements the steps of the report generation method of any of claims 1-7.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202311861783.8A CN117807971A (en) | 2023-12-29 | 2023-12-29 | Report generation method, device, terminal equipment and storage medium |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202311861783.8A CN117807971A (en) | 2023-12-29 | 2023-12-29 | Report generation method, device, terminal equipment and storage medium |
Publications (1)
Publication Number | Publication Date |
---|---|
CN117807971A true CN117807971A (en) | 2024-04-02 |
Family
ID=90432974
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202311861783.8A Pending CN117807971A (en) | 2023-12-29 | 2023-12-29 | Report generation method, device, terminal equipment and storage medium |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN117807971A (en) |
-
2023
- 2023-12-29 CN CN202311861783.8A patent/CN117807971A/en active Pending
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20220207163A1 (en) | Apparatuses, methods, and computer program products for programmatically parsing, classifying, and labeling data objects | |
US11403347B2 (en) | Automated master data classification and curation using machine learning | |
CN109582717A (en) | A kind of database unified platform and its read method towards electric power big data | |
US20080065588A1 (en) | Selectively Logging Query Data Based On Cost | |
CN103336791A (en) | Hadoop-based fast rough set attribute reduction method | |
CN112347071B (en) | Power distribution network cloud platform data fusion method and power distribution network cloud platform | |
US20220207429A1 (en) | Apparatuses, methods, and computer program products for programmatically parsing, classifying, and labeling data objects | |
CN113535677B (en) | Data analysis query management method, device, computer equipment and storage medium | |
CN112801607A (en) | Management service platform and construction method | |
CN112419129A (en) | Method and system for constructing platform system infrastructure in government affair service | |
CN113791586A (en) | Novel industrial APP and identification registration analysis integration method | |
US11914624B2 (en) | Systems and methods for managing connections in scalable clusters | |
CN114003600A (en) | Data processing method, system, electronic device and storage medium | |
CN113836237A (en) | Method and device for auditing data operation of database | |
CN117082148A (en) | Data access method and system of management platform | |
US8229946B1 (en) | Business rules application parallel processing system | |
CN117807971A (en) | Report generation method, device, terminal equipment and storage medium | |
US11907243B2 (en) | Core reconciliation system with cross-platform data aggregation and validation | |
CN114297223A (en) | Small and medium-sized enterprise informatization service platform based on big data | |
CN113779215A (en) | Data processing platform | |
CN115168297A (en) | Bypassing log auditing method and device | |
Yang | Human resource big data analysis and decision making of group enterprises based on cloud platform | |
CN113111046A (en) | Data management system based on main data drive | |
Wang et al. | An intelligent DevOps platform research and design based on machine learning | |
de Camargo Magano et al. | Abstracting Big Data Processing Tools for Smart Cities |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |