CN107240011B

CN107240011B - Index configuration processing method and system

Info

Publication number: CN107240011B
Application number: CN201710256834.2A
Authority: CN
Inventors: 李志顺; 赵晨玮; 李孔仁; 罗川匀; 窦文梅; 林继承; 温琼; 卢雯; 蔡科
Original assignee: China Construction Bank Corp
Current assignee: China Construction Bank Corp
Priority date: 2017-04-19
Filing date: 2017-04-19
Publication date: 2021-07-06
Anticipated expiration: 2037-04-19
Also published as: CN107240011A

Abstract

The invention discloses an index configuration processing method and system, wherein the method comprises the following steps: classifying the indexes according to the index source, and defining the measurement types corresponding to the indexes; according to the classified template to which the index belongs, carrying out configuration processing and engine processing on the index; and outputting the index processed by the engine. Therefore, a client-based targeted management scheme is realized, specialized services are promoted, and more complete data support is provided. Therefore, coupling dependence on services and technologies is reduced, communication and process propelling costs are reduced, timeliness from presentation to presentation of indexes is improved, index analysis management efficiency is improved, and efficient and accurate data support is provided for management decisions.

Description

Index configuration processing method and system

Technical Field

The invention relates to the field of data processing, in particular to an index configuration processing method and system.

Background

The bank data management is continuously developed, and as the data generated in the business management process has the characteristics of large quantity, high updating frequency, complex analysis and processing and the like, a certain data processing and analyzing tool is required to analyze and manage mass data so as to meet the requirements of management, wind control, decision making and the like. Therefore, the management information system processes and analyzes the index data through the index engine, and the timeliness and the accuracy of data information analysis can be improved.

In a management system based on index analysis and evaluation, service personnel often need to complete the definition or modification of indexes according to assessment calibers or industrial experience and check the data result display of index analysis. In the traditional method, the design and maintenance of indexes are realized by means of the whole flow of requirement proposing, analysis, testing and online to realize the final data analysis and result display. Because of the pre-flow process involved, there is some lag in the timeliness of data analysis, and maintenance work is also relatively complex. With the increasing requirement of improving the management efficiency, the requirements for fine management and maintainability of the index also increase. In the traditional method, communication cost, development cost and maintenance cost are difficult to meet the requirements of timeliness and accuracy. In addition, the realization of the indexes is usually concentrated on the technical level, the direct operation is difficult for business personnel, and the operations of index addition, deletion, modification and the like can be completed after the code realization is technically carried out and the online. Meanwhile, due to the fact that technology development is not flexible enough, the final display of the indexes depends on understanding of business and technology parties to each other and the capability of promoting the whole business process, and the analysis effect cannot be guaranteed.

Disclosure of Invention

In view of the foregoing defects in the prior art, embodiments of the present invention provide an index configuration processing method and system, which can effectively solve the problem that the requirements for fine management and maintainability of the current index cannot be met.

Specifically, the embodiment of the invention provides an index configuration processing method, wherein the method comprises the following steps:

classifying the indexes according to the index source, and defining the measurement types corresponding to the indexes;

according to the classified template to which the index belongs, carrying out configuration processing and engine processing on the index;

and outputting the index processed by the engine.

Correspondingly, the embodiment of the invention also provides an index configuration processing system, wherein the system comprises:

the index classification module is used for classifying the indexes according to the index source and defining the measurement types corresponding to the indexes;

the index configuration processing module is used for performing configuration processing and engine processing on the indexes according to the classified templates to which the indexes belong;

and the index result output module is used for outputting the index processed by the engine.

The embodiment of the invention has the following beneficial effects: the method has the advantages of reducing coupling dependence on services and technologies, reducing communication and flow propelling costs, improving timeliness from presentation to presentation of indexes, improving index analysis and management efficiency, and further providing more efficient and accurate data support for management decisions.

Drawings

FIG. 1 is a schematic flow chart of an index configuration processing method according to an embodiment of the present invention;

FIG. 2 is a schematic flow chart of step S1 in the embodiment shown in FIG. 1;

FIG. 3 is a schematic flow chart of step S2 in the embodiment shown in FIG. 1;

FIG. 4 is an architecture diagram of an index configuration processing system according to an embodiment of the present invention;

FIG. 5 is a block diagram of the metric classification module 100 in the system of FIG. 4;

FIG. 6 shows a block diagram of the index configuration processing module 200 in the system of FIG. 4.

Detailed Description

To facilitate an understanding of the various aspects, features and advantages of the present inventive subject matter, reference is made to the following detailed description taken in conjunction with the accompanying drawings. It should be understood that the various embodiments described below are illustrative only and are not intended to limit the scope of the invention.

First, names or terms that may be referred to according to the present invention are explained.

Dimension: the visual angle or direction of observing an object is a concept of judging, explaining, evaluating and determining different directions, different angles and different levels of an object.

The fact that: reflects the evaluation of things and describes the environment of various metrics.

ETL: extraction (Extract), transformation (Transform), and loading (Load) processes of data.

Example 1:

FIG. 1 is a schematic flow chart of an index configuration processing method according to an embodiment of the present invention; referring to fig. 1, the method includes:

step S1, classifying the indexes according to the index source, and defining the measurement types corresponding to the indexes;

step S2, according to the classified template to which the index belongs, the index is configured and processed by engine processing;

and step S3, outputting the index after the engine machining.

The index configuration processing method provided by the invention can reduce the coupling dependence on services and technologies, reduce the cost of communication and flow promotion, improve the timeliness from presentation to presentation of indexes, improve the index analysis management efficiency, and further provide more efficient and accurate data support for management decisions.

Example 2:

in an embodiment of the present invention, the step S1 further includes:

classifying the indexes according to the types of ETL processing indexes, derived indexes and manual standing book indexes; and defining the index by measuring types of time, mechanism and currency.

FIG. 2 is a schematic flow chart of step S1 in the embodiment shown in FIG. 1; referring to fig. 2, in the data source classification, the indexes can be classified into source data processing indexes, derived indexes and manual ledger indexes, which are described in the following table.

On the basis of the classification, the three types of indexes are used as three modules for design. For the part with larger fitness between the source system and the engine target index system, the data can be directly placed in the target system index system through simple ETL processing.

The dimension is the specific location of one index and is also the finest granularity of the index data which can be obtained. The indexes in the invention use three dimensions of time, mechanism and currency. In the index engine, a metric corresponding to each index can be defined, and the index metrics in the invention are classified as follows:

example 3:

FIG. 3 is a schematic flow chart of step S2 in the embodiment shown in FIG. 1; referring to fig. 3, wherein the step S2 further includes:

selecting a corresponding template according to the category to which the index belongs to carry out index configuration, and importing the index into the template; and analyzing and checking the configuration information of the imported template, and performing engine processing on the index. The method specifically comprises the following steps: (A) uploading an Excel configuration file; (B) analyzing and checking Excel configuration information; (C) and (6) processing the engine.

And (C) for the step (A), configuring the indexes by business personnel through three Excel templates of an ETL (extract transform load) processing index, a derived index and a manual standing book index, and completing the functions of offline editing and online importing of the indexes. Several main terms of the index template are described as follows:

1) ETL processing index introduction template

The ETL processing index import template is divided into an index definition page and an SQL/Perl configuration page, wherein the index definition page comprises basic information of indexes, such as index numbers, source systems, generation periods, processing methods and the like;

the SQL/Perl configuration page comprises SQL processing statement/Perl script names, index numbers and processing calibers thereof. SQL sentences are source system data processing sentences, and complex processing can be realized in Perl. For variables such as date and database source, common variables are used for replacement, and actual values are replaced when an engine is actually processed.

Information item	Description of the invention
		Index number	Unique identification index
Index SQL processing statement/Perl script name	Index access SQL statement or Perl script name

2) Derived index import template

The derived index import template contains derived definition information of the index, such as index number, derived formula and other information.

Information item	Description of the invention
		Index number	Unique identification index
Derived formula	Index of working diameter
		Generating a period	Frequency of index processing

The derived formula supports four arithmetic operations among indexes, and the indexes of the formula can comprise ETL processing indexes and derived indexes and support cross-system derivation. Besides the four arithmetic operations, common variables of fixed applications, such as date, days and other parameters, are supplemented and provided as required, and the expandability of formula definition is increased. The reference of the constants and the common variables in the formula is marked by brackets [ ]. For example, (ZB.100010.01+ ZB.100010.01+ [0.3])/[ # mon _ dys ], # mon _ dys indicates the days of the month.

3) Manual ledger index import template

The manual standing book index import template comprises a basic information page and an index value import page of indexes. Wherein the index basic information page is consistent with the ETL processing index. The index value import page includes information such as an index number and an index value of a desired import index. The part of the indexes are directly imported into an index result table and are not processed by an engine.

Information item	Description of the invention
		Index number	Unique identification index
Date	Date of service to which the index belongs
		Mechanism	Department of institution to which the index belongs
Coin kind	The currency type of the index
		Index value	Default metric value of an index
Other metric values	Other measure of the index, filling in on-demand

And the Excel is edited offline and imported online, so that service personnel can autonomously maintain the configuration of the indexes, and can directly perform flexible operations such as addition and modification on the indexes, thereby reducing the dependence on technical details.

For step (B), the verification of the template includes the following points: checking the uniqueness of the index number; the existence check of the index number used in the derived formula: ensuring that the index number exists in the existing stock index list; and (3) verifying the validity of a derivative formula: the operation logic of the formula IV is error-free, public variables exist, and the validity check rule can refer to the following scheme:

uniformly replacing the indexes and variables with 1, and performing judgment processing by utilizing an eval function in Java; the analysis work mainly comprises index source system, analysis of generation period and analysis of index level: the analysis of the index 'source system' is used as the blood relationship between the index and the system, and has an important role in the processing of the index among multiple systems. The processing logic is as follows:

a) in the ETL processing index, the 'related index range' is an empty index, and an index source system is a 'source system' in the template;

b) directly incorporating the index of the index system of the invention from a source system, wherein the source system is the source system;

c) except the two points, the source system collection of the 'related index range' attribute in the ETL processing index and each index in the derived index formula is taken;

the analysis of the generation period has the same logic as 1) the source system;

and (5) analyzing the index level. The sequence of the indexes depends on the index level, and the upper-layer indexes can be processed after the processing of the lower-layer indexes depends on the upper-layer indexes, so that the accuracy of the processing result of the upper-layer indexes is ensured. The hierarchy in the present invention is defined as follows:

a) the level of the index directly processed by the ETL from the source system is defaulted to 0, and the index directly accessed into the index system from the source system and the index with empty attribute of 'related index range' in the ETL processing index are included;

b) for ETL processing indexes with 'related index range' not being empty and indexes used in derivative index formulas, the level of each index is the maximum value of the level +1, and therefore the sequence relation of the levels is confirmed.

For step (C), the engine processing the index includes: and analyzing, loading, preprocessing and processing the data file and the configuration information of the index to generate index fact data. The actual processing flow of the index is as follows, and the flow is triggered when any source system data arrives:

1) the source system is placed into a state of processing. As the judgment basis for whether the cross-multi-system index is processed

2) And screening out a list of indexes to be processed. According to the 'system source' attribute of the index, all indexes containing the source system are taken out. The following indexes are also required to be removed for the indexes across multiple systems: a) in the arrival state table of the source system, the system data in the multiple systems do not arrive and the pre-dependency of the index processing can not be met

b) The condition a) does not exist, more than two systems are arranged among the plurality of crossed systems, the data arrival state of the systems is in processing, concurrent conflict exists, the index is set to be a conflict index table, and the subsequent continuous processing is carried out

3) And (5) index processing statement preprocessing. And preloading processing sentences of the indexes according to different types of the indexes.

a) ETL processing index-SQL processing index splice SQL statement, replace common variable as actual value (e.g. replace time variable by current date)

b) ETL processing index-Perl processing index splicing Perl execution statement and preparing script parameters

c) Deriving an index analysis formula, replacing a common variable as an actual value, taking the common measurement (intersection) of the formula indexes, and splicing SQL processing statements

4) The index is dependent on the data load. And data depended by the index to be processed is loaded in advance, so that the efficiency of data query is improved.

a) SQL/Perl processing index takes data in' index related range

b) The derived index takes the range data of the used index in the formula

5) And (5) index processing. Processing indexes according to levels, calling different operators according to different index types during processing to calculate, and comprising the following steps: SQL operator, Perl operator, derivative index operator, can be extended according to need

6) And (5) processing the conflict indexes. And (3) checking the condition b) in the step (2), skipping the step to continue to execute the operation if the condition b) still has conflict, otherwise, indicating that all the systems spanned by the index have finished processing, meeting the pre-dependency of the index processing, and repeating the processing processes of 3), 4) and 5) once again to enable the source system to reach the state of completion. The method is used as a judgment basis for judging whether the multi-system-crossing indexes are processed or not.

In the engine processing, the processes of 3), 4) and 5) can support the expansion of the index type, and the processing statement processing, the data preloading processing and the processing operator processing of the new index type are provided in a plug-in mode, so that the expansion can be completed, and the development amount and the test work are reduced.

The engine processing provides a uniform processing mode, so that each accessed source system and the subsequent newly added system can be fused into the index system of the invention through simple adaptive conversion; in addition, analysis and processing operation of the index formula are automated, so that a large amount of development work is saved besides the function of supporting independent maintenance of business personnel, the efficiency of the whole index calculation flow is improved, and the defects in code development are reduced.

Example 4:

in another embodiment of the present invention, in addition to the above processing method, the method further includes: the generated index fact data can be inquired through a front end, and result display of the index is completed.

According to the invention, a set of complete index configuration and processing engine scheme is established, so that the automatic processing from index configuration to processing of the whole process is realized; the standardized index definition is utilized to carry out integrated analysis on the whole index system, and the requirements of flexible operation and autonomous maintenance in subsequent index management are met; and the analysis and processing operation of the indexes under different classifications, measurement and frequencies in the indexes are finished by a uniform engine processing flow.

In the above description of various embodiments of an index arrangement processing method provided in the present application, embodiments of an index arrangement processing system provided in the present application are described below.

FIG. 4 is an architecture diagram of an index configuration processing system according to an embodiment of the present invention, as shown, the system comprising:

an index classification module 100, configured to classify an index according to an index source, and define a measurement type corresponding to the index;

the index configuration processing module 200 is configured to perform configuration processing and engine processing on the index according to the template to which the index belongs;

and an index result output module 300, configured to output the index after the engine is processed.

FIG. 5 is a block diagram of the metric classification module 100 in the system of FIG. 4; referring to fig. 5, in another embodiment of the present invention, the index classification module 100 includes:

a classification unit 110, configured to classify the indexes according to the categories of the ETL processing index, the derived index, and the manual ledger index;

a defining unit 120, configured to define the index as a measure type of time, institution and currency.

FIG. 6 is a block diagram of the index configuration processing module 200 of the system of FIG. 4; referring to fig. 6, in another embodiment of the present invention, the index configuration processing module 200 includes:

the template importing unit 210 is configured to select a corresponding template according to a category to which the index belongs to perform index configuration, and import the index into the template;

and an engine processing unit 220, configured to analyze and verify the configuration information of the imported template, and perform engine processing on the index.

In another embodiment of the present invention, the engine processing the index includes:

and analyzing, loading, preprocessing and processing the data file and the configuration information of the index to generate index fact data.

In a final embodiment of the present invention, the outputting the index after the engine machining includes: the generated index fact data can be inquired through a front end, and result display of the index is completed.

The respective embodiments of the index arrangement machining system described above are completely identical to the corresponding technical contents of the index arrangement machining method, and will not be described in detail here in order to avoid redundancy.

Through the above description of the embodiments, those skilled in the art will clearly understand that the present invention can be implemented by combining software and a hardware platform. With this understanding in mind, all or part of the technical solutions of the present invention that contribute to the background can be embodied in the form of a software product, which can be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., and includes instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods according to the embodiments or some parts of the embodiments of the present invention.

It will be appreciated by those skilled in the art that the foregoing is only illustrative of the embodiments of the invention, and that no limitation to the scope of the invention is intended thereby, such equivalents being within the scope of the invention as defined by the claims appended hereto.

Claims

1. An index configuration processing method, characterized by comprising:

outputting the index processed by the engine;

wherein the classifying the indexes and defining the metric types corresponding to the indexes comprise:

classifying the indexes according to the types of ETL processing indexes, derived indexes and manual standing book indexes;

defining the index by measuring types of time, mechanism and currency;

the step of performing configuration processing and engine processing on the indexes according to the templates of the classes to which the indexes belong comprises the following steps:

selecting a corresponding template according to the category to which the index belongs to carry out index configuration, and importing the index into the template;

analyzing and checking the configuration information of the imported template, and performing engine processing on the index, wherein the engine processing on the index comprises the following steps: and analyzing, loading, preprocessing and processing the data file and the configuration information of the index to generate index fact data.

2. The method of claim 1, wherein outputting the engine processed metrics comprises:

the generated index fact data can be inquired through a front end, and result display of the index is completed.

3. A system, characterized in that the system comprises:

the index result output module is used for outputting the index processed by the engine;

wherein the index classification module comprises:

the classification unit is used for classifying the indexes according to the types of the ETL processing indexes, the derived indexes and the manual standing book indexes;

the defining unit is used for defining the indexes by measuring types of time, mechanisms and currency;

wherein, index configuration processing module includes:

the template importing unit is used for selecting a corresponding template according to the category to which the index belongs to perform index configuration and importing the index into the template;

the engine processing unit is used for analyzing and verifying the configuration information of the imported template and performing engine processing on the index, wherein the engine processing on the index comprises the following steps: and analyzing, loading, preprocessing and processing the data file and the configuration information of the index to generate index fact data.

4. The system of claim 3, wherein said outputting the engine processed metrics comprises: