Data violation monitoring method and device
Technical Field
The invention relates to the technical field of data monitoring, in particular to a method and a device for monitoring violation of data.
Background
With the development and the deepening of information technology, data gradually become valuable strategic assets of enterprises. The main data management is to capture key data which have high business value and are reused by each business department in an enterprise from the disordered data of the enterprise for management, and construct a single, accurate and authoritative data source, so that the overall data quality of the enterprise is improved, the data asset value is improved, the business innovation is promoted, and the enterprise competitiveness is comprehensively enhanced.
With the continuous development of informatization, the data condition is more and more complex, and higher requirements are put forward for data monitoring stored daily. In the past, data maintenance is a practical problem which is difficult to solve, a specific person needs to be arranged to complete monitoring work, and due to the fact that abnormal conditions are easily ignored in a manual monitoring mode, the data violation monitoring function needs to be developed and researched in a targeted mode.
In the existing data violation monitoring process, a corresponding check rule is set for each item of metadata, and a special code needs to be written for the check rule of each data object. When the demand changes, the code development is required to be carried out continuously, the development cost is increased, unnecessary waste is caused, and the data monitoring efficiency is reduced. Therefore, the mode cannot meet the requirement that data violation monitoring changes caused by continuous increase of data objects and continuous conversion of business objects in the enterprise development process.
Disclosure of Invention
In view of the foregoing analysis, the present invention aims to provide a method and an apparatus for monitoring violation of rules of data, so as to solve the problem of low monitoring efficiency caused by writing a corresponding check rule according to metadata in the existing violation monitoring process.
The purpose of the invention is mainly realized by the following technical scheme:
a data violation monitoring method comprises the following steps:
acquiring data objects and data information of the data objects including metadata, inter-table relations and mapping relations;
generating N judgment conditions for carrying out conditional resource classification on the data objects by configuring a condition judgment template according to the metadata of the data objects; each judgment condition corresponds to one check rule by configuring a check rule template; thus establishing a check rule base;
screening out one or more check rules corresponding to each metadata of the data object from the check rule base to obtain a check rule packet aiming at the current data object;
inputting the main data of the data object; and judging the metadata of the data object in sequence according to the judgment condition of each verification rule corresponding to each metadata in the verification rule packet, and when all the metadata in the main data of the data object are verified to be qualified, inputting the verified main data into a database.
The invention has the following beneficial effects: the condition judgment template and the verification rule template are set to generate the verification rule base, the verification rule packet generated by the verification rule base is used for monitoring the main data of the data object, and a special code does not need to be written for each metadata, so that the development cost is reduced; meanwhile, the data packet for the data object, which is set in the invention, can complete the inspection of all metadata in the data object and all check rules corresponding to the metadata at one time, thereby effectively improving the accuracy and the monitoring efficiency of the data violation monitoring.
On the basis of the scheme, the invention is further improved as follows:
further, the method for establishing the check rule base comprises the following steps:
step S201, screening metadata information in data information of a data object;
step S202, parameter setting is carried out on the set condition judgment template, N judgment conditions are generated, and the classification of the metadata is judged;
step S203, setting parameters of the set check rule template to enable each judgment condition to correspond to one check rule;
and step S204, summarizing and storing the metadata information, the N judgment conditions and the verification rules corresponding to the judgment conditions to form a verification rule base for the metadata of the data object.
The beneficial effect of adopting the further scheme is that: by giving a specific mode of establishing a verification rule base, the incidence relation among metadata, a condition judgment template and a rule verification template can be established, various verification rules can be met through rule configuration without compiling special codes, and the development cost is reduced;
further, the condition judging template is used for,
acquiring a subject object judged by conditions;
providing a function operation relation;
providing a comprehensive operation relation symbol;
providing a judgment constant value comprising numerical values, characters, dates and reference relations;
and setting a condition judgment template, selecting a judgment constant value and an operation relation character provided in the template to establish an operation relation formula for condition judgment, and classifying the subject object subjected to condition judgment.
Further, the verification rule template at least comprises:
6) and (4) integrity checking: the field value is not null;
7) and (3) checking uniqueness: checking the field repeatedly;
8) authority verification: whether the field value meets the requirements of format and length;
9) and (3) validity checking: the field value is required in the corresponding reference relationship;
10) and (3) consistency checking: the field value is equal to an item value.
Further, the check rule template further comprises a self-defined template, the self-defined template is realized by setting an expression, and the expression is obtained by randomly combining the following controls:
and, or, alternatively, if, exactly, otherwise, inclusive, exclusive, non, null, non-null.
The beneficial effect of adopting the further scheme is that: when the common rule template can not meet the application requirements, the rule verification can be realized through the user-defined template, and the rule verification requirements on the metadata under different scenes and different services are met.
Further, the method for performing rule check on the data object comprises the following steps:
step S401, inputting main data of a data object;
s402, sequentially acquiring metadata of an input data object main body;
step S403, judging whether the metadata meets one or more corresponding verification rules according to the verification rule packet, if so, inputting the verified main data of the data object into a database, wherein the main data of the data object is qualified in verification.
The beneficial effect of adopting the further scheme is that: by utilizing the data packet aiming at the data object, the verification of all metadata in the data object and all verification rules corresponding to the metadata can be completed at one time, and the accuracy and the monitoring efficiency of data violation monitoring are effectively improved.
Further, when the data object main data is unqualified in verification, caching the main data failed in storage, and carrying out violation prompt on the metadata in the main data failed in verification according to the verification rule corresponding to the metadata failed in verification.
And further, modifying the metadata of the violation main data according to the violation prompt, re-inputting the main data of the data object, and verifying the data object.
A data violation monitoring device comprises a data object acquisition module, a verification rule base generation module, a verification rule packet generation module and a data object monitoring module;
the data object acquisition module is used for acquiring data objects and data information of the data objects including metadata, inter-table relations and mapping relations;
the verification rule base generation module is connected with the data object acquisition module, and generates N judgment conditions for carrying out conditional resource classification on the data object through configuring a condition judgment template according to the metadata of the data object; each judgment condition corresponds to one check rule by configuring a check rule template; thus establishing a check rule base;
the verification rule packet generating module is connected with the verification rule base generating module and is used for screening out one or more verification rules corresponding to each metadata of the data object from the verification rule base to obtain a verification rule packet for the current data object;
the data object monitoring module is connected with the data object acquisition module and the verification rule packet generation module and is used for inputting the main data of the data object; and judging the metadata of the data object in sequence according to the judgment condition of each verification rule corresponding to each metadata in the verification rule packet, and when all the metadata in the main data of the data object are verified to be qualified, inputting the verified main data into a database.
Further, the check rule base generation module includes: a data information acquisition module, a condition judgment template, a condition setting module and a check rule template,
the data information acquisition module is used for acquiring metadata information in the data information of the data object by a user;
the condition judgment template is preset and internally comprises all elements for generating a condition judgment formula;
the condition setting module is used for setting a condition judgment template and generating N judgment conditions to judge the classification of the metadata;
the checking rule template is preset and internally comprises all elements for generating the checking rule;
the check rule setting module is used for setting the check rule template so that each judgment condition corresponds to one check rule;
and the summarizing storage module is used for summarizing and storing the metadata information, the N judging conditions and the verification rules corresponding to the judging conditions to form a verification rule base for the metadata of the data object.
In the invention, the technical schemes can be combined with each other to realize more preferable combination schemes. Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
The drawings are only for purposes of illustrating particular embodiments and are not to be construed as limiting the invention, wherein like reference numerals are used to designate like parts throughout.
FIG. 1 is a flow chart of a method for monitoring data violations in an embodiment of the present invention;
FIG. 2 is a flowchart of a method for establishing a verification rule base according to an embodiment of the present invention;
fig. 3 is a schematic connection diagram of a data violation monitoring apparatus according to an embodiment of the present invention.
Detailed Description
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate preferred embodiments of the invention and together with the description, serve to explain the principles of the invention and not to limit the scope of the invention.
A specific embodiment of the present invention discloses a data violation monitoring method, as shown in fig. 1, the steps are as follows:
step S1: acquiring data objects and data information of the data objects including metadata, inter-table relations and mapping relations;
step S2: generating N judgment conditions for carrying out conditional resource classification on the data objects by configuring a condition judgment template according to the metadata of the data objects; each judgment condition corresponds to one check rule by configuring a check rule template; thus establishing a check rule base;
step S3: screening out one or more check rules corresponding to each metadata of the data object from the check rule base to obtain a check rule packet aiming at the current data object;
step S4: inputting the main data of the data object; and judging the metadata of the data object in sequence according to the judgment condition of each verification rule corresponding to each metadata in the verification rule packet, and when all the metadata in the main data of the data object are verified to be qualified, inputting the verified main data into a database.
Compared with the prior art, the data violation monitoring method provided by the embodiment has the advantages that the condition judgment template and the verification rule template are set to generate the verification rule base, the verification rule packet generated by the verification rule base is used for monitoring the main data of the data object, a special code does not need to be written for each metadata, and the development cost is reduced; meanwhile, the data packet for the data object, which is set in the invention, can complete the inspection of all metadata in the data object and all check rules corresponding to the metadata at one time, thereby effectively improving the accuracy and the monitoring efficiency of the data violation monitoring.
Preferably, as shown in fig. 2, the method for establishing the verification rule base includes:
step S201, screening metadata information in data information of a data object;
step S202, parameter setting is carried out on the set condition judgment template, N judgment conditions are generated, and the classification of the metadata is judged;
specifically, the condition judging template is used for,
acquiring a subject object judged by conditions;
providing a function operation relation;
providing a comprehensive operation relation symbol;
providing a judgment constant value comprising numerical values, characters, dates and reference relations;
and setting a condition judgment template, selecting a judgment constant value and an operation relation character provided in the template to establish an operation relation formula for condition judgment, and classifying the subject object subjected to condition judgment.
Step S203, setting parameters of the set check rule template to enable each judgment condition to correspond to one check rule;
specifically, the check rule template may include the following form:
1) and (4) integrity checking: the field value is not null;
2) and (3) checking uniqueness: checking the field repeatedly;
3) authority verification: whether the field value meets the requirements of format and length;
4) and (3) validity checking: the field value is required in the corresponding reference relationship;
5) and (3) consistency checking: the field value is equal to an item value.
Other verification rule templates can also be designed according to the requirements of the data object.
And step S204, summarizing and storing the metadata information, the N judgment conditions and the verification rules corresponding to the judgment conditions to form a verification rule base for the metadata of the data object.
Under a certain judgment condition, a verification format corresponding to the judgment condition is formed by setting parameters in the verification rule template, and the specific parameter setting is arbitrarily set according to the requirements of a user, which is not the protection point of the invention.
The method is used for expanding the application range of the method and meeting the control on data under different scenes and different services. The check rule template can also comprise a self-defined template, the self-defined template is realized by setting an expression, and the expression is obtained by randomly combining the following controls: and, or, alternatively, if, exactly, otherwise, inclusive, exclusive, non, null, non-null. The 9 controls can be freely combined to set various rules. For example, to set that when a person belongs to a unit, his department cannot be null, we can set the expression [ if the organization type is a unit, then the department is not null ].
Preferably, the method for rule checking of the data object includes:
step S401, inputting main data of a data object;
s402, sequentially acquiring metadata of an input data object main body;
step S403, judging whether the metadata meets one or more corresponding verification rules according to the verification rule packet, if so, inputting the verified main data of the data object into a database, wherein the main data of the data object is qualified in verification.
Preferably, when the data object main data is unqualified in verification, caching the main data failed to be put in storage, and carrying out violation prompt on the metadata in the main data failed to be verified according to the verification rule corresponding to the metadata failed to be verified.
Preferably, the metadata of the violation main data is modified according to the violation prompt, the main data of the data object is re-entered, and the data object is verified.
And when the main data which are failed to be continuously put in storage exceed a certain number, stopping receiving the main data to be put in storage, analyzing the reason of the failure of putting the main data in storage according to the violation prompt, and continuously receiving the main data to be put in storage after subsequent data are modified.
In another embodiment of the present invention, a data violation monitoring apparatus is provided, as shown in fig. 3, including a data object obtaining module, a verification rule base generating module, a verification rule packet generating module, and a data object monitoring module;
the data object acquisition module is used for acquiring data objects and data information of the data objects including metadata, inter-table relations and mapping relations;
the verification rule base generation module is connected with the data object acquisition module, and generates N judgment conditions for carrying out conditional resource classification on the data object through configuring a condition judgment template according to the metadata of the data object; each judgment condition corresponds to one check rule by configuring a check rule template; thus establishing a check rule base;
the verification rule packet generating module is connected with the verification rule base generating module and is used for screening out one or more verification rules corresponding to each metadata of the data object from the verification rule base to obtain a verification rule packet for the current data object;
the data object monitoring module is connected with the data object acquisition module and the verification rule packet generation module and is used for inputting the main data of the data object; and judging the metadata of the data object in sequence according to the judgment condition of each verification rule corresponding to each metadata in the verification rule packet, and when all the metadata in the main data of the data object are verified to be qualified, inputting the verified main data into a database.
Preferably, the check rule base generation module includes: a data information acquisition module, a condition judgment template, a condition setting module and a check rule template,
the data information acquisition module is used for acquiring metadata information in the data information of the data object by a user;
the condition judgment template is preset and internally comprises all elements for generating a condition judgment formula;
the condition setting module is used for setting a condition judgment template and generating N judgment conditions to judge the classification of the metadata;
the checking rule template is preset and internally comprises all elements for generating the checking rule;
the check rule setting module is used for setting the check rule template so that each judgment condition corresponds to one check rule;
and the summarizing storage module is used for summarizing and storing the metadata information, the N judging conditions and the verification rules corresponding to the judging conditions to form a verification rule base for the metadata of the data object.
The method embodiment and the device embodiment are based on the same principle, and the related parts can be referenced mutually, and the same technical effect can be achieved.
Those skilled in the art will appreciate that all or part of the flow of the method implementing the above embodiments may be implemented by hardware associated with computer program instructions, and the program may be stored in a computer readable storage medium. The computer readable storage medium is a magnetic disk, an optical disk, a read-only memory or a random access memory.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention.