CN105824870A - Classification and quality inspection method and system based on verification rules - Google Patents
Classification and quality inspection method and system based on verification rules Download PDFInfo
- Publication number
- CN105824870A CN105824870A CN201610030974.3A CN201610030974A CN105824870A CN 105824870 A CN105824870 A CN 105824870A CN 201610030974 A CN201610030974 A CN 201610030974A CN 105824870 A CN105824870 A CN 105824870A
- Authority
- CN
- China
- Prior art keywords
- rule
- classification
- verification rule
- data
- verification
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/273—Asynchronous replication or reconciliation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Economics (AREA)
- Strategic Management (AREA)
- Technology Law (AREA)
- Marketing (AREA)
- General Business, Economics & Management (AREA)
- Finance (AREA)
- Development Economics (AREA)
- Computing Systems (AREA)
- Accounting & Taxation (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a classification and quality inspection method and system based on verification rules. The classification and quality inspection method based on verification rules comprises the following steps: monitoring data sheets in a database and creating new verification rules according to the monitored dimensionality; performing important type classification on the verification rules and classifying operation according to the import type classification; setting an actuating frequency for newly added operation; starting up the newly added operation for quality inspection; performing quality inspection to determine whether abnormal data exist or not, and if yes, feeding back according to alarming rules. The classification and quality inspection system comprises a monitoring module, a rule setting module, a processing module, a quality inspection module, an operation module and the database. According to the classification and quality inspection method and system based on the verification rules, background monitoring of the database is changed into visual operation of a platform; the complex inspection rules are subjected to effective classification and management, and classification and regular monitoring according to importance are realized.
Description
Technical field
The invention belongs to database processing technical field, relate to a kind of data quality checking system, particularly relate to the quality inspection system of financial Information data；Meanwhile, the invention still further relates to the verification rule of a kind of above-mentioned quality inspection system.
Background technology
Along with financial Information data present the growth of " geometrical progression " formula, the information data gathered are carried out quality inspection timely, effective, ensure the correctness of data, follow-up data are stored, distribute, derive to calculate and apply and play important early warning guaranteeing role, be that bottom data produces upper the key link.Artificial enquiry database mode is taked in traditional quality inspection, and efficiency is low, be difficult to the source of pinpointing the problems, big to data base's pressure, for indivedual a small amount of problem manual types still can, but be difficult to tackle big data environment.
Summary of the invention
The technical problem to be solved in the present invention is, by by numerous and diverse verification rule by effective Classification Management, it is achieved that classify according to the carrying out of importance, timing monitoring and carry out quality inspection according to the cycle.
Solve above-mentioned technical problem, the invention provides a kind of classification based on verification rule and quality detecting method, including,
Tables of data in data base is monitored, sets up new verification rule according to the dimension of described monitoring；
Described verification rule is carried out important kind classification；
And according to the classification of described important kind, operation is classified；And newly-increased operation is set execution frequency；
Open newly-increased operation, and carry out quality inspection；
Whether quality inspection exists abnormal data, if existing, then feeds back according to alarm rule.
Further, the method being monitored described in is that the data base needing monitoring is added data source.
Further, new verification rule is according to increasing one by one or EXCEL batch importing.
Further, the attribute of described new verification rule includes, needs the tables of data of inspection, the description of rule, important kind classification, corresponding inspection personnel.
Further, described execution frequency includes, per hour, every day, monthly, every year.
Further, described detect whether that the method that there is abnormal data is, for operation a certain in system, perform by the frequency set, after running succeeded, if the data base of verification exists the record violating a certain rule under this operation, then at the alert interface of quality inspection system, the rule of correspondence and abnormal data amount information can occur.
Further, described new verification rule includes: logical relation between field and field, record and record between logical relation, table and table between logical relation, described field includes: numeric type field, time type field, parametric type field, integer type field carry out important kind classification；Logical relation between described record from record is: in the most individual table, the logical relation between different records verifies；Logical relation between described table and table refers to: have internal relation different tables record between verification.
Further, described verification rule carries out important kind classification include: promptness verification rule classification, accuracy verification rule classification, completeness check rule classification.
Present invention also offers a kind of classification based on verification rule and quality inspection system, including,
Monitoring module, described monitoring module in order to the tables of data in data base is monitored,
Rule sets up module, and described rule sets up module in order to set up new verification rule according to the dimension of described monitoring；
Processing module, described processing module in order to carry out important kind classification by described verification rule, and classifies operation according to the classification of described important kind；And newly-increased operation is set execution frequency；
Quality testing module, whether described quality testing module exists abnormal data in order to quality inspection, if existing, then feeds back according to alarm rule；
Operation unit, described operation unit is in order to carry out newly-increased job task；And perform job task；
Data base, described data base treats the data of quality inspection in order to store.
Further, described rule set up module set up include: promptness verification rule, accuracy verification rule and completeness check rule.
Beneficial effects of the present invention:
1) database background application monitoring is embodied as the visualized operation of platform by the method for the present invention.
2) method of the present invention by numerous and diverse verification rule by effective Classification Management, it is achieved that classify according to the carrying out of importance, timing monitoring.
3) the system automation operation quality inspection system of the present invention, the monitoring of rule is tested by the inspection of data itself being converted into confrontation calibration, simultaneously, take automatic programming execution and the visual alert interface of personal monitoring, energy orientation problem point rapidly, is greatly improved efficiency and the effect of data quality checking.
Accompanying drawing explanation
Fig. 1 is the classification based on verification rule in one embodiment of the invention and quality detecting method schematic flow sheet.
Fig. 2 is the structural representation of verification rule new in Fig. 1.
Fig. 3 is the method flow schematic diagram detecting whether to exist abnormal data in one embodiment of the invention.
Fig. 4 is the classification based on verification rule in one embodiment of the invention and the structural representation of quality inspection system.
Fig. 5 is the classification based on verification rule in one embodiment of the present invention and quality detecting method schematic flow sheet.
Detailed description of the invention
For making the object, technical solutions and advantages of the present invention clearer, below in conjunction with specific embodiment, and referring to the drawings, the present invention is described in more detail.
Refer to Fig. 1 is the classification based on verification rule in one embodiment of the invention and quality detecting method schematic flow sheet.
Classification and quality detecting method flow process in the present embodiment are as follows:
Tables of data in data base is monitored by step S101, sets up new verification rule according to the dimension of described monitoring.Preferred as in the present embodiment, described in the method that is monitored be that the data base needing monitoring is added data source.Such as, can be contrasted with data source or tripartite by the time of data loading, examine the warehouse-in amount difference of same time point or warehouse-in record the earliest to disclose the time difference of bulletin the earliest with data source, in order to guarantee that promptness verifies.
Described verification rule is carried out important kind classification by step S102, preferred as in the present embodiment, described verification rule carries out important kind classification and includes: promptness verification rule classification, accuracy verification rule classification, completeness check rule classification, specifically, described promptness verification rule classification, can be contrasted with data source or tripartite by the time of data loading.Such as, ODBC Open Database Connection (Open Database Connectivity,) use open services architecture (WOSA, Windows Open Services Architecture) in about the ingredient of data base, ODBC establishes one group of specification, and provides one group of standard application DLL API to database access.These API utilize SQL database to complete its major part task.Further, described new verification rule is according to increasing one by one or EXCEL batch importing.
Specifically, in step s 102, the configuration information needed for web page includes: the data source of web services needs, action type (such as, add, revise, delete or inquire about), required parameter and return value, result filtercondition, web services title and URL, version number, service describing.Then relational database query statement is dynamically generated according to the action type configured, required parameter, return value, filtercondition.Then, generate the data model for preserving wall scroll data record, carry out the significance level classification of rule, dynamically generate the data model code for preserving wall scroll data record according to the value information that returns of configuration, generate data model.
Operation is classified by step S103 according to the classification of described important kind；And newly-increased operation is set execution frequency, described operation referred to classify the most scattered verification rule according to frequency of seeking unity of action, time, it is achieved the execution task of unified management.Further, operation is classified based on: numeric type field, time type field, parametric type field, integer type field.Or by the angular divisions of quality inspection it is: promptness verification rule, accuracy verification rule, completeness check rule.Preferred as in the present embodiment, sets newly-increased operation and performs frequency and include, per hour, every day, monthly, every year.
Step S104 opens newly-increased operation, and carries out quality inspection.After setting execution frequency, quality inspection can be proceeded by.Those skilled in the art can understand, the quality inspection carrying out operation includes but not limited to: be responsible for listed company, fund, manage money matters, insure, bond, mechanism, macroscopic view, the audit work of all kinds of finance datas such as industry and mistake collection work (including daily data examination and data purge)；Or, it is responsible for the censorship of related financial data database table structure technical documentation.
Whether the quality inspection of step S105 exists abnormal data, if existing, then feeds back according to alarm rule.Whether described quality inspection exists abnormal data, and the verification rule according to arranging judges, if meeting verification rule, does not reports to the police.
Specifically, in step S105, as there is following rule A:
Any one issuance of stock date is necessarily less than listing date；
So, corresponding table B in described rule A correspondence database carries out full table inspection, when described table B exists data [issuing date is more than or equal to listing date], then can show rule A (comprising data table name, rule of correspondence name, exception record bar number) at the alert interface of quality inspection platform.If the data in table B all meet [issuing date is less than listing date], this rule is run complete, then will not be shown at the alert interface of quality inspection platform.
Fig. 2 is the structural representation of verification rule new in Fig. 1.
The attribute of described new verification rule includes: the tables of data 201 that needs check, regular description 202, important kind classification 203, corresponding inspection personnel 204.In particular it is required that the tables of data 201 checked includes, table 1, table 2, table 3 and table N.The description 202 of described rule includes, logical relation between field and field, record and record between logical relation, table and table between logical relation.Important kind classification 203 includes, promptness verification rule classification, accuracy verification rule classification, completeness check rule classification.Described promptness verification rule classification, can be contrasted with data source or tripartite by the time of data loading, warehouse-in amount difference or the warehouse-in record the earliest of examining same time point disclose the time difference of bulletin with data source, the earliest in order to guarantee that promptness verifies.Described accuracy verification rule classification, the accuracy of logical relation between field and field, record and record between the accuracy of logical relation, table and table between the accuracy of logical relation.Described completeness check rule classification, including: field and field, record and record, the integrity of table and table, such as, the integrity of record mainly verified by the integrity of record by the business relations of record between exploitation table, transformational relation.The integrity of field and field needs, the integrity of source table record and purpose table record corresponding field, or with the inspection of tripartite's corresponding field.The inspection personnel 204 of described correspondence is finance data Quality Inspector.
Fig. 3 is the method flow schematic diagram detecting whether to exist abnormal data in one embodiment of the invention.
In the present embodiment, detect whether that the method that there is abnormal data is:
Step S301 for operation a certain in system, is performed by the frequency set, preferred as in the present embodiment, and the frequency of setting is, per hour, every day, monthly, every year.The most a certain described operation includes but not limited to, refers to according to the verification rule arranged, and after carrying out significance level classification according to frequency of seeking unity of action, the time of seeking unity of action, it is achieved the execution task of unified management.
Whether step S302 runs succeeded？If success, then enter step S303；Whether perform according to the frequency set firstly the need of the work flow judged in step S301, i.e. judge whether to run succeeded.If unsuccessful, the most do not carry out below step.
Whether the data base of step S303 verification exists the record of the rule violated under this operation, described a certain rule is it may be that regular in the verification of field aspect, and such as numeric type field can only deposit Arabic numerals, if phone number field can only be numeral, id number can only be numeral；Or the value of numeric type field must within the specific limits, and the such as input amount of money have to be larger than 0；Or the value kind of parametric type field have to be larger than field value condition and must be in corresponding parameter area, such as parameter A > parameter B is when being essential condition, classification is the most meaningful.Or there is logic comparison, such as parameter A between field>parameter B, parameter A+ parameter B<parameter C ....
The data base of step S304 verification exists and violates the record of a certain rule under this operation, according to the judgement of above-mentioned steps, determine in operation, whether there is the record broken the rules.Owing to have employed automated job quality inspection mode, the inspection of data itself is converted into confrontation calibration and tests the monitoring of rule, can orientation problem point rapidly, coordinate the most artificial upper alert process, it is possible to improve efficiency and the effect of data quality checking greatly simultaneously.
There is the rule of correspondence and abnormal data amount information at quality inspection alert interface in step S305.Those skilled in the art can understand, described quality inspection alert interface can use UI (User Interface) to process in front end, and backstage can select to include but not limited to that the programming language of increasing income such as JAVA realizes.Described abnormal data amount information includes but not limited to: the verification rule between data is not inconsistent, record between verification regular be not inconsistent and the verification rule of field and field is not inconsistent.
Fig. 4 is the classification based on verification rule in one embodiment of the invention and the structural representation of quality inspection system.
In the present embodiment, classification based on verification rule and quality inspection system include:
Monitoring module 401, described monitoring module 401 in order to the tables of data in data base is monitored, described in the method that is monitored be that the data base needing monitoring is added data source by described monitoring module 401.Such as, can be contrasted with data source or tripartite by the time of data loading, examine the warehouse-in amount difference of same time point or warehouse-in record the earliest to disclose the time difference of bulletin the earliest with data source, in order to guarantee that promptness verifies.
Rule sets up module 402, and described rule sets up module 402 in order to set up new verification rule according to the dimension of described monitoring；Verification rule in described rule sets up module 402, it is by manual analysis data, according to existing specification agreement or the Fluctuation of discovery record, and then the logical relation between determining between field and field, recording and record, between table and table, these affinity criterions are turned to the discernible language of computer, formation can be realized the logical constraint to data with the SQL of independent operating by computer or human assistance computer.Dimension according to described monitoring is set up new verification rule and is included: numeric type field, time type field, parametric type field, integer type field etc. can develop its verification rule from multi-angle.Include from by the angular divisions checked: promptness verification rule, accuracy verification rule, completeness check rule simultaneously.
Specifically, promptness in the present embodiment verification rule, mainly contrasted with data source or tripartite by the time of data loading, examine the warehouse-in amount difference of same time point or warehouse-in record the earliest and data source to disclose the time difference of bulletin the earliest.Specifically, accuracy in the present embodiment verification rule, mainly according to 1) the verification rule of field aspect: numeric type field can only deposit Arabic numerals, if phone number field can only be numeral.The value of numeric type field must within the specific limits, the such as time, and the value kind of parametric type field have to be larger than the value condition of field and in corresponding parameter area, and must there is logic comparison between field.2) verification rule between record: the logical relation verification between difference records in the table in a SQL, such as, the time order and function relation between the record in table, i.e. determine verification relation according to timing node.The most such as, the numerical values recited relation between record.3) between tables of data verification rule: have internal relation different tables record between verification.Such as, between table record exist dependence, between table record there is transformational relation.
Specifically, completeness check rule in the present embodiment includes:
1) record completeness check, the integrity of record mainly verified by the integrity such as recorded by the business relations recorded between exploitation table, transformational relation.
1.1) business relations of record between table
The record of such as A table depends on the record (such as the listing on the same day of certain security, then it is necessary to have this quotation record in quotations list) of B table；
1.2) transformational relation (the stock abbreviation field such as A table is then possibly converting to stock code to B table) of record between table
2) field completeness check
2.2) source table record is checked with purpose table record corresponding field, and such as, the unit such as source literary name section A is: Wan Yuan, and the unit of field B of purpose table is: unit；Forward field B to after then needing field A is multiplied by 10000, then need the data corresponding relation verifying purpose table the most correct.
2.3) with the inspection of third party's corresponding field, such as, the Data Comparison of company of the same trade.As ten thousand, the third party such as sequence.
Processing module 403, described processing module 403 in order to carry out important kind classification by described verification rule, and classifies operation according to the classification of described important kind；And newly-increased operation is set execution frequency；Preferred as in the present embodiment, the frequency arranged in described processing module 403 is, per hour, every day, monthly, every year.The most a certain described operation includes but not limited to, refers to according to the verification rule arranged, and after carrying out significance level classification according to frequency of seeking unity of action, the time of seeking unity of action, it is achieved the execution task of unified management.Important kind is categorized as rule and sets up in module 402: promptness verification rule classification, accuracy verification rule classification, completeness check rule classification.
Quality testing module 404, whether described quality testing module 404 exists abnormal data in order to quality inspection, if existing, then feeds back according to alarm rule；The quality inspection in quality testing module 404 carried out operation includes but not limited to: be responsible for listed company, fund, manage money matters, insure, bond, mechanism, macroscopic view, the audit work of all kinds of finance datas such as industry and mistake collection work (including daily data examination and data purge)；Or, it is responsible for the censorship of related financial data database table structure technical documentation.Abnormal data is i.e. the data not meeting verification rule.
Operation unit 405, described operation unit 405 is in order to carry out newly-increased job task；And perform job task；Perform complete job task at described operation unit 405, i.e. according to verification rule, finance data is classified and quality inspection.
Data base 406, and described data base 406 is in order to store the data treating quality inspection.Database background application monitoring can be embodied as the visualized operation of platform in the present embodiment；And by numerous and diverse verification rule by effective Classification Management, it is achieved that classify according to the carrying out of importance, timing monitors.Specifically, for operation a certain in system, perform by the frequency set, after running succeeded, if the data base of verification exists the record violating a certain rule under this operation, then at the alert interface of quality inspection system, the information such as the rule of correspondence and abnormal data amount can occur.The responsible person of this rule then can see relative alarm the very first time.It addition, the rule being newly uploaded to system cannot be included operation before examination & verification in and perform.Whether auditor " is audited " is screened rule by system field, filter out the rule not yet audited evaluate whether meet rule development specifications, whether the data of checking meet logic, grammer is the most correct.
Additionally, in the present embodiment rule set up module 402 also in order to: according to artificial agreement during data loading, the Analysis and summary of problem data, between field and field, record and record between the multiple means such as logical relation, verification rule documents is write in conjunction with verification rule development specifications, and be converted into SQL in data base, carry out data query, the practicality of proof rule.
Fig. 5 is the classification based on verification rule in one embodiment of the present invention and quality detecting method schematic flow sheet.
Rule list 50 includes: logical relation 501 between field and field, record and record between logical relation 501, table and table between logical relation 501.Following steps are carried out after setting up rule list 50:
Step S501 sets up various dimensions verification rule, step S501 can include, between artificial agreement, the Analysis and summary of problem data, logical relation, field and field between table and table, record and record between the multiple means such as logical relation, verification rule documents is write in conjunction with verification rule development specifications, and be converted into SQL in data base, carry out data query, the practicality of proof rule.
Step S502 is audited, and can be manual examination and verification, it is also possible to be the machine setting the rule of correspondence examination & verification to field.It addition, the rule being newly uploaded to system cannot be included operation before examination & verification in and perform.Whether auditor " is audited " is screened rule by system field, filter out the rule not yet audited evaluate whether meet rule development specifications, whether the data of checking meet logic, grammer is the most correct.
Whether step S503 meets rule and logic, such as, numeric type field, time type field, parametric type field, integer type field.Or by the angular divisions of quality inspection it is: promptness verification rule, accuracy verification rule, completeness check rule.
Rule is included in different operations according to rule severity level, classifying importance by step S504, described verification rule carries out important kind classification and includes: promptness verification rule classification, accuracy verification rule classification, completeness check rule classification, specifically, described promptness verification rule classification, can be contrasted with data source or tripartite by the time of data loading.Such as, ODBC Open Database Connection (Open Database Connectivity,) use open services architecture (WOSA, Windows Open Services Architecture) in about the ingredient of data base, ODBC establishes one group of specification, and provides one group of standard application DLL API to database access.These API utilize SQL database to complete its major part task.Further, described new verification rule is according to increasing one by one or EXCEL batch importing.
Step S505 setpoint frequency brings into operation operation, sets newly-increased operation and performs frequency and include, per hour, every day, monthly, every year.
Whether step S506 has abnormal data, according to above-mentioned verification rule, judges.Accuracy verification rule in the present embodiment, mainly according to 1) verification of field aspect rule: numeric type field can only deposit Arabic numerals, if phone number field can only be numeral.The value of numeric type field must within the specific limits, the such as time, and the value kind of parametric type field have to be larger than the value condition of field and in corresponding parameter area, and must there is logic comparison between field.2) verification rule between record: the logical relation verification between difference records in the table in a SQL, such as, the time order and function relation between the record in table, i.e. determine verification relation according to timing node.The most such as, the numerical values recited relation between record.3) between tables of data verification rule: have internal relation different tables record between verification.
Step S507 alert interface is reminded, and quality inspection alert interface can use UI (User Interface) to process in front end, and backstage can select to include but not limited to that the programming language of increasing income such as JAVA realizes.Described abnormal data amount information includes but not limited to: the verification rule between data is not inconsistent, record between verification regular be not inconsistent and the verification rule of field and field is not inconsistent.
Step S508 quality inspection personnel, be mainly responsible for listed company, fund, manage money matters, insure, bond, mechanism, macroscopic view, the audit work of all kinds of finance datas such as industry and mistake collection work (including daily data examination and data purge)；Or, it is responsible for the censorship of related financial data database table structure technical documentation.
Step S509 problem data, does not meets quality inspection rule after classification,
Step S510 data processing unit, feed back to alert interface remind, for operation a certain in system, perform by the frequency set, after running succeeded, if the data base of verification exists the record violating a certain rule under this operation, then at the alert interface of quality inspection system, the information such as the rule of correspondence and abnormal data amount can occur.The responsible person of this rule then can see relative alarm the very first time.
Those of ordinary skill in the field it is understood that more than; the described specific embodiment being only the present invention, is not limited to the present invention, all within the spirit and principles in the present invention; the any modification, equivalent substitution and improvement etc. done, should be included within the scope of the present invention.
Claims (10)
1. a classification based on verification rule and quality detecting method, it is characterised in that include,
Tables of data in data base is monitored, sets up new verification rule according to the dimension of described monitoring；
Described verification rule is carried out important kind classification；
And according to the classification of described important kind, operation is classified；And newly-increased operation is set execution
Frequency；
Open newly-increased operation, and carry out quality inspection；
Whether quality inspection exists abnormal data, if existing, then feeds back according to alarm rule.
Classification based on verification rule the most according to claim 1 and quality detecting method, it is characterised in that
The described method being monitored is that the data base needing monitoring is added data source.
Classification based on verification rule the most according to claim 1 and quality detecting method, it is characterised in that
New verification rule is according to increasing one by one or EXCEL batch importing.
Classification based on verification rule the most according to claim 3 and quality detecting method, it is characterised in that
The attribute of described new verification rule includes, needs the tables of data of inspection, the description of rule, important kind
Classification, corresponding inspection personnel.
Classification based on verification rule the most according to claim 1 and quality detecting method, it is characterised in that
Described execution frequency includes, per hour, every day, monthly, every year.
Classification based on verification rule the most according to claim 1 and quality detecting method, it is characterised in that
Described detect whether that the method that there is abnormal data is, for operation in system, perform by the frequency set,
After running succeeded, if there is the record violating rule under this operation in the data base of verification, then can be
There is the rule of correspondence and abnormal data amount information in quality inspection alert interface.
Classification based on verification rule the most according to claim 1 and quality detecting method, it is characterised in that
Described new verification rule includes: logical relation between field and field, record and record between logic
Logical relation between relation, table and table, described field includes: numeric type field, time type field,
Parametric type field, integer type field carry out important kind classification；Logic between described record and record is closed
System is: in the most individual table, the logical relation between different records verifies；Logic between described table and table
Relation refers to: have internal relation different tables record between verification.
Classification based on verification rule the most according to claim 1 and quality detecting method, it is characterised in that
Described verification rule carries out important kind classification include: promptness verification rule classification, accuracy verification
Rule classification, completeness check rule classification.
9. a classification based on verification rule and quality inspection system, it is characterised in that include,
Monitoring module, described monitoring module in order to the tables of data in data base is monitored,
Rule sets up module, and described rule sets up module in order to set up new school according to the dimension of described monitoring
Test rule；
Processing module, described processing module is in order to carry out important kind classification by described verification rule, and presses
According to the classification of described important kind, operation is classified；And newly-increased operation is set execution frequency；
Quality testing module, whether described quality testing module exists abnormal data in order to quality inspection, if existing, then basis
Alarm rule is fed back；
Operation unit, described operation unit is in order to carry out newly-increased job task；And perform job task；
Data base, described data base treats the data of quality inspection in order to store.
System the most according to claim 9, it is characterised in that described rule is set up module and set up
Including: promptness verification rule, accuracy verification rule and completeness check rule.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610030974.3A CN105824870A (en) | 2016-01-15 | 2016-01-15 | Classification and quality inspection method and system based on verification rules |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610030974.3A CN105824870A (en) | 2016-01-15 | 2016-01-15 | Classification and quality inspection method and system based on verification rules |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105824870A true CN105824870A (en) | 2016-08-03 |
Family
ID=56986936
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610030974.3A Pending CN105824870A (en) | 2016-01-15 | 2016-01-15 | Classification and quality inspection method and system based on verification rules |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105824870A (en) |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106886946A (en) * | 2017-02-24 | 2017-06-23 | 泰康保险集团股份有限公司 | For the method and system that declaration form surrender is calculated |
| CN107357940A (en) * | 2017-08-28 | 2017-11-17 | 中煤航测遥感集团有限公司 | A kind of method and apparatus of real estate Data Integration |
| CN107357589A (en) * | 2017-07-21 | 2017-11-17 | 深圳易嘉恩科技有限公司 | Displaying and editor's financial affair work document bill check the interface system of rule |
| CN107871283A (en) * | 2016-09-26 | 2018-04-03 | 平安科技（深圳）有限公司 | The processing method and processing device of settlement of insurance claim |
| CN109102080A (en) * | 2018-08-21 | 2018-12-28 | 广发证券股份有限公司 | A kind of numeric type finance data Quality Monitoring Control System and method |
| CN109299950A (en) * | 2017-07-25 | 2019-02-01 | 中国科学院沈阳自动化研究所 | A kind of quality inspection management method and system of flexibly configurable |
| CN109359277A (en) * | 2018-10-22 | 2019-02-19 | 众安在线财产保险股份有限公司 | Data monitoring method, equipment and computer storage medium |
| CN109783479A (en) * | 2018-12-13 | 2019-05-21 | 平安医疗健康管理股份有限公司 | A kind of data standardization processing method, device and storage medium |
| CN109947833A (en) * | 2019-02-27 | 2019-06-28 | 浪潮软件集团有限公司 | A kind of data quality management method based on B/S framework |
| CN110245049A (en) * | 2019-06-06 | 2019-09-17 | 深圳前海微众银行股份有限公司 | Monitoring method, device, equipment and the storage medium of product configuration data |
| CN110704502A (en) * | 2019-11-20 | 2020-01-17 | 中电万维信息技术有限责任公司 | Componentized data quality checking method |
| CN110837496A (en) * | 2019-11-08 | 2020-02-25 | 浪潮云信息技术有限公司 | Data quality management method and system based on dynamic sql |
| CN111782489A (en) * | 2020-07-16 | 2020-10-16 | 上海乾臻信息科技有限公司 | Data table monitoring method and device |
| CN112231310A (en) * | 2020-10-15 | 2021-01-15 | 浪潮云信息技术股份公司 | Data checking and classifying processing system and method |
| CN112818411A (en) * | 2021-01-22 | 2021-05-18 | 深圳市今日投资数据科技有限公司 | Data detection method and device |
| CN113722352A (en) * | 2021-08-31 | 2021-11-30 | 航天信息***工程(北京)有限公司 | Intelligent data verification method, system and storage medium for reporting and reviewing scheme |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101990208A (en) * | 2009-07-31 | 2011-03-23 | ***通信集团公司 | Automatic data checking method, system and equipment |
| CN102571403A (en) * | 2010-12-31 | 2012-07-11 | 北京亿阳信通软件研究院有限公司 | Realization method and device for general data quality control adapter |
| CN103473672A (en) * | 2013-09-30 | 2013-12-25 | 国家电网公司 | System, method and platform for auditing metadata quality of enterprise-level data center |
-
2016
- 2016-01-15 CN CN201610030974.3A patent/CN105824870A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101990208A (en) * | 2009-07-31 | 2011-03-23 | ***通信集团公司 | Automatic data checking method, system and equipment |
| CN102571403A (en) * | 2010-12-31 | 2012-07-11 | 北京亿阳信通软件研究院有限公司 | Realization method and device for general data quality control adapter |
| CN103473672A (en) * | 2013-09-30 | 2013-12-25 | 国家电网公司 | System, method and platform for auditing metadata quality of enterprise-level data center |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107871283A (en) * | 2016-09-26 | 2018-04-03 | 平安科技（深圳）有限公司 | The processing method and processing device of settlement of insurance claim |
| CN106886946A (en) * | 2017-02-24 | 2017-06-23 | 泰康保险集团股份有限公司 | For the method and system that declaration form surrender is calculated |
| CN107357589A (en) * | 2017-07-21 | 2017-11-17 | 深圳易嘉恩科技有限公司 | Displaying and editor's financial affair work document bill check the interface system of rule |
| CN109299950B (en) * | 2017-07-25 | 2021-10-15 | 中国科学院沈阳自动化研究所 | Flexibly configurable quality inspection management method and system |
| CN109299950A (en) * | 2017-07-25 | 2019-02-01 | 中国科学院沈阳自动化研究所 | A kind of quality inspection management method and system of flexibly configurable |
| CN107357940A (en) * | 2017-08-28 | 2017-11-17 | 中煤航测遥感集团有限公司 | A kind of method and apparatus of real estate Data Integration |
| CN109102080A (en) * | 2018-08-21 | 2018-12-28 | 广发证券股份有限公司 | A kind of numeric type finance data Quality Monitoring Control System and method |
| CN109102080B (en) * | 2018-08-21 | 2021-12-10 | 广发证券股份有限公司 | Numerical financial data quality monitoring system and method |
| CN109359277A (en) * | 2018-10-22 | 2019-02-19 | 众安在线财产保险股份有限公司 | Data monitoring method, equipment and computer storage medium |
| CN109783479A (en) * | 2018-12-13 | 2019-05-21 | 平安医疗健康管理股份有限公司 | A kind of data standardization processing method, device and storage medium |
| CN109783479B (en) * | 2018-12-13 | 2023-03-31 | 平安医疗健康管理股份有限公司 | Data standardization processing method and device and storage medium |
| CN109947833A (en) * | 2019-02-27 | 2019-06-28 | 浪潮软件集团有限公司 | A kind of data quality management method based on B/S framework |
| CN110245049A (en) * | 2019-06-06 | 2019-09-17 | 深圳前海微众银行股份有限公司 | Monitoring method, device, equipment and the storage medium of product configuration data |
| CN110837496A (en) * | 2019-11-08 | 2020-02-25 | 浪潮云信息技术有限公司 | Data quality management method and system based on dynamic sql |
| CN110704502A (en) * | 2019-11-20 | 2020-01-17 | 中电万维信息技术有限责任公司 | Componentized data quality checking method |
| CN111782489A (en) * | 2020-07-16 | 2020-10-16 | 上海乾臻信息科技有限公司 | Data table monitoring method and device |
| CN111782489B (en) * | 2020-07-16 | 2024-05-10 | 上海乾臻信息科技有限公司 | Data table monitoring method and device |
| CN112231310A (en) * | 2020-10-15 | 2021-01-15 | 浪潮云信息技术股份公司 | Data checking and classifying processing system and method |
| CN112818411A (en) * | 2021-01-22 | 2021-05-18 | 深圳市今日投资数据科技有限公司 | Data detection method and device |
| CN113722352A (en) * | 2021-08-31 | 2021-11-30 | 航天信息***工程(北京)有限公司 | Intelligent data verification method, system and storage medium for reporting and reviewing scheme |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105824870A (en) | Classification and quality inspection method and system based on verification rules | |
| Madeyski et al. | Which process metrics can significantly improve defect prediction models? An empirical study | |
| US8612479B2 (en) | Systems and methods for monitoring and detecting fraudulent uses of business applications | |
| US9026467B2 (en) | Systems and methods for monitoring and detecting fraudulent uses of business applications | |
| Wang et al. | A framework for analysis of data quality research | |
| US9031873B2 (en) | Methods and apparatus for analysing and/or pre-processing financial accounting data | |
| Caron et al. | Comprehensive rule-based compliance checking and risk management with process mining | |
| AU2012211167B2 (en) | Generating data pattern information | |
| US20080033587A1 (en) | A system and method for mining data from high-volume text streams and an associated system and method for analyzing mined data | |
| US9978031B2 (en) | Systems and methods for monitoring and detecting fraudulent uses of business applications | |
| Bigonha et al. | The usefulness of software metric thresholds for detection of bad smells and fault prediction | |
| TW202029037A (en) | Automatic monitoring method and system for financial account | |
| Zhang et al. | A data driven approach for discovering data quality requirements | |
| CN107766500A (en) | The auditing method of fixed assets card | |
| Tan et al. | Does it matter who pays back Technical Debt? An empirical study of self-fixed TD | |
| Soltanifar et al. | Software analytics in practice: A defect prediction model using code smells | |
| Kasunic et al. | An investigation of techniques for detecting data anomalies in earned value management data | |
| Rosnidah et al. | Detecting and preventing fraud with big data analytics: Auditing perspective | |
| US7992126B2 (en) | Apparatus and method for quantitatively measuring the balance within a balanced scorecard | |
| Li | Data quality and data cleaning in database applications | |
| CN116701358B (en) | Data processing method and system | |
| US20100050230A1 (en) | Method of inspecting spreadsheet files managed within a spreadsheet risk reconnaissance network | |
| CN115525575A (en) | Data automation testing method and system based on Dataworks platform | |
| CN113297146A (en) | Processing model and method for local supervision submission data | |
| Arvanitou et al. | A metric for quantifying the ripple effects among requirements |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160803 |
|
| RJ01 | Rejection of invention patent application after publication |