CN107870827A - Data quality control method and device based on verification - Google Patents
Data quality control method and device based on verification Download PDFInfo
- Publication number
- CN107870827A CN107870827A CN201711083526.0A CN201711083526A CN107870827A CN 107870827 A CN107870827 A CN 107870827A CN 201711083526 A CN201711083526 A CN 201711083526A CN 107870827 A CN107870827 A CN 107870827A
- Authority
- CN
- China
- Prior art keywords
- verification
- data
- full dose
- data record
- record
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0751—Error or fault detection not based on redundancy
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Automatic Analysis And Handling Materials Therefor (AREA)
- Stored Programmes (AREA)
Abstract
The present invention discloses a kind of data quality control method and device based on verification, and methods described includes：Obtain full dose data；Checking parameter table corresponding to the full dose data is read, passes through combination verification logic generation the first verification rule set；The full dose data are verified with the described first verification rule set, obtain the first result set；Data record corresponding to check errors information in first result set is deposited to error library；The filter operation instruction is responded, reads the verification type parameter in the data record and the checking parameter table in the error library, it is determined that corresponding to the data record of certainty verification, generates certainty error library；The full dose data are screened with the certainty error library, using the full dose data Jing Guo the screening as the data for reporting and submitting storehouse.Technical scheme provided by the invention, it can be good at being applied to the ageing system for requiring that height, data volume are big and verification rule is complicated.
Description
Technical field
The present invention relates to software technology field, more particularly to a kind of data quality control method and device based on verification.
Background technology
For traditional verification mode in the complicated verification rule of processing, the scheme of generally use is desk checking or school one by one
Test.Desk checking refers to allow the attendant for being familiar with system check rule according to data cases voluntarily amended record, that is, by manpower
Error in data investigation is carried out, the personnel requirement safeguarded to manpower is higher.Verify, refer to every verification rule is single one by one one by one
Only write-in program, it is desirable to which complete verification rule is converted into code statement and write in a program by developer, so as to realize verification rule
Then relative complex data quality control.
And requirement of the desk checking to attendant is very high, it is necessary to very familiar every verification rule, and workload is very big,
It is less efficient, it can not also ensure the accuracy of final data, usually occur needing situation about doing over again, misarrangement difficulty height.The opposing party
Face, very low using the running efficiency of system verified one by one, verification is all a tables of data read-write operation every time, and IO expenses are big,
Reaction speed is slow, can not realize the timely feedback of check results, also more difficult for developer, it is necessary to carry out a large amount of
The duplication of labour, the bit error rate is high, also influences the accuracy of final data.
Therefore, in existing technology, it is low low with the accuracy rate of data check efficiency to be all present, so as to can not be effectively to data
The problem of quality is controlled, so as to which the ageing system for requiring that height, data volume are big and verification rule is complicated can not be applied to.
The content of the invention
In view of this, the invention provides a kind of data quality control method and device based on verification, can effectively solve
Certainly efficiency present in prior art is low low with the accuracy rate of data check, so as to what effectively can not be controlled to the quality of data
Problem, it can be good at being applied to the ageing system for requiring that height, data volume are big and verification rule is complicated.
To achieve the above object, the present invention provides following technical scheme：
A kind of data quality control method based on verification, including：
Full dose data are obtained, the full dose data include data record；
Checking parameter table corresponding to the full dose data is read, passes through combination verification logic generation the first verification rule set；
The full dose data are verified with the described first verification rule set, obtain the first result set；
Data record corresponding to check errors information in first result set is deposited to error library；
Obtain filter operation instruction；
The filter operation instruction is responded, reads the school in the data record and the checking parameter table in the error library
Type parameter is tested, it is determined that corresponding to the data record of certainty verification, generates certainty error library；
The full dose data are screened with the certainty error library, using the full dose data Jing Guo the screening as reporting and submitting storehouse
Data.
Further, checking parameter table corresponding to the full dose data is read, the first school is generated by combination verification logic
Rule set is tested, including：
Read the checking mark number being located at corresponding to the full dose data in checking parameter table, check information description, verification
Object table, verification object field, the configuration of verification sentence and verification switch, pass through combination verification logic generation the first verification rule
Collection.
Further, the full dose data arrange including data item standard, described to verify rule set to described with described first
Full dose data are verified, and obtain the first result set, including：
The full dose data are verified with the described first verification rule set, obtain including data item standard row, verification
First result set of identification number and verification state；The verification state includes verification correct information and check errors information.
Further, data record corresponding to the check errors information by first result set is deposited to mistake
Storehouse, including：
By the data item standard row corresponding with the check errors information difference and check mark number, and the verification
The data record that data item standard row corresponding to error message navigate to is deposited to error library.
Further, before the acquisition filter operation instruction, in addition to：
Show the data record in error library.
Further, after the data record shown in error library, before the acquisition filter operation instruction, also wrap
Include：
Obtain the data record of user's amended record；The data record of the amended record is relative with the data record in the error library
Should；
The data record of the amended record is write into full dose data, the data record of renewal is made marks；
Pass through the checking parameter table and marked data record combination verification logic, the verification rule set of generation second；
Marked data record is verified with the described second verification rule set, obtains the second result set；
The data in the error library are deleted, by data record corresponding to the check errors information in second result set
Deposit to error library.
A kind of data quality control device based on verification, including：
First acquisition module, for obtaining full dose data, the full dose data include data record；
Read module, for reading checking parameter table corresponding to the full dose data, pass through combination verification logic generation the
One verification rule set；
First correction verification module, for being verified with the described first verification rule set to the full dose data, obtain first
Result set；
Storage module, for data record corresponding to the check errors information in first result set to be deposited to mistake
Storehouse；
Second acquisition module, for obtaining filter operation instruction；
First generation module, for responding the filter operation instruction, read the data record in the error library and institute
The verification type parameter in checking parameter table is stated, it is determined that corresponding to the data record of certainty verification, generates certainty error library；
Filtering module, for screening the full dose data with the certainty error library, by the full dose Jing Guo the screening
Data are as the data for reporting and submitting storehouse.
Further, the read module is specifically used for：
Read the checking mark number being located at corresponding to the full dose data in checking parameter table, check information description, verification
Object table, verification object field, the configuration of verification sentence and verification switch, pass through combination verification logic generation the first verification rule
Collection.
Further, the full dose data arrange including data item standard, and first correction verification module is specifically used for：
The full dose data are verified with the described first verification rule set, obtain including data item standard row, verification
First result set of identification number and verification state；The verification state includes verification correct information and check errors information.
Further, the storage module is specifically used for：
By the data item standard row corresponding with the check errors information difference and check mark number, and the verification
The data record that data item standard row corresponding to error message navigate to is deposited to error library.
Further, in addition to：
Display module, for showing the data record in error library.
Further, in addition to：
3rd acquisition module, for obtaining the data record of user's amended record；The data record of the amended record and the mistake
Data record in storehouse is corresponding；
Mark module, for the data record of the amended record to be write into full dose data, the data record of renewal is made marks；
Second generation module, for passing through the checking parameter table and marked data record combination verification logic, life
Into the second verification rule set；
Second correction verification module, for being verified with the described second verification rule set to marked data record, obtain
Second result set；
Update module, for deleting the data in the error library, by the check errors information in second result set
Corresponding data record is deposited to error library.
Understand that compared with prior art, the invention provides a kind of data based on verification via above-mentioned technical scheme
Method of quality control and device.Technical scheme provided by the invention, full dose data are obtained first, wherein, the full dose packet
Data record is included, then reads checking parameter table corresponding to the full dose data, passes through the verification of combination verification logic generation first
Rule set, disposably all full dose data are verified with the described first verification rule set, obtain the first result set, only once
I/O operation, relative to the mode verified one by one in the prior art, IO expenses can be greatly lowered, effectively improve efficiency, meanwhile,
Relative to the mode of desk checking, efficiency can be also effectively improved, then by the check errors information in first result set
Corresponding data record is deposited to error library, obtains filter operation instruction, is responded the filter operation instruction, is read the mistake
The verification type parameter in data record and the checking parameter table in storehouse, it is determined that remembering corresponding to the data of certainty verification
Record, certainty error library is generated, the full dose data are finally screened with the certainty error library, by by the complete of the screening
Data are measured as the data for reporting and submitting storehouse, can in filter false storehouse certainty mistake data record, in addition, on the one hand, due to
Technical scheme provided by the invention performs automatic Verification and not relies primarily on desk checking, can avoid relying primarily on artificial school
The problem of testing easy error, on the other hand, technical scheme provided by the invention is relative to prior art, it is not necessary to which developer enters
The substantial amounts of duplication of labour of row, can reduce the bit error rate, and so as to improve the accuracy of data check, therefore, the present invention provides
Technical scheme can effectively improve the accuracy rate of data check, so as to be effectively controlled to the quality of data.That is this hair
The technical scheme of bright offer, it can effectively solve the problem that efficiency present in prior art is low low with the accuracy rate of data check, so as to
The problem of effectively quality of data can not being controlled, it can be good at being applied to ageing requiring that high, data volume is big and school
The system for testing regular complexity.
Brief description of the drawings
In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, below will be to embodiment or existing
There is the required accompanying drawing used in technology description to be briefly described, it should be apparent that, drawings in the following description are only this
The embodiment of invention, for those of ordinary skill in the art, on the premise of not paying creative work, can also basis
The accompanying drawing of offer obtains other accompanying drawings.
Fig. 1 is a kind of flow chart of the data quality control method based on verification provided in an embodiment of the present invention；
Fig. 2 is the flow chart of another data quality control method based on verification provided in an embodiment of the present invention；
Fig. 3 is a kind of structure chart of the data quality control device based on verification provided in an embodiment of the present invention；
Fig. 4 is the structure chart of another data quality control device based on verification provided in an embodiment of the present invention.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clear, complete
Site preparation describes, it is clear that described embodiment is only part of the embodiment of the present invention, rather than whole embodiments.It is based on
Embodiment in the present invention, those of ordinary skill in the art are obtained every other under the premise of creative work is not made
Embodiment, belong to the scope of protection of the invention.
In order to facilitate the understanding of the purposes, features and advantages of the present invention, it is below in conjunction with the accompanying drawings and specific real
Applying mode, the present invention is further detailed explanation.
Embodiment
Referring to Fig. 1, Fig. 1 is a kind of flow of the data quality control method based on verification provided in an embodiment of the present invention
Figure.As shown in figure 1, this method includes：
Step S101, obtain full dose data；
Optionally, obtain and lifted automatically by batch program or initiated by attendant, be passed to verification data area in preparation
Newest full dose data.Wherein, the full dose data are used to provide data to be reported and submitted, including multiple tables of data, are wrapped in tables of data
Include data record.
Optionally, the full dose data include data item standard row (DATA_ID) and data item remaining information.Data item mark
Quasi- row include a series of fields, for positioning the entry of specific data record in different pieces of information table；Remaining packet of data item contains
Specific data record details.
Step S102, checking parameter table corresponding to the full dose data is read, the first school is generated by combination verification logic
Test rule set；
Optionally, the checking parameter table is used to deposit verification Rule Information and relevant configuration, including：
Checking mark number (CHK_ID), for positioning the identification number of every verification rule；
Check information describes, for being expressly recited specific verification rule；
Verification object table, the appropriate table for clearly current verification rule；
Verification object field, for field in the clearly current regular applicable table of verification；
Sentence configuration is verified, for check information description to be translated into code, is called during to perform checking routine；
Type is verified, for setting the severity level of verification, including " certainty mistake " and " indicative mistake "；
Verification switch, for controlling whether every verification rule comes into force；This field realizes jointly with " verification type "
Flexible configuration of the attendant to verification rule.
Optionally, the step S102 includes：
Read the checking mark number being located at corresponding to the full dose data in checking parameter table, check information description, verification
Object table, verification object field, the configuration of verification sentence and verification switch, pass through combination verification logic generation the first verification rule
Collection.
Step S103, the full dose data are verified with the described first verification rule set, obtain the first result set；
Optionally, the step S103 includes：
The full dose data are verified with the described first verification rule set, obtain including data item standard row, verification
First result set of identification number and verification state；The verification state includes verification correct information and check errors information.
Specifically, illustrating the step S102 and step S103 in order to clearer, it is exemplified below：
For example, thering are two fields a and b to need to be verified and (verification rule be present) in tables of data A, saying for rule is verified
It is bright to be shown in Table 1：
Table 1
After rule supplying checking parameter table will be verified, the extraction verification rule set in units of checking parameter table, then data
The rule set of Table A is exactly two rules, is shown in Table 2：
Table 2
After being extracted the verification rule set of checking parameter table, the content in verification rule set combination full dose data is given birth to
Into result set.
Consideration based on system high efficiency, is designed as a checking parameter table, i.e., only needs an I/O operation.Therefore, exist
Configuration parameter is made into ranks by unpivot functions during generated statement mutually to turn, verification object identical verification rule set is put into
Numerical value extraction is carried out to source table in one DML (data manipulation language, DML) sentence, realized
Disposable unified whole rules to the table are handled, that is, realize every physical table only once I/O operation, so as to effectively reduce
The IO expenses of large data volume table, effectively improve running efficiency of system.
Verification rule set is evened up first, is associated with full dose data, it is assumed that there are two records in tables of data A, it is false
The situation for determining field a and field b is as shown in table 3：
Table 3
| Data item standard arranges | Field a | Field b |
| Record 1 | 1 | 1 |
| Record 2 | 2 | 22 |
It can be seen that record 1 meets verification rule 1, but verification rule 2 is not met；2 two verification rules are recorded to meet.Will
It verifies rule set and evens up rear form such as table 4：
Table 4
| Data item standard arranges | RULE-01 | RULE-02 |
| Record 1 | Correctly | Mistake |
| Record 2 | Correctly | Correctly |
Because after verification rule set is arranged in into the data record of full dose data in the form of fields, in this process
Full dose data, which are only needed once to read, to be completed, and effectively increase operational efficiency.Then by the data record evened up to verify
Identification number is that dimension carries out transposition, is formed result set, and result set can be with " data item standard row " " checking mark number " " verification
The form displaying of three abstract dimensions of state ".Take record 1 and record 2, example such as table 5：
Table 5
| Data item standard arranges | Checking mark number | Verification state |
| Record 1 | RULE-01 | Correctly |
| Record 1 | RULE-02 | Mistake |
| Record 2 | RULE-01 | Correctly |
| Record 2 | RULE-02 | Correctly |
Result set can intuitively embody the result correctness of the corresponding every verification of record per data in table.
Step S104, data record corresponding to the check errors information in first result set is deposited to error library；
Optionally, the step S104 includes：
By the data item standard row corresponding with the check errors information difference and check mark number, and the verification
The data record that data item standard row corresponding to error message navigate to is deposited to error library.
Specifically, still continuing to use above-mentioned example, then will wherein verify state extracts for the data record of mistake, is just formed
Error library.Final error library only preserves the data record of one [record 1- verifications rule 2] in above-mentioned example.
Step S105, obtain filter operation instruction；
Specifically, obtain the filter operation instruction of attendant's input.
Step S106, the filter operation instruction is responded, read the data record in the error library and the verification ginseng
Verification type parameter in number table, it is determined that corresponding to the data record of certainty verification, generate certainty error library；
Specifically, the verification type parameter in the data record and the checking parameter table in the error library is read, really
The fixed data record corresponding to " certainty mistake ", extracts the data record corresponding to " certainty mistake ", and generation certainty is wrong
Storehouse by mistake.
Step S107, the full dose data are screened with the certainty error library, by the full dose data Jing Guo the screening
As the data for reporting and submitting storehouse；
Specifically, because verification rule seriousness it is generally different, be divided into seriousness higher " certainty verification " and
Seriousness is relatively low, only needs to make " the indicative verification " of prompting, that is, the data record for certainty verification class mistake occur can not be reported
Send, the data record for indicative verification class mistake only occur can report and submit.Therefore in screening process, system can be by checking parameter
Table combination error library, if any data record one or more than one certainty check errors be present if need it is to be filtered, if
Data record is correct or only exists indicative check errors then by filtering, and the data record by filtering is placed on and reports and submits storehouse
In in case report and submit at any time, reach the target of data filtering.
Technical scheme provided in an embodiment of the present invention, full dose data are obtained first, wherein, the full dose data include data
Record, then reads checking parameter table corresponding to the full dose data, and rule set is verified by combination verification logic generation first,
Disposably all full dose data are verified with the described first verification rule set, obtain the first result set, only once I/O operation,
Relative to the mode verified one by one in the prior art, IO expenses can be greatly lowered, effectively improve efficiency, meanwhile, relative to
The mode of desk checking, efficiency can be also effectively improved, then by corresponding to the check errors information in first result set
Data record is deposited to error library, obtains filter operation instruction, is responded the filter operation instruction, is read in the error library
Verification type parameter in data record and the checking parameter table, it is determined that corresponding to the data record of certainty verification, generation
Certainty error library, the full dose data are finally screened with the certainty error library, by the full dose data Jing Guo the screening
As the data for reporting and submitting storehouse, can in filter false storehouse certainty mistake data record, in addition, on the one hand, due to the present invention
The technical scheme that embodiment provides performs automatic Verification and not relies primarily on desk checking, can avoid relying primarily on artificial school
The problem of testing easy error, on the other hand, technical scheme provided in an embodiment of the present invention is relative to prior art, it is not necessary to develops
Personnel carry out the substantial amounts of duplication of labour, the bit error rate can be reduced, so as to improve the accuracy of data check, therefore, this hair
The technical scheme that bright embodiment provides can effectively improve the accuracy rate of data check, so as to effectively be carried out to the quality of data
Control.Technical scheme i.e. provided in an embodiment of the present invention, it can effectively solve the problem that efficiency present in prior art is low and data school
The accuracy rate tested is low, so as to the problem of effectively can not being controlled the quality of data, can be good at being applied to ageing requirement
System high, data volume is big and verification rule is complicated.
In addition, technical scheme provided in an embodiment of the present invention, verifies rule set, matching verifies object and implements school in generation
During testing, the operation such as extraction, matching is by the way of parameter configuration.On the one hand, matched somebody with somebody by the form of checking parameter table
Verification rule is put can timely and conveniently to make the renewal of simplicity in verification rule change, can effectively save subsequent development
Cost；On the other hand, it can also realize that attendant easily carries out autonomous control to verification rule, so as to agree with practical business
Development condition adjusts, and meets some individual demands.
Optionally, the data quality control method based on verification that another embodiment of the present invention provides, the step
Before S105, in addition to：
Show the data record in error library.
Specifically, the data record in error library is shown into attendant, the personnel that can maintain easily check error library
In data record, so as to decide whether initiate filter operation, be determined whether input filter operation instruction.
Optionally, if the data record in attendant's decision error storehouse is more, filter operation can't temporarily be initiated
When, it is necessary to corresponding to the data record in error library, the new data record of correspondence position amended record in full dose data, and to it
Verification is re-started, the embodiment of the present invention is defined as increment verification.And it is base to define the step S101~step S104
Plinth verifies.
Referring to Fig. 2, Fig. 2 is another data quality control method based on verification provided in an embodiment of the present invention
Flow chart.Another data quality control method based on verification provided in an embodiment of the present invention verifies comprising increment, such as Fig. 2
Shown, this method includes：
Step S201, obtain full dose data；
Optionally, obtain and lifted automatically by batch program or initiated by attendant, be passed to verification data area in preparation
Newest full dose data.Wherein, the full dose data are used to provide data to be reported and submitted, including multiple tables of data, are wrapped in tables of data
Include data record.
Optionally, the full dose data include data item standard row (DATA_ID) and data item remaining information.Data item mark
Quasi- row include a series of fields, for positioning the entry of specific data record in different pieces of information table；Remaining packet of data item contains
Specific data record details.
Step S202, checking parameter table corresponding to the full dose data is read, the first school is generated by combination verification logic
Test rule set；
Optionally, the checking parameter table is used to deposit verification Rule Information and relevant configuration, including：
Checking mark number (CHK_ID), for positioning the identification number of every verification rule；
Check information describes, for being expressly recited specific verification rule；
Verification object table, the appropriate table for clearly current verification rule；
Verification object field, for field in the clearly current regular applicable table of verification；
Sentence configuration is verified, for check information description to be translated into code, is called during to perform checking routine；
Type is verified, for setting the severity level of verification, including " certainty mistake " and " indicative mistake "；
Verification switch, for controlling whether every verification rule comes into force；This field realizes jointly with " verification type "
Flexible configuration of the attendant to verification rule.
Optionally, the step S202 includes：
Read the checking mark number being located at corresponding to the full dose data in checking parameter table, check information description, verification
Object table, verification object field, the configuration of verification sentence and verification switch, pass through combination verification logic generation the first verification rule
Collection.
Step S203, the full dose data are verified with the described first verification rule set, obtain the first result set；
Optionally, the step S203 includes：
The full dose data are verified with the described first verification rule set, obtain including data item standard row, verification
First result set of identification number and verification state；The verification state includes verification correct information and check errors information.
Step S204, data record corresponding to the check errors information in first result set is deposited to error library；
Optionally, the step S204 includes：
By the data item standard row corresponding with the check errors information difference and check mark number, and the verification
The data record that data item standard row corresponding to error message navigate to is deposited to error library.
Step S205, show the data record in error library；
Step S206, obtain the data record of user's amended record；
Specifically, the data record of the amended record is corresponding with the data record in the error library.Such as in error library
Data record be A tables in record 1, then obtain user's amended record A tables in record 1.
Step S207, the data record of the amended record is write into full dose data, the data record of renewal is made marks；
Specifically, the data record of the amended record is write into full dose data, as the full dose data after renewal.To renewal
Data record makes marks, and can facilitate the data record by marking fast positioning to update.
Step S208, by the checking parameter table and marked data record combination verification logic, generate the second school
Test rule set；
Step S209, marked data record is verified with the described second verification rule set, obtains the second result
Collection；
Step S210, the data in the error library are deleted, the check errors information in second result set is corresponding
Data record deposit to error library；
It should be noted that the concrete methods of realizing of the step S208~step S210 and the step S102~step
S104 concrete methods of realizing is identical, and difference is：On the one hand, it is necessary to marked in the step S208~step S210
Data record substitutes full dose data and carries out subsequent check, and verifying work amount is much smaller, and efficiency is higher；On the other hand, it is necessary to delete
Except the data in former error library, data record corresponding to the check errors information in second result set is deposited to mistake
Storehouse.As for other specific implementation parts of the step S208~step S210, the step S102~step S104 can refer to
Specific implementation part, the present embodiment repeats no more.
Step S211, obtain filter operation instruction；
Specifically, obtain the filter operation instruction of attendant's input.
Step S212, the filter operation instruction is responded, read the data record in the error library and the verification ginseng
Verification type parameter in number table, it is determined that corresponding to the data record of certainty verification, generate certainty error library；
Specifically, the verification type parameter in the data record and the checking parameter table in the error library is read, really
The fixed data record corresponding to " certainty mistake ", extracts the data record corresponding to " certainty mistake ", and generation certainty is wrong
Storehouse by mistake.
Step S213, the full dose data are screened with the certainty error library, by the full dose data Jing Guo the screening
As the data for reporting and submitting storehouse；
Specifically, because verification rule seriousness it is generally different, be divided into seriousness higher " certainty verification " and
Seriousness is relatively low, only needs to make " the indicative verification " of prompting, that is, the data record for certainty verification class mistake occur can not be reported
Send, the data record for indicative verification class mistake only occur can report and submit.Therefore in screening process, system can be by checking parameter
Table combination error library, if any data record one or more than one certainty check errors be present if need it is to be filtered, if
Data record is correct or only exists indicative check errors then by filtering, and the data record by filtering is placed on and reports and submits storehouse
In in case report and submit at any time, reach the target of data filtering.
Technical scheme provided in an embodiment of the present invention, the mode of increment verification is with the addition of, maintain easily personnel's amended record verification
State is wrong data record, and marks the data record of amended record, and labeled data record is only verified in subsequent check,
Full dose data need not be re-started with verification, the data volume of verification is smaller, so as to which operational efficiency is higher.
In order to illustrate technical scheme provided by the invention more fully hereinafter, school is based on corresponding to provided in an embodiment of the present invention
The data quality control method tested, the present invention disclose a kind of data quality control device based on verification.
Referring to Fig. 3, Fig. 3 is a kind of structure of the data quality control device based on verification provided in an embodiment of the present invention
Figure.As shown in figure 3, the device includes：
First acquisition module 301, for obtaining full dose data, the full dose data include data record；
Read module 302, for checking parameter table corresponding to reading the full dose data, generated by combination verification logic
First verification rule set；
Optionally, the read module 302 is specifically used for：
Read the checking mark number being located at corresponding to the full dose data in checking parameter table, check information description, verification
Object table, verification object field, the configuration of verification sentence and verification switch, pass through combination verification logic generation the first verification rule
Collection.
First correction verification module 303, for being verified with the described first verification rule set to the full dose data, obtain the
One result set；
Optionally, the full dose data arrange including data item standard, and first correction verification module 303 is specifically used for：
The full dose data are verified with the described first verification rule set, obtain including data item standard row, verification
First result set of identification number and verification state；The verification state includes verification correct information and check errors information.
Storage module 304, for by data record corresponding to the check errors information in first result set deposit to
Error library；
Optionally, the storage module 304 is specifically used for：
By the data item standard row corresponding with the check errors information difference and check mark number, and the verification
The data record that data item standard row corresponding to error message navigate to is deposited to error library.
Second acquisition module 305, for obtaining filter operation instruction；
First generation module 306, for responding filter operation instruction, read data record in the error library and
Verification type parameter in the checking parameter table, it is determined that corresponding to the data record of certainty verification, generate certainty mistake
Storehouse；
Filtering module 307, for screening the full dose data with the certainty error library, by by the complete of the screening
Data are measured as the data for reporting and submitting storehouse.
Using the data quality control device provided in an embodiment of the present invention based on verification, data check can be effectively improved
Accuracy rate, so as to be effectively controlled to the quality of data.Data matter i.e. provided in an embodiment of the present invention based on verification
Amount control device, it can effectively solve the problem that efficiency present in prior art is low low with the accuracy rate of data check, so as to have
The problem of effect is controlled the quality of data, it can be good at requiring that height, data volume are big and verification is regular suitable for ageing
Complicated system.
In addition, the data quality control device provided in an embodiment of the present invention based on verification, rule set is verified in generation,
With verification object and implement during verifying, the operation such as extraction, matching is by the way of parameter configuration.On the one hand, pass through
The form configuration verification rule of checking parameter table can timely and conveniently make the renewal of simplicity, energy in verification rule change
It is enough effectively to save subsequent development cost；On the other hand, it can also realize that attendant is easily carried out from master control to verification rule
System, adjusts so as to agree with practical business development condition, meets some individual demands.
Optionally, the data quality control device based on verification that another embodiment of the present invention provides, in addition to：
Display module, for showing the data record in error library.
Optionally, if the data record in attendant's decision error storehouse is more, filter operation can't temporarily be initiated
When, it is necessary to corresponding to the data record in error library, the new data record of correspondence position amended record in full dose data, and to it
Verification is re-started, the embodiment of the present invention is defined as increment verification.And define first acquisition module, read module,
Verified based on the part of first correction verification module and storage module.
Referring to Fig. 4, Fig. 4 is another data quality control device based on verification provided in an embodiment of the present invention
Flow chart.Another data quality control device based on verification provided in an embodiment of the present invention verifies comprising increment, such as Fig. 4
Shown, the device includes：
First acquisition module 401, for obtaining full dose data, the full dose data include data record；
Read module 402, for checking parameter table corresponding to reading the full dose data, generated by combination verification logic
First verification rule set；
First correction verification module 403, for being verified with the described first verification rule set to the full dose data, obtain the
One result set；
Storage module 404, for by data record corresponding to the check errors information in first result set deposit to
Error library；
Display module 405, for showing the data record in error library；
3rd acquisition module 406, for obtaining the data record of user's amended record；The data record of the amended record and the mistake
Data record by mistake in storehouse is corresponding；
Mark module 407, for the data record of the amended record to be write into full dose data, the data record of renewal is marked
Note；
Second generation module 408, for by the checking parameter table and marked data record combination verification logic,
The verification rule set of generation second；
Second correction verification module 409, for being verified with the described second verification rule set to marked data record, obtain
To the second result set；
Update module 410, for deleting the data in the error library, the check errors in second result set are believed
Data record corresponding to breath is deposited to error library；
Second acquisition module 411, for obtaining filter operation instruction；
First generation module 412, for responding filter operation instruction, read data record in the error library and
Verification type parameter in the checking parameter table, it is determined that corresponding to the data record of certainty verification, generate certainty mistake
Storehouse；
Filtering module 413, for screening the full dose data with the certainty error library, by by the complete of the screening
Data are measured as the data for reporting and submitting storehouse.
Data quality control device provided in an embodiment of the present invention based on verification, it with the addition of the mode of increment verification, side
Just attendant's amended record verification state is the data record of mistake, and marks the data record of amended record, the only school in subsequent check
The data record that standard inspection is recorded a demerit, it is not necessary to full dose data are re-started with verification, the data volume of verification is smaller, so as to operational efficiency compared with
It is high.
Understand that compared with prior art, the invention provides a kind of data based on verification via above-mentioned technical scheme
Method of quality control and device.Technical scheme provided by the invention, full dose data are obtained first, wherein, the full dose packet
Data record is included, then reads checking parameter table corresponding to the full dose data, passes through the verification of combination verification logic generation first
Rule set, disposably all full dose data are verified with the described first verification rule set, obtain the first result set, only once
I/O operation, relative to the mode verified one by one in the prior art, IO expenses can be greatly lowered, effectively improve efficiency, meanwhile,
Relative to the mode of desk checking, efficiency can be also effectively improved, then by the check errors information in first result set
Corresponding data record is deposited to error library, obtains filter operation instruction, is responded the filter operation instruction, is read the mistake
The verification type parameter in data record and the checking parameter table in storehouse, it is determined that remembering corresponding to the data of certainty verification
Record, certainty error library is generated, the full dose data are finally screened with the certainty error library, by by the complete of the screening
Data are measured as the data for reporting and submitting storehouse, can in filter false storehouse certainty mistake data record, in addition, on the one hand, due to
Technical scheme provided by the invention performs automatic Verification and not relies primarily on desk checking, can avoid relying primarily on artificial school
The problem of testing easy error, on the other hand, technical scheme provided by the invention is relative to prior art, it is not necessary to which developer enters
The substantial amounts of duplication of labour of row, can reduce the bit error rate, and so as to improve the accuracy of data check, therefore, the present invention provides
Technical scheme can effectively improve the accuracy rate of data check, so as to be effectively controlled to the quality of data.That is this hair
The technical scheme of bright offer, it can effectively solve the problem that efficiency present in prior art is low low with the accuracy rate of data check, so as to
The problem of effectively quality of data can not being controlled, it can be good at being applied to ageing requiring that high, data volume is big and school
The system for testing regular complexity.
In addition, technical scheme provided in an embodiment of the present invention, verifies rule set, matching verifies object and implements school in generation
During testing, the operation such as extraction, matching is by the way of parameter configuration.On the one hand, matched somebody with somebody by the form of checking parameter table
Verification rule is put can timely and conveniently to make the renewal of simplicity in verification rule change, can effectively save subsequent development
Cost；On the other hand, it can also realize that attendant easily carries out autonomous control to verification rule, so as to agree with practical business
Development condition adjusts, and meets some individual demands.
In addition, another technical scheme provided in an embodiment of the present invention, also added the mode of increment verification, convenient dimension
Shield personnel amended record verification state is wrong data record, and marks the data record of amended record, and mark is only verified in subsequent check
The data record recorded a demerit, it is not necessary to full dose data are re-started with verification, the data volume of verification is smaller, so as to which operational efficiency is higher.
Finally, it is to be noted that, herein, such as first and second or the like relational terms be used merely to by
One entity or operation make a distinction with another entity or operation, and not necessarily require or imply these entities or operation
Between any this actual relation or order be present.Moreover, term " comprising ", "comprising" or its any other variant meaning
Covering including for nonexcludability, so that process, method, article or equipment including a series of elements not only include that
A little key elements, but also the other element including being not expressly set out, or also include for this process, method, article or
The intrinsic key element of equipment.In the absence of more restrictions, the key element limited by sentence "including a ...", is not arranged
Except other identical element in the process including the key element, method, article or equipment being also present.
Each embodiment is described by the way of progressive in this specification, what each embodiment stressed be and other
The difference of embodiment, between each embodiment identical similar portion mutually referring to.For device disclosed in embodiment
For, because it is corresponded to the method disclosed in Example, so description is fairly simple, related part is said referring to method part
It is bright.
Directly it can be held with reference to the step of method or algorithm that the embodiments described herein describes with hardware, processor
Capable software module, or the two combination are implemented.Software module can be placed in random access memory (RAM), internal memory, read-only deposit
Any other shape well known in reservoir (ROM), electrically programmable ROM, electrically erasable ROM, register or technical field
In the storage medium of formula.
The foregoing description of the disclosed embodiments, professional and technical personnel in the field are enable to realize or using the present invention.
A variety of modifications to these embodiments will be apparent for those skilled in the art, as defined herein
General Principle can be realized in other embodiments without departing from the spirit or scope of the present invention.Therefore, it is of the invention
The embodiments shown herein is not intended to be limited to, and is to fit to and principles disclosed herein and features of novelty phase one
The most wide scope caused.
Claims (12)
- A kind of 1. data quality control method based on verification, it is characterised in that including：Full dose data are obtained, the full dose data include data record；Checking parameter table corresponding to the full dose data is read, passes through combination verification logic generation the first verification rule set；The full dose data are verified with the described first verification rule set, obtain the first result set；Data record corresponding to check errors information in first result set is deposited to error library；Obtain filter operation instruction；The filter operation instruction is responded, reads the verification class in the data record and the checking parameter table in the error library Shape parameter, it is determined that corresponding to the data record of certainty verification, generate certainty error library；The full dose data are screened with the certainty error library, using the full dose data Jing Guo the screening as the number for reporting and submitting storehouse According to.
- 2. according to the method for claim 1, it is characterised in that read checking parameter table corresponding to the full dose data, lead to Combination verification logic generation the first verification rule set is crossed, including：Read the checking mark number being located at corresponding to the full dose data in checking parameter table, check information description, verification object Table, verification object field, the configuration of verification sentence and verification switch, pass through combination verification logic generation the first verification rule set.
- 3. according to the method for claim 1, it is characterised in that the full dose data include data item standard row, it is described with The first verification rule set verifies to the full dose data, obtains the first result set, including：The full dose data are verified with the described first verification rule set, obtain including data item standard row, checking mark Number and verification state the first result set；The verification state includes verification correct information and check errors information.
- 4. according to the method for claim 3, it is characterised in that the check errors information by first result set Corresponding data record is deposited to error library, including：By the data item standard row corresponding with the check errors information difference and check mark number, and the check errors The data record that data item standard row corresponding to information navigate to is deposited to error library.
- 5. according to the method described in any one of Claims 1 to 4, it is characterised in that before the acquisition filter operation instruction, also Including：Show the data record in error library.
- 6. according to the method for claim 5, it is characterised in that it is described display error library in data record after, it is described Before obtaining filter operation instruction, in addition to：Obtain the data record of user's amended record；The data record of the amended record is corresponding with the data record in the error library；The data record of the amended record is write into full dose data, the data record of renewal is made marks；Pass through the checking parameter table and marked data record combination verification logic, the verification rule set of generation second；Marked data record is verified with the described second verification rule set, obtains the second result set；The data in the error library are deleted, data record corresponding to the check errors information in second result set is deposited To error library.
- A kind of 7. data quality control device based on verification, it is characterised in that including：First acquisition module, for obtaining full dose data, the full dose data include data record；Read module, for reading checking parameter table corresponding to the full dose data, the first school is generated by combination verification logic Test rule set；First correction verification module, for being verified with the described first verification rule set to the full dose data, obtain the first result Collection；Storage module, for data record corresponding to the check errors information in first result set to be deposited to error library；Second acquisition module, for obtaining filter operation instruction；First generation module, for responding the filter operation instruction, read the data record in the error library and the school The verification type parameter tested in parameter list, it is determined that corresponding to the data record of certainty verification, generate certainty error library；Filtering module, for screening the full dose data with the certainty error library, by the full dose data Jing Guo the screening As the data for reporting and submitting storehouse.
- 8. device according to claim 7, it is characterised in that the read module is specifically used for：Read the checking mark number being located at corresponding to the full dose data in checking parameter table, check information description, verification object Table, verification object field, the configuration of verification sentence and verification switch, pass through combination verification logic generation the first verification rule set.
- 9. device according to claim 7, it is characterised in that the full dose data include data item standard row, and described the One correction verification module is specifically used for：The full dose data are verified with the described first verification rule set, obtain including data item standard row, checking mark Number and verification state the first result set；The verification state includes verification correct information and check errors information.
- 10. device according to claim 9, it is characterised in that the storage module is specifically used for：By the data item standard row corresponding with the check errors information difference and check mark number, and the check errors The data record that data item standard row corresponding to information navigate to is deposited to error library.
- 11. according to the device described in any one of claim 7~10, it is characterised in that also include：Display module, for showing the data record in error library.
- 12. device according to claim 11, it is characterised in that also include：3rd acquisition module, for obtaining the data record of user's amended record；In the data record of the amended record and the error library Data record it is corresponding；Mark module, for the data record of the amended record to be write into full dose data, the data record of renewal is made marks；Second generation module, for by the checking parameter table and marked data record combination verification logic, generation the Two verification rule sets；Second correction verification module, for being verified with the described second verification rule set to marked data record, obtain second Result set；Update module, it is for deleting the data in the error library, the check errors information in second result set is corresponding Data record deposit to error library.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711083526.0A CN107870827B (en) | 2017-11-07 | 2017-11-07 | Data quality control method and device based on verification |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711083526.0A CN107870827B (en) | 2017-11-07 | 2017-11-07 | Data quality control method and device based on verification |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107870827A true CN107870827A (en) | 2018-04-03 |
| CN107870827B CN107870827B (en) | 2021-01-08 |
Family
ID=61753667
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711083526.0A Active CN107870827B (en) | 2017-11-07 | 2017-11-07 | Data quality control method and device based on verification |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107870827B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108694522A (en) * | 2018-07-06 | 2018-10-23 | 中国银行股份有限公司 | A kind of data analysing method and device |
| CN109359277A (en) * | 2018-10-22 | 2019-02-19 | 众安在线财产保险股份有限公司 | Data monitoring method, equipment and computer storage medium |
| CN110427277A (en) * | 2019-07-30 | 2019-11-08 | 腾讯科技（深圳）有限公司 | Data verification method, device, equipment and storage medium |
| CN112148721A (en) * | 2020-09-25 | 2020-12-29 | 新华三大数据技术有限公司 | Data checking method and device, electronic equipment and storage medium |
| CN112948429A (en) * | 2021-02-02 | 2021-06-11 | 中国工商银行股份有限公司 | Data reporting method, device and equipment |
| CN113076285A (en) * | 2021-03-05 | 2021-07-06 | 上海明略人工智能(集团)有限公司 | Processing method and system for uploading data files |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105183609A (en) * | 2015-09-16 | 2015-12-23 | 焦点科技股份有限公司 | Real-time monitoring system and method applied to software system |
| CN105740121A (en) * | 2016-01-26 | 2016-07-06 | 中国银行股份有限公司 | Log text monitoring and early-warning method and apparatus |
| CN106254045A (en) * | 2016-08-09 | 2016-12-21 | 中国银行股份有限公司 | A kind of data verification method and device |
| CN106407028A (en) * | 2016-08-26 | 2017-02-15 | 佛山智能装备技术研究院 | Double-buffering type robot software log storage method |
-
2017
- 2017-11-07 CN CN201711083526.0A patent/CN107870827B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105183609A (en) * | 2015-09-16 | 2015-12-23 | 焦点科技股份有限公司 | Real-time monitoring system and method applied to software system |
| CN105740121A (en) * | 2016-01-26 | 2016-07-06 | 中国银行股份有限公司 | Log text monitoring and early-warning method and apparatus |
| CN106254045A (en) * | 2016-08-09 | 2016-12-21 | 中国银行股份有限公司 | A kind of data verification method and device |
| CN106407028A (en) * | 2016-08-26 | 2017-02-15 | 佛山智能装备技术研究院 | Double-buffering type robot software log storage method |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108694522A (en) * | 2018-07-06 | 2018-10-23 | 中国银行股份有限公司 | A kind of data analysing method and device |
| CN108694522B (en) * | 2018-07-06 | 2023-05-09 | 中国银行股份有限公司 | Data analysis method and device |
| CN109359277A (en) * | 2018-10-22 | 2019-02-19 | 众安在线财产保险股份有限公司 | Data monitoring method, equipment and computer storage medium |
| CN110427277A (en) * | 2019-07-30 | 2019-11-08 | 腾讯科技（深圳）有限公司 | Data verification method, device, equipment and storage medium |
| CN112148721A (en) * | 2020-09-25 | 2020-12-29 | 新华三大数据技术有限公司 | Data checking method and device, electronic equipment and storage medium |
| CN112148721B (en) * | 2020-09-25 | 2022-08-19 | 新华三大数据技术有限公司 | Data checking method and device, electronic equipment and storage medium |
| CN112948429A (en) * | 2021-02-02 | 2021-06-11 | 中国工商银行股份有限公司 | Data reporting method, device and equipment |
| CN112948429B (en) * | 2021-02-02 | 2024-04-26 | 中国工商银行股份有限公司 | Data reporting method, device and equipment |
| CN113076285A (en) * | 2021-03-05 | 2021-07-06 | 上海明略人工智能(集团)有限公司 | Processing method and system for uploading data files |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107870827B (en) | 2021-01-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107870827A (en) | Data quality control method and device based on verification | |
| CN106846203B (en) | Intelligent marking method and system | |
| CN110008254B (en) | Transformer equipment standing book checking processing method | |
| CN106503222A (en) | Batch based on Excel imports the method and device of management data base | |
| CN104182484B (en) | A kind of method and apparatus for realizing HBase data and the mapping of Java field object | |
| CN103942143B (en) | A kind of PC Lint code check result screening techniques compared based on version repository | |
| CN108255702A (en) | A kind of test case creation method, apparatus, equipment and storage medium | |
| CN106469215B (en) | Data lead-in method and system based on page end | |
| CN109669917B (en) | Waveform storage method based on priority grouping | |
| CN105956410B (en) | A kind of Universal-purpose quick detection method of IEC61850 full models | |
| CN102087657B (en) | IO (Input/output) to point method and IO (Input/output) to point for nuclear power plant simulation platform | |
| CN108763185A (en) | The method of calibration and system of Excel file | |
| CN109359277A (en) | Data monitoring method, equipment and computer storage medium | |
| CN111309581B (en) | Application performance detection method and device in database upgrading scene | |
| CN110209585A (en) | A kind of software test case intelligent training method, terminal and storage medium | |
| CN102945262B (en) | A kind of RNC configuration data control methods and device | |
| CN109597763B (en) | Consistency verification method and device for normalizing multiple items of data | |
| CN110211581A (en) | A kind of laboratory automatic speech recognition record identification system and method | |
| CN105183641B (en) | The data consistency verification method and system of a kind of kernel module | |
| CN106294128A (en) | A kind of automated testing method deriving report data and device | |
| CN110377566A (en) | Arrange the method and system of the S parameter test file of PCB | |
| CN107783890A (en) | Software defect data processing method and device | |
| CN108427675A (en) | Build the method and apparatus of index | |
| CN110071844A (en) | A kind of detection script creation system, method and relevant apparatus | |
| CN106649501B (en) | Consistency checking system and method for access model of substation monitoring database and SCD |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |