CN109918271A - Data quality monitoring method, system and storage medium - Google Patents
Data quality monitoring method, system and storage medium Download PDFInfo
- Publication number
- CN109918271A CN109918271A CN201910246182.3A CN201910246182A CN109918271A CN 109918271 A CN109918271 A CN 109918271A CN 201910246182 A CN201910246182 A CN 201910246182A CN 109918271 A CN109918271 A CN 109918271A
- Authority
- CN
- China
- Prior art keywords
- data
- monitoring
- metadata
- quality
- data quality
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 80
- 238000000034 method Methods 0.000 title claims abstract description 43
- 230000002159 abnormal effect Effects 0.000 claims abstract description 25
- 238000001514 detection method Methods 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 6
- 238000003908 quality control method Methods 0.000 claims 1
- 230000006870 function Effects 0.000 description 6
- 230000005540 biological transmission Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000012217 deletion Methods 0.000 description 3
- 230000037430 deletion Effects 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 230000007257 malfunction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012937 correction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 230000001502 supplementing effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
This application involves data information correlative technology fields, and in particular to a kind of data quality monitoring method, system and storage medium.Wherein data quality monitoring method provided by the present application includes: monitoring metadata with the presence or absence of being modified, and modifies, alerts if it exists；It whether complete monitors metadata, is alerted if imperfect；Stored data are monitored, if it is abnormal to find that stored data exist, analyze abnormal cause, and alert；Result based on monitoring generates Data Quality Analysis report.It is modified in this way, metadata can be monitored and whether there is；Whether complete monitor metadata；It monitors stored data and there is exception, and alert in time；Result based on monitoring generates Data Quality Analysis report.The monitoring for the quality of data is so completed, is alerted in time when quality of data difference, is reminded the owner of the data quality of data situation, the owner of data is avoided to make inappropriate decision according to the data.So complete the monitoring for the quality of data.
Description
Technical field
This application involves data information correlative technology fields, and in particular to a kind of data quality monitoring method, system and deposits
Storage media.
Background technique
With the development of science and technology, the transmitting and acquisition of data are more and more convenient.Enterprises many at present can all be based on
Mass data carries out decision.When so the quality of data is poor, enterprise can be caused largely to lose.Wherein quality of data difference packet
Include: data are not complete, and data are there are mistake, data the problems such as there are undefined enumerated values.
The method of current monitoring data quality is mainly that the circulation transmission process of monitoring data avoids data from being modified.
But this mode may only guarantee that data will not malfunction in transmission process, can not monitoring data matter itself
Amount.
Summary of the invention
The application provides a kind of data quality monitoring method, system and storage medium, to solve currently without monitoring data
Quality measure itself the problem of.
The application provides a kind of data quality monitoring method, system and storage medium, comprising:
Monitoring metadata was modified if it exists, is alerted with the presence or absence of the case where being modified；
It whether complete monitors metadata, is alerted if imperfect；
Stored data are monitored, if it is abnormal to find that stored data exist, analyze abnormal cause, and accuse
It is alert；
Result based on the monitoring generates Data Quality Analysis report.
Optionally, monitoring metadata, which whether there is, is modified, comprising:
Whether detection metadata is deleted by physics.
Optionally, whether the monitoring metadata is complete, comprising:
Whether detection metadata, which has, increases undefined enumerated value newly, and then metadata is imperfect if it exists；
Whether detection enumerated value has remarks, and metadata is imperfect if without remarks；
Whether detection enumerated value has index, and metadata is imperfect if not indexing.
It is optionally, described that stored data are monitored, comprising:
Stored Data Data such as total amount of data ring is monitored than difference, the monitoring of field repetition values, field exceptional value prison
Control, index operation ring compare abnormal monitoring.
Optionally, the result based on the monitoring generates Data Quality Analysis report, comprising:
Summarize the result of the detection to generate Data Quality Analysis report.
Optionally, the mode of the alarm includes:
Short message is sent to reserved number to alert.
Optionally, the mode of the alarm includes:
Mailbox is sent to reserved mailbox to alert.
Optionally, the mode of the alarm includes:
It issues prompt tone or lights indicator light to alert.
Optionally, comprising: processor and memory；
The processor and memory communicate to connect:
Wherein, the processor, for calling and executing the program stored in the memory；
The memory, for storing program, it is described in any item that described program is at least used for perform claim requirement 1~8
Each step of screw thread method for maintaining.
The application also provides a kind of storage medium, is stored thereon with computer program, the computer program is by processor
When execution, each step of data quality monitoring method described herein is realized.
In the data quality monitoring method of the offer of the application, can monitor metadata whether there is the case where being modified,
It modifies, alerts if it exists；It whether complete monitors metadata, is alerted if imperfect；Stored data are monitored, if
It was found that stored data have abnormal, analysis abnormal cause, and alert；Result based on monitoring generates Data Quality Analysis report
It accuses.The monitoring for the quality of data is so completed, and when quality of data difference (such as metadata is imperfect, metadata is modified,
There is situations such as abnormal in the data of storage) alarm in time, the owner of the data quality of data situation is reminded, gathering around for data is avoided
The person of having makes inappropriate decision according to the data.So complete the monitoring for the quality of data.
It should be understood that above general description and following detailed description be only it is exemplary and explanatory, not
The application can be limited.
Detailed description of the invention
The drawings herein are incorporated into the specification and forms part of this specification, and shows the implementation for meeting the application
Example, and together with specification it is used to explain the principle of the application.
Fig. 1 is that the embodiment of the present application one provides a kind of flow diagram of data quality monitoring method；
Fig. 2 is that the embodiment of the present application two provides a kind of structural schematic diagram of data quality monitoring system.
Specific embodiment
Example embodiments are described in detail here, and the example is illustrated in the accompanying drawings.Following description is related to
When attached drawing, unless otherwise indicated, the same numbers in different drawings indicate the same or similar elements.Following exemplary embodiment
Described in embodiment do not represent all embodiments consistent with the application.On the contrary, they be only with it is such as appended
The example of the consistent device and method of some aspects be described in detail in claims, the application.
With the development of science and technology, the transmitting and acquisition of data are more and more convenient.Enterprises many at present can all be based on
Mass data carries out decision.When so the quality of data is poor, enterprise can be caused largely to lose.Wherein quality of data difference packet
Include: data are not complete, and data are there are mistake, data the problems such as there are undefined enumerated values.
The method of current monitoring data quality is mainly that the circulation transmission process of monitoring data avoids data from being modified.
But this mode may only guarantee that data will not malfunction in transmission process, can not monitoring data matter itself
Amount.To solve the above-mentioned problems, this application provides a kind of data quality monitoring method, system and storage mediums.
Embodiment one
Fig. 1 is that the embodiment of the present application one provides a kind of flow diagram of data quality monitoring method.Referring to Fig.1, this Shen
The data quality monitoring method that please be provided includes:
S101, monitoring metadata were modified if it exists, are alerted with the presence or absence of the case where being modified；
Wherein, monitoring metadata, which whether there is, includes: whether detection metadata is deleted by physics by modification；
It is another to cry that " physics is deleted one is " logic deletions " it is to be understood that delete there are two types of the modes of data
Except ".
Logic is deleted as its name suggests, and file is not deleted really, and only the first character section of filename is changed
At the unrecognized character of operating system, usual this delete operation is reversible, that is can with proper implements or software
The file access pattern of deletion is come out.Physics, which is deleted, refers to that the used magnetic storage region of file storage is really wiped or clearly
Zero, the file deleted in this way cannot restore.Both deletion modes can be detected by relative program.
S102, whether monitoring metadata is complete, alerts if imperfect；
Further, check data whether completely include:
Whether detection metadata, which has, increases undefined enumerated value newly, and then metadata is imperfect if it exists；
Whether detection enumerated value has remarks, and metadata is imperfect if without remarks；
Whether detection enumerated value has index, and metadata is imperfect if not indexing.
The integrality of data includes many conditions in practical application, does not increase undefined piece newly with metadata in the application
Act value, each single item enumerated value, which all has remarks and indexes to be used as, judges the complete necessary condition of data.
Enumerate be a kind of fixation type, and one is enumerated and corresponds to several values defined, and enumerated value is known as.Example
Such as: defining week, this is enumerated, since being week (week), its value has seven, Monday to Sunday, and this seven
Each of be referred to as enumerated value.
S103 is monitored stored data, if it is abnormal to find that stored data exist, analyzes abnormal cause,
And it alerts；
Wherein, there are many modes of alarm, specific alarm mode can be following several: send to reserved number short
Letter is to alert；Mailbox is sent to reserved mailbox to alert；It issues prompt tone or lights indicator light to alert.
S104, the result based on monitoring generate Data Quality Analysis report.
In the data quality monitoring method of the offer of the application, metadata can be monitored with the presence or absence of being modified, if it exists
Modification then alerts；It whether complete monitors metadata, is alerted if imperfect；Stored data are monitored, if discovery is
There is abnormal, analysis abnormal cause in the data of storage, and alert；Result based on monitoring generates Data Quality Analysis report.Such as
This completes the monitoring for the quality of data, and when quality of data difference (such as metadata is imperfect, metadata is modified, and has been stored
Data there is situations such as abnormal) alarm in time, remind the owner of the data quality of data situation, avoid the owner of data
Inappropriate decision is made according to the data.So complete the monitoring for the quality of data.
In practical application, the application carries out data monitoring from each process such as the generation, transmission, storage of data.
During data generate:
In method provided by the present application, (whether mainly detection metadata is by object with the presence or absence of being modified for monitoring metadata
Reason is deleted) it can effectively place data and modified；Further, whether detection metadata, which has, increases undefined enumerated value newly,
Then metadata is imperfect if it exists；Whether detection enumerated value has remarks, and metadata is imperfect if without remarks；Detect enumerated value
Whether index is had, metadata is imperfect if not indexing.So ensure that data generate when integrality, if while data not
It up to requiring, is alerted, in order to which according to supplementing, the modes alerted can mention majority related personnel to issue in time herein
Show that sound alerts, prompt tone alarm can timely notify related personnel, and when data generation typing, generally have related personnel
Before the recording devices such as computer, so related personnel can be prompted as soon as possible by issuing prompt tone by recording device.
In data storage procedure:
In method provided by the present application, stored data are monitored, if it is abnormal to find that stored data exist,
Abnormal cause is analyzed, and is alerted；
Wherein, major way is monitored to stored data are as follows: total amount of data ring ratio is carried out to stored data
Difference monitoring, the monitoring of field repetition values, the monitoring of field exceptional value, index operation ring compare abnormal monitoring.Pass through total amount of data ring ratio
Difference monitoring, the monitoring of field repetition values, the monitoring of field exceptional value, index operation ring not only can be with monitoring data than abnormal monitoring
Variation, data report can also be generated.The mode alerted at this time can choose send short message to reserved number to alert or
To reserved number call warning by telephone.Sending short message and making a phone call to be alerted not only quickly can also depart from distance limit
System.The personnel whether personnel for no matter needing to inform can timely inform to needs near storage equipment or Administrative Area
Alarm.
It is to be understood that total amount of data ring can be with monitoring data total amount ring than changing, when changing greatly than difference monitoring
When obtain when variation occurs abnormal (such as total amount of data becomes more suddenly or tails off) and alert in time, in order to the timely logarithm of related personnel
According to circulation process checked, detect total amount of data variation the reason of.Same field repetition values monitoring, field exceptional value
Monitoring, index operation ring is more equal than abnormal monitoring can be alerted when testing result exceeds preset range, in order to relevant people
Member in time checks the circulation process of data.
Further, the application can also summarize the result of detection to generate Data Quality Analysis report.
Examining report mainly has following two purpose: first is that, monitoring data generates the step of malfunctioning in storage links,
In order to carry out subsequent correction；Second is that reasonable utilization Quality Analysis Report, analyze that some data can express to public affairs
Department or the helpful information of enterprise, reasonable utilization data.
After examining report generates, mail can be sent to scheduled mailbox.In order to preferably related personnel be reminded to connect
It receives, short message can also be sent to reserved telephone number simultaneously or makes a phone call, in order to which related personnel receives mail as soon as possible.
Embodiment two
Fig. 2 is that the embodiment of the present application two provides a kind of structural schematic diagram of data quality monitoring system.Referring to Fig. 2, this Shen
The data quality monitoring system that please be provided includes: processor 202 and memory 201；
Processor 202 and memory 201 communicate to connect:
Wherein, processor 202, for calling and executing the program stored in memory 201；
Memory 201, for storing program, program is at least used for each of the data quality monitoring method of the offer of embodiment one
A step.
So set, the data quality monitoring system of the offer of the application, metadata can be monitored with the presence or absence of being modified,
It modifies, alerts if it exists；It whether complete monitors metadata, is alerted if imperfect；Stored data are monitored, if
It was found that stored data have abnormal, analysis abnormal cause, and alert；Result based on monitoring generates Data Quality Analysis report
It accuses.The monitoring for the quality of data is so completed, and when quality of data difference (such as metadata is imperfect, metadata is modified,
There is situations such as abnormal in the data of storage) alarm in time, the owner of the data quality of data situation is reminded, gathering around for data is avoided
The person of having makes inappropriate decision according to the data.So complete the monitoring for the quality of data.
Embodiment three
The application also provides a kind of storage medium, is stored thereon with computer program, and computer program is executed by processor
When, each step of the data quality monitoring method provided such as claim embodiment one is provided.
About the device in above-described embodiment, wherein modules execute the concrete mode of operation in related this method
Embodiment in be described in detail, no detailed explanation will be given here.
It is understood that same or similar part can mutually refer in the various embodiments described above, in some embodiments
Unspecified content may refer to the same or similar content in other embodiments.
It should be noted that term " first ", " second " etc. are used for description purposes only in the description of the present application, without
It can be interpreted as indication or suggestion relative importance.In addition, in the description of the present application, unless otherwise indicated, the meaning of " multiple "
Refer at least two.
Any process described otherwise above or method description are construed as in flow chart or herein, and expression includes
It is one or more for realizing specific logical function or process the step of executable instruction code module, segment or portion
Point, and the range of the preferred embodiment of the application includes other realization, wherein can not press shown or discussed suitable
Sequence, including according to related function by it is basic simultaneously in the way of or in the opposite order, Lai Zhihang function, this should be by the application
Embodiment person of ordinary skill in the field understood.
It should be appreciated that each section of the application can be realized with hardware, software, firmware or their combination.Above-mentioned
In embodiment, software that multiple steps or method can be executed in memory and by suitable instruction execution system with storage
Or firmware is realized.It, and in another embodiment, can be under well known in the art for example, if realized with hardware
Any one of column technology or their combination are realized: having a logic gates for realizing logic function to data-signal
Discrete logic, with suitable combinational logic gate circuit specific integrated circuit, programmable gate array (PGA), scene
Programmable gate array (FPGA) etc..
Those skilled in the art are understood that realize all or part of step that above-described embodiment method carries
Suddenly be that relevant hardware can be instructed to complete by program, program can store in a kind of computer readable storage medium
In, which when being executed, includes the steps that one or a combination set of embodiment of the method.
It, can also be in addition, can integrate in a processing module in each functional unit in each embodiment of the application
It is that each unit physically exists alone, can also be integrated in two or more units in a module.Above-mentioned integrated mould
Block both can take the form of hardware realization, can also be realized in the form of software function module.If integrated module with
The form of software function module is realized and when sold or used as an independent product, also can store computer-readable at one
It takes in storage medium.
Storage medium mentioned above can be read-only memory, disk or CD etc..
In the description of this specification, reference term " one embodiment ", " some embodiments ", " example ", " specifically show
The description of example " or " some examples " etc. means specific features, structure, material or spy described in conjunction with this embodiment or example
Point is contained at least one embodiment or example of the application.In the present specification, schematic expression of the above terms are not
Centainly refer to identical embodiment or example.Moreover, particular features, structures, materials, or characteristics described can be any
One or more embodiment or examples in can be combined in any suitable manner.
Although embodiments herein has been shown and described above, it is to be understood that above-described embodiment is example
Property, it should not be understood as the limitation to the application, those skilled in the art within the scope of application can be to above-mentioned
Embodiment is changed, modifies, replacement and variant.
Claims (10)
1. a kind of data quality monitoring method characterized by comprising
Monitoring metadata was modified if it exists, is alerted with the presence or absence of the case where being modified；
It whether complete monitors metadata, is alerted if imperfect；
Stored data are monitored, if it is abnormal to find that stored data exist, analyze abnormal cause, and alert；
Result based on the monitoring generates Data Quality Analysis report.
2. being modified the method according to claim 1, wherein monitoring metadata whether there is, comprising:
Whether detection metadata is deleted by physics.
3. the method according to claim 1, wherein whether the monitoring metadata is complete, comprising:
Whether detection metadata, which has, increases undefined enumerated value newly, and then metadata is imperfect if it exists；
Whether detection enumerated value has remarks, and metadata is imperfect if without remarks；
Whether detection enumerated value has index, and metadata is imperfect if not indexing.
4. the method according to claim 1, wherein described be monitored stored data, comprising:
It carries out total amount of data ring to stored data to monitor than difference, the monitoring of field repetition values, the monitoring of field exceptional value, index
Operation ring compares abnormal monitoring.
5. the method according to claim 1, wherein the result based on the monitoring generates Data Quality Analysis report
It accuses, comprising:
Summarize the result of the detection to generate Data Quality Analysis report.
6. the method according to claim 1, wherein the mode of the alarm includes:
Short message is sent to reserved number to alert.
7. the method according to claim 1, wherein the mode of the alarm includes:
Mailbox is sent to reserved mailbox to alert.
8. the method according to claim 1, wherein the mode of the alarm includes:
It issues prompt tone or lights indicator light to alert.
9. a kind of data quality monitoring system characterized by comprising processor and memory；
The processor and memory communicate to connect:
Wherein, the processor, for calling and executing the program stored in the memory；
The memory, for storing program, described program is at least used for perform claim and requires 1~8 described in any item data
Each step of quality control method.
10. a kind of storage medium, which is characterized in that be stored thereon with computer program, the computer program is held by processor
When row, each step of data quality monitoring method as described in any one of claims 1 to 8 is realized.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910246182.3A CN109918271A (en) | 2019-03-28 | 2019-03-28 | Data quality monitoring method, system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910246182.3A CN109918271A (en) | 2019-03-28 | 2019-03-28 | Data quality monitoring method, system and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109918271A true CN109918271A (en) | 2019-06-21 |
Family
ID=66967533
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910246182.3A Pending CN109918271A (en) | 2019-03-28 | 2019-03-28 | Data quality monitoring method, system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109918271A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110704322A (en) * | 2019-09-30 | 2020-01-17 | 上海中通吉网络技术有限公司 | Software testing method and system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140336984A1 (en) * | 2013-05-13 | 2014-11-13 | Abb Technology Ag. | Conditional monitoring of industrial systems |
| CN104766151A (en) * | 2014-12-29 | 2015-07-08 | 国家电网公司 | Quality management and control method for electricity transaction data warehouses and management and control system thereof |
| CN105095056A (en) * | 2015-08-14 | 2015-11-25 | 焦点科技股份有限公司 | Method for monitoring data in data warehouse |
| CN107484189A (en) * | 2017-07-27 | 2017-12-15 | 北京市天元网络技术股份有限公司 | LTE data handling systems |
-
2019
- 2019-03-28 CN CN201910246182.3A patent/CN109918271A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140336984A1 (en) * | 2013-05-13 | 2014-11-13 | Abb Technology Ag. | Conditional monitoring of industrial systems |
| CN104766151A (en) * | 2014-12-29 | 2015-07-08 | 国家电网公司 | Quality management and control method for electricity transaction data warehouses and management and control system thereof |
| CN105095056A (en) * | 2015-08-14 | 2015-11-25 | 焦点科技股份有限公司 | Method for monitoring data in data warehouse |
| CN107484189A (en) * | 2017-07-27 | 2017-12-15 | 北京市天元网络技术股份有限公司 | LTE data handling systems |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110704322A (en) * | 2019-09-30 | 2020-01-17 | 上海中通吉网络技术有限公司 | Software testing method and system |
| CN110704322B (en) * | 2019-09-30 | 2023-03-10 | 上海中通吉网络技术有限公司 | Software testing method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11902316B2 (en) | Real-time cybersecurity status system with event ticker | |
| CN106961352B (en) | Monitoring system and monitoring method | |
| US7456736B2 (en) | Extensible sensor monitoring, alert processing and notification system and method | |
| US6289379B1 (en) | Method for monitoring abnormal behavior in a computer system | |
| US6367034B1 (en) | Using query language for event filtering and aggregation | |
| US7779423B2 (en) | Detecting entity changes in a storage area network environment | |
| US20070011300A1 (en) | Monitoring method and system for monitoring operation of resources | |
| US20070050777A1 (en) | Duration of alerts and scanning of large data stores | |
| US20150086018A1 (en) | Centralized key discovery and management | |
| US20150086009A1 (en) | Handling key rotation problems | |
| CN105471932A (en) | Front-end application monitoring method, front-end application and front-end application monitoring system | |
| WO2006138469A2 (en) | Duration of alerts and scanning of large data stores | |
| CN106331176A (en) | Interaction platform of internal and external networks | |
| CN109710444A (en) | The method and relevant device of the abnormality processing of intelligent hospital information system | |
| CN109918271A (en) | Data quality monitoring method, system and storage medium | |
| CN115860912A (en) | Wind control monitoring early warning system and method based on variable blood margin | |
| CN113836160B (en) | Data stream state monitoring alarm system based on master-slave synchronization | |
| EP1929706B1 (en) | Device management system using log management object and method for generating and controlling logging data therein | |
| US20140143215A1 (en) | Assured federated records management | |
| CN111754077A (en) | Alarm convergence analysis method and system | |
| CN109597728A (en) | The control method and device of test equipment, computer readable storage medium | |
| CN105574055B (en) | Prevent the method and device that memory exhausts | |
| CN103701667A (en) | Method, device and system for monitoring heartbeat of server | |
| CN109344036A (en) | Alarm display method and system applied to storage system | |
| CN109992483A (en) | A kind of temperature monitoring method, device, equipment and readable storage medium storing program for executing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190621 |