CN111597088A - Data warehouse data monitoring method, warehouse system and electronic equipment - Google Patents
Data warehouse data monitoring method, warehouse system and electronic equipment Download PDFInfo
- Publication number
- CN111597088A CN111597088A CN202010411431.2A CN202010411431A CN111597088A CN 111597088 A CN111597088 A CN 111597088A CN 202010411431 A CN202010411431 A CN 202010411431A CN 111597088 A CN111597088 A CN 111597088A
- Authority
- CN
- China
- Prior art keywords
- data
- fluctuation interval
- monitoring
- reference value
- historical
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 65
- 238000000034 method Methods 0.000 title claims abstract description 31
- 230000001360 synchronised effect Effects 0.000 claims abstract description 55
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 36
- 238000013524 data verification Methods 0.000 claims abstract description 8
- 238000004364 calculation method Methods 0.000 claims description 14
- 230000002159 abnormal effect Effects 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 10
- 230000006870 function Effects 0.000 description 5
- 238000004590 computer program Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000002085 persistent effect Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 3
- 238000013500 data storage Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 239000002355 dual-layer Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2308—Concurrency control
- G06F16/2315—Optimistic concurrency control
- G06F16/2322—Optimistic concurrency control using timestamps
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2477—Temporal data queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Computing Systems (AREA)
- Fuzzy Systems (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Quality & Reliability (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application relates to a data warehouse data monitoring method, a warehouse system and electronic equipment. The method comprises the following steps: timing and synchronizing source data to a data warehouse; counting the monitoring indexes of the source data synchronized on the same day; calculating a reference value of historical synchronous data through a time sequence algorithm; calculating a data fluctuation interval according to the monitoring index and the reference value of the historical synchronous data; and generating a monitoring report according to the comparison result of the data fluctuation interval and a preset data fluctuation interval. The historical synchronization data and the actual statistical value of the source data synchronized on the same day are analyzed and compared through a time sequence algorithm, and the fluctuation interval is configured to realize automatic data verification, so that the reliability of data of a data warehouse can be improved, and the data problem after the data warehouse data is synchronized can be quickly found.
Description
Technical Field
The present application relates to the field of data storage technologies, and in particular, to a data warehouse data monitoring method, a warehouse system, and an electronic device.
Background
A data warehouse is a strategic set that provides all types of data support for all levels of decision-making processes of an enterprise. It is a single data store created for analytical reporting and decision support purposes. Currently, the data warehouse, which provides data synchronization and data storage functions, is generally divided into three parts: a source database, a data warehouse, and a scheduling system. And the synchronization, conversion and loading of data are realized by scheduling a system to call tasks at regular time. In the existing data warehouse, data of a source database is directly synchronized to the data warehouse through a data synchronization tool, the data is stored in the data warehouse, and then the data is reprocessed, so that data support is provided for various data analysis and decision making. The existing data warehouse system does not check the synchronous data, so that the reliability of the data cannot be guaranteed.
Disclosure of Invention
In order to overcome the problems in the related art, the data warehouse data monitoring method, the data warehouse system and the electronic device are provided, so that the reliability of data warehouse data can be improved, and the data problem after the data warehouse data are synchronized can be quickly found.
According to a first aspect of embodiments of the present application, a data warehouse data monitoring method is provided, which includes periodically synchronizing source data to a data warehouse;
counting the monitoring indexes of the source data synchronized on the same day;
calculating a reference value of historical synchronous data through a time sequence algorithm;
calculating a data fluctuation interval according to the monitoring index and the reference value of the historical synchronous data;
and generating a monitoring report according to the comparison result of the data fluctuation interval and a preset data fluctuation interval.
According to a second aspect of the embodiments of the present application, a data warehouse system is provided, which includes a synchronization module, a statistics module, a reference value calculation module, a fluctuation interval calculation module, and a monitoring module;
the synchronization module is used for regularly synchronizing the source data to the data warehouse;
the statistical module is used for counting the monitoring indexes of the source data synchronized on the same day;
the reference value calculating module is used for calculating a reference value of the historical synchronous data through a time sequence algorithm;
the fluctuation interval calculation module is used for calculating a data fluctuation interval according to the monitoring index and the reference value of the historical synchronous data;
and the monitoring module is used for generating a monitoring report according to the comparison result of the fluctuation interval and a preset data fluctuation interval.
According to a third aspect of embodiments of the present application, there is provided an electronic apparatus, including:
a processor; and
a memory having executable code stored thereon, which when executed by the processor, causes the processor to perform the method as described above.
The technical scheme provided by the embodiment of the application can have the following beneficial effects: the historical synchronization data and the actual statistical value of the source data synchronized on the same day are analyzed and compared through a time sequence algorithm, and the fluctuation interval is configured to realize automatic data verification, so that the reliability of data of a data warehouse can be improved, and the data problem after the data warehouse data is synchronized can be quickly found.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.
Drawings
The foregoing and other objects, features and advantages of the application will be apparent from the following more particular descriptions of exemplary embodiments of the application, as illustrated in the accompanying drawings wherein like reference numbers generally represent like parts throughout the exemplary embodiments of the application.
FIG. 1 is a schematic flow chart diagram illustrating a data warehouse data monitoring method according to an exemplary embodiment of the present application;
FIG. 2 is a schematic diagram of a data warehouse system architecture according to an exemplary embodiment of the present application;
fig. 3 is a schematic structural diagram of an electronic device according to an exemplary embodiment of the present application.
Detailed Description
Preferred embodiments of the present application will be described in more detail below with reference to the accompanying drawings. While the preferred embodiments of the present application are shown in the drawings, it should be understood that the present application may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It should be understood that although the terms "first," "second," "third," etc. may be used herein to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the present application, "a plurality" means two or more unless specifically limited otherwise.
The application provides a data warehouse data monitoring method, a warehouse system and electronic equipment, which can analyze statistical values of historical synchronous data and actual source data synchronized on the same day through a time sequence algorithm for comparison, and configure a fluctuation interval to realize automatic data verification, so that the reliability of data warehouse data can be improved, and data problems after data warehouse data synchronization can be quickly found.
The technical solutions of the embodiments of the present application are described in detail below with reference to the accompanying drawings.
Fig. 1 is a schematic flow chart diagram illustrating a data warehouse data monitoring method according to an exemplary embodiment of the present application.
Referring to fig. 1, a data warehouse data monitoring method includes:
and S101, timing and synchronizing the source data to the data warehouse. After S101 is completed, the process proceeds to S102 or S103.
And S102, counting the monitoring indexes of the source data synchronized on the same day. In a preferred embodiment, the monitoring metrics include: the recording number of data and the data size. It should be understood by those skilled in the art that the monitoring index herein includes not only the number of data records and the size of data volume, but also other data indexes, and all data indexes that can be used for data verification can be used in the present invention.
And S103, calculating a reference value of the historical synchronous data through a time sequence algorithm.
In a preferred embodiment, the record number of the historical synchronization data is calculated through a time sequence algorithm of a moving average algorithm, and a moving predicted value of the data volume is used as a reference value of the historical synchronization data. In a preferred embodiment, the historical synchronization data is data that is synchronized for approximately seven days.
It should be noted that the execution sequence of S101 and S102 is not limited in this application, and may be executed simultaneously or sequentially. After S101 is completed, S102 or S103 is automatically started.
After completing S102 and S103, the process proceeds to S104, and a data fluctuation interval is calculated by the monitoring index and the reference value of the history synchronization data.
In a preferred embodiment, the number of records of source data synchronized on the current day, the data size statistic value are compared with the number of records of the historical synchronization data calculated by the time sequence algorithm of the moving average algorithm, and the moving prediction value of the data size is compared to calculate the data fluctuation interval. In a preferred embodiment, the timing algorithm selects the last seven day moving average algorithm. Of course, other timing algorithms may be selected for implementation in the present application.
And S105, generating a monitoring report according to the comparison result of the data fluctuation interval and a preset data fluctuation interval.
In a preferred embodiment, generating a monitoring report according to the comparison result of the data fluctuation interval and a preset data fluctuation interval includes:
and comparing the fluctuation interval with a preset data fluctuation interval.
And when the fluctuation interval is within a preset data fluctuation interval range, generating a monitoring report with normal data synchronization.
And when the fluctuation interval is not in the preset data fluctuation interval range, generating a monitoring report of abnormal data synchronization.
The preset data fluctuation interval is obtained by longitudinally comparing historical synchronous data. In a preferred embodiment, the fluctuation interval may be set to 10%, that is, the number of records of source data synchronized during the current day, the size of data amount and the number of records of historical synchronization data calculated by the time sequence algorithm, and if the amplitude of comparison of the moving average of the size of data amount exceeds 10%, the synchronization data is considered to be abnormal, and a monitoring report of the abnormal synchronization data is generated. And comparing the amplitude of the moving average of the data volume size with the record number of the historical synchronous data calculated by the time sequence algorithm, wherein the record number of the source data synchronized on the same day, the data volume size and the record number of the historical synchronous data do not exceed 10%, and generating a monitoring report that the synchronous data are normal.
As can be seen from the embodiment, the method synchronizes source data to the data warehouse in a timing manner; counting the monitoring indexes of the source data synchronized on the same day; calculating a reference value of historical synchronous data through a time sequence algorithm; calculating a data fluctuation interval according to the monitoring index and the reference value of the historical synchronous data; and generating a monitoring report according to the comparison result of the data fluctuation interval and a preset data fluctuation interval. The historical synchronization data and the actual statistical value of the source data synchronized on the same day are analyzed and compared through a time sequence algorithm, and the fluctuation interval is configured to realize automatic data verification, so that the reliability of data of a data warehouse can be improved, and the data problem after the data warehouse data is synchronized can be quickly found.
Corresponding to the embodiment of the application function implementation method, the application also provides a data warehouse system, electronic equipment and corresponding embodiments.
Fig. 2 is a block diagram of a data warehouse system according to an exemplary embodiment of the present application.
Referring to fig. 2, a data warehouse system, comprising: a synchronization module 201, a statistics module 202, a reference value calculation module 203, a fluctuation interval calculation module 204, and a monitoring module 205.
A synchronization module 201 for timing synchronization of the source data to the data warehouse.
In a preferred embodiment, the auto-launch statistics module 202 and the reference value calculation module 203 begin operation after the synchronization module 201 has synchronized the source data to the data warehouse.
The statistical module 202 is configured to count monitoring indexes of source data synchronized on the same day. In a preferred embodiment, the monitoring metrics include: the recording number of data and the data size.
And the reference value calculating module 203 is used for calculating the reference value of the historical synchronization data through a time sequence algorithm.
In a preferred embodiment, the reference value calculating module 203 calculates the record number of the history synchronous data through a time sequence algorithm of a moving average algorithm, and a moving predicted value of the data size is used as the reference value of the history synchronous data. In a preferred embodiment, the historical synchronization data is data that is synchronized for approximately seven days.
And a fluctuation interval calculation module 204, configured to calculate a data fluctuation interval according to the monitoring index and the reference value of the historical synchronization data.
In a preferred embodiment, the fluctuation interval calculation module 204 compares the number of records of source data synchronized during the current day, the data size statistic with the number of records of the historical synchronization data calculated by the time sequence algorithm of the moving average algorithm, and the moving prediction value of the data size to calculate the data fluctuation interval. In a preferred embodiment, the timing algorithm selects the last seven day moving average algorithm. Of course, other timing algorithms may be selected for implementation in the present application.
And the monitoring module 205 is configured to generate a monitoring report according to a comparison result between the fluctuation interval and a preset data fluctuation interval.
In a preferred embodiment, the monitoring module 205 generates the monitoring report according to the comparison result between the data fluctuation interval and the preset data fluctuation interval, and includes the following steps:
and comparing the fluctuation interval with a preset data fluctuation interval.
And when the fluctuation interval is within a preset data fluctuation interval range, generating a monitoring report with normal data synchronization.
And when the fluctuation interval is not in the preset data fluctuation interval range, generating a monitoring report of abnormal data synchronization. The preset data fluctuation interval is obtained by longitudinally comparing historical synchronous data.
In a preferred embodiment, the data warehouse system of the present application further includes a data verification result database 206, configured to store a current day data monitoring index, a reference value of the historical synchronization data, a data fluctuation interval, and a comparison result between the data fluctuation interval and a preset data fluctuation interval.
With regard to the system in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
As can be seen from this embodiment, the synchronization module of the system of the present application periodically synchronizes source data to the data warehouse; the statistical module is used for counting the monitoring indexes of the source data synchronized on the same day; the reference value calculation module calculates the reference value of the historical synchronous data through a time sequence algorithm; the fluctuation interval calculation module calculates a data fluctuation interval through the monitoring index and the reference value of the historical synchronous data; and the monitoring module generates a monitoring report according to the comparison result of the data fluctuation interval and a preset data fluctuation interval. The historical synchronization data and the actual statistical value of the source data synchronized on the same day are analyzed and compared through a time sequence algorithm, and the fluctuation interval is configured to realize automatic data verification, so that the reliability of data of a data warehouse can be improved, and the data problem after the data warehouse data is synchronized can be quickly found.
Fig. 3 is a schematic structural diagram of an electronic device according to an exemplary embodiment of the present application.
Referring to fig. 3, the electronic device 300 includes a memory 310 and a processor 320.
The Processor 320 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 310 may include various types of storage units, such as system memory, Read Only Memory (ROM), and permanent storage. Wherein the ROM may store static data or instructions for the processor 320 or other modules of the computer. The persistent storage device may be a read-write storage device. The persistent storage may be a non-volatile storage device that does not lose stored instructions and data even after the computer is powered off. In some embodiments, the persistent storage device employs a mass storage device (e.g., magnetic or optical disk, flash memory) as the persistent storage device. In other embodiments, the permanent storage may be a removable storage device (e.g., floppy disk, optical drive). The system memory may be a read-write memory device or a volatile read-write memory device, such as a dynamic random access memory. The system memory may store instructions and data that some or all of the processors require at runtime. Further, the memory 310 may comprise any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), magnetic and/or optical disks, may also be employed. In some embodiments, memory 310 may include a removable storage device that is readable and/or writable, such as a Compact Disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, dual layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, min SD card, Micro-SD card, etc.), a magnetic floppy disc, or the like. Computer-readable storage media do not contain carrier waves or transitory electronic signals transmitted by wireless or wired means.
The memory 310 has stored thereon executable code that, when processed by the processor 320, may cause the processor 320 to perform some or all of the methods described above.
The aspects of the present application have been described in detail hereinabove with reference to the accompanying drawings. In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments. Those skilled in the art should also appreciate that the acts and modules referred to in the specification are not necessarily required in the present application. In addition, it can be understood that the steps in the method of the embodiment of the present application may be sequentially adjusted, combined, and deleted according to actual needs, and the modules in the device of the embodiment of the present application may be combined, divided, and deleted according to actual needs.
Furthermore, the method according to the present application may also be implemented as a computer program or computer program product comprising computer program code instructions for performing some or all of the steps of the above-described method of the present application.
Alternatively, the present application may also be embodied as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) having stored thereon executable code (or a computer program, or computer instruction code) which, when executed by a processor of an electronic device (or electronic device, server, etc.), causes the processor to perform part or all of the various steps of the above-described method according to the present application.
Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the applications disclosed herein may be implemented as electronic hardware, computer software, or combinations of both.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems and methods according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
Having described embodiments of the present application, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen in order to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.
Claims (10)
1. A data warehouse data monitoring method, comprising:
timing and synchronizing source data to a data warehouse;
counting the monitoring indexes of the source data synchronized on the same day;
calculating a reference value of historical synchronous data through a time sequence algorithm;
calculating a data fluctuation interval according to the monitoring index and the reference value of the historical synchronous data;
and generating a monitoring report according to the comparison result of the data fluctuation interval and a preset data fluctuation interval.
2. The method of claim 1, comprising:
the monitoring indexes comprise: the recording number of data and the data size.
3. The method of claim 2, comprising: wherein, the calculation of the reference value of the historical synchronization data by the time sequence algorithm comprises the following steps:
and calculating the record number of the historical synchronous data through a time sequence algorithm of a moving average algorithm, wherein a moving predicted value of the data volume is used as a reference value of the historical synchronous data.
4. The method of claim 3, wherein calculating a data fluctuation interval from the monitoring index and the reference value of the historical synchronization data comprises:
and comparing the record number of the source data synchronized on the current day, the data size statistic with the record number of the historical synchronization data calculated by the time sequence algorithm of the moving average algorithm and the moving predicted value of the data size to calculate the data fluctuation interval.
5. The method of claim 4, wherein generating a monitoring report according to the comparison result of the data fluctuation interval and a preset data fluctuation interval comprises:
comparing the fluctuation interval with a preset data fluctuation interval;
when the fluctuation interval is within a preset data fluctuation interval range, generating a monitoring report with normal data synchronization;
and when the fluctuation interval is not in the preset data fluctuation interval range, generating a monitoring report of abnormal data synchronization.
6. The method according to any one of claims 1 to 5, wherein the preset data fluctuation interval is obtained by longitudinal comparison of historical synchronization data.
7. A data warehouse system, comprising: the device comprises a synchronization module, a statistic module, a reference value calculation module, a fluctuation interval calculation module and a monitoring module;
the synchronization module is used for regularly synchronizing the source data to the data warehouse;
the statistical module is used for counting the monitoring indexes of the source data synchronized on the same day;
the reference value calculating module is used for calculating a reference value of the historical synchronous data through a time sequence algorithm;
the fluctuation interval calculation module is used for calculating a data fluctuation interval according to the monitoring index and the reference value of the historical synchronous data;
and the monitoring module is used for generating a monitoring report according to the comparison result of the fluctuation interval and a preset data fluctuation interval.
8. The system of claim 7, further comprising: and the data verification result database is used for storing the data monitoring index of the current day, the reference value of the historical synchronous data, the data fluctuation interval and the comparison result of the data fluctuation interval and the preset data fluctuation interval.
9. The system of claim 7 or 8, wherein the monitoring metrics comprise: the recording number of data and the data size.
10. An electronic device, comprising:
a processor; and
a memory having executable code stored thereon, which when executed by the processor, causes the processor to perform the method of any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010411431.2A CN111597088A (en) | 2020-05-15 | 2020-05-15 | Data warehouse data monitoring method, warehouse system and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010411431.2A CN111597088A (en) | 2020-05-15 | 2020-05-15 | Data warehouse data monitoring method, warehouse system and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111597088A true CN111597088A (en) | 2020-08-28 |
Family
ID=72191056
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010411431.2A Pending CN111597088A (en) | 2020-05-15 | 2020-05-15 | Data warehouse data monitoring method, warehouse system and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111597088A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112417841A (en) * | 2020-11-20 | 2021-02-26 | 浪潮卓数大数据产业发展有限公司 | Data checking method |
| CN112423032A (en) * | 2020-10-21 | 2021-02-26 | 当趣网络科技(杭州)有限公司 | Data monitoring method and device based on smart television, electronic equipment and medium |
| CN112688922A (en) * | 2020-12-11 | 2021-04-20 | 深圳前海微众银行股份有限公司 | Data transmission method, system, device and medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2004005305A (en) * | 2002-06-03 | 2004-01-08 | Hitachi Ltd | Monitoring method of memory use capacity, and computer system |
| CN102339288A (en) * | 2010-07-21 | 2012-02-01 | ***通信集团辽宁有限公司 | Method and device for detecting abnormal data of data warehouse |
| CN104572391A (en) * | 2013-10-16 | 2015-04-29 | 深圳市腾讯计算机***有限公司 | Monitoring alarm strategy collocation method and device and monitoring alarm method and device |
| CN107705149A (en) * | 2017-09-22 | 2018-02-16 | 平安科技（深圳）有限公司 | Data method for real-time monitoring, device, terminal device and storage medium |
| CN107766299A (en) * | 2017-10-24 | 2018-03-06 | 携程旅游信息技术（上海）有限公司 | The abnormal monitoring method of data target and its system, storage medium, electronic equipment |
| CN108920336A (en) * | 2018-05-25 | 2018-11-30 | 麒麟合盛网络技术股份有限公司 | A kind of service abnormity prompt method and device based on time series |
| CN110287229A (en) * | 2019-05-30 | 2019-09-27 | 阿里巴巴集团控股有限公司 | A kind of data statistics processing method and device |
-
2020
- 2020-05-15 CN CN202010411431.2A patent/CN111597088A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2004005305A (en) * | 2002-06-03 | 2004-01-08 | Hitachi Ltd | Monitoring method of memory use capacity, and computer system |
| CN102339288A (en) * | 2010-07-21 | 2012-02-01 | ***通信集团辽宁有限公司 | Method and device for detecting abnormal data of data warehouse |
| CN104572391A (en) * | 2013-10-16 | 2015-04-29 | 深圳市腾讯计算机***有限公司 | Monitoring alarm strategy collocation method and device and monitoring alarm method and device |
| CN107705149A (en) * | 2017-09-22 | 2018-02-16 | 平安科技（深圳）有限公司 | Data method for real-time monitoring, device, terminal device and storage medium |
| CN107766299A (en) * | 2017-10-24 | 2018-03-06 | 携程旅游信息技术（上海）有限公司 | The abnormal monitoring method of data target and its system, storage medium, electronic equipment |
| CN108920336A (en) * | 2018-05-25 | 2018-11-30 | 麒麟合盛网络技术股份有限公司 | A kind of service abnormity prompt method and device based on time series |
| CN110287229A (en) * | 2019-05-30 | 2019-09-27 | 阿里巴巴集团控股有限公司 | A kind of data statistics processing method and device |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112423032A (en) * | 2020-10-21 | 2021-02-26 | 当趣网络科技(杭州)有限公司 | Data monitoring method and device based on smart television, electronic equipment and medium |
| CN112417841A (en) * | 2020-11-20 | 2021-02-26 | 浪潮卓数大数据产业发展有限公司 | Data checking method |
| CN112417841B (en) * | 2020-11-20 | 2023-09-05 | 浪潮卓数大数据产业发展有限公司 | Data verification method |
| CN112688922A (en) * | 2020-12-11 | 2021-04-20 | 深圳前海微众银行股份有限公司 | Data transmission method, system, device and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111597088A (en) | Data warehouse data monitoring method, warehouse system and electronic equipment | |
| US20180203767A1 (en) | Method and apparatus for job operation retry | |
| CN109934712B (en) | Account checking method and account checking device applied to distributed system and electronic equipment | |
| CN106899654B (en) | Sequence value generation method, device and system | |
| CN111090502B (en) | Stream data task scheduling method and device | |
| US7995300B2 (en) | Detection of defective tape drive by aggregating read error statistics | |
| CN106897342B (en) | Data verification method and equipment | |
| CN112579327A (en) | Fault detection method, device and equipment | |
| CN109739659B (en) | Message consumption method and system of distributed message queue | |
| KR101990329B1 (en) | Method and apparatus for improving database recovery speed using log data analysis | |
| CN111708812A (en) | Distributed data processing method | |
| CN113297009B (en) | Information backup method, equipment, platform and storage medium | |
| CN107463450B (en) | Data processing method and device for container | |
| CN112766536A (en) | Model training method, device and terminal for calculating road engineering labor unit price | |
| CN110928941B (en) | Data fragment extraction method and device | |
| CN115442262B (en) | Resource evaluation method and device, electronic equipment and storage medium | |
| CN116414914A (en) | Data synchronization method and device, processor and electronic equipment | |
| CN113032201B (en) | Hard disk bad track detection method | |
| CN110020744A (en) | Dynamic prediction method and its system | |
| CN112000521B (en) | Full backup method and device for distributed database system and computer readable storage medium | |
| CN110991945B (en) | Information processing method and device for equipment spare part, storage medium and processor | |
| US10102052B2 (en) | Dumping resources | |
| CN108712284B (en) | Fault service positioning method and device and service server | |
| CN108234196B (en) | Fault detection method and device | |
| CN107797915B (en) | Fault repairing method, device and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200828 |