CN109828979A - A kind of data consistency detection and system - Google Patents
A kind of data consistency detection and system Download PDFInfo
- Publication number
- CN109828979A CN109828979A CN201910100217.2A CN201910100217A CN109828979A CN 109828979 A CN109828979 A CN 109828979A CN 201910100217 A CN201910100217 A CN 201910100217A CN 109828979 A CN109828979 A CN 109828979A
- Authority
- CN
- China
- Prior art keywords
- data
- queue
- target data
- detection
- initial
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention relates to a kind of data consistency detections, belong to big data processing technology field, realize the consistency detection of target data and initial data.Include the following steps: S1: host receives initial data and target data；S2: host setting detection node；S3: host is according to detection node, extract initial data and data corresponding in target data, carry out data bulk comparison, judge data consistency, if quantity is consistent, the corresponding initial data of the detection node is consistent with target data, if quantity is inconsistent, then the corresponding initial data of the detection node and target data are inconsistent, return step S2.
Description
Technical field
The present invention relates to big data processing technology field more particularly to a kind of data consistency detection and systems.
Background technique
Vocabulary of the big data as IT industry most burning hot at present, the following data warehouse, data safety, data point
Analysis, data mining etc. are increasingly becoming the profit focus that industry personage falls over each other to pursue around the utilization of the commercial value of big data.
With the arriving of big data era, big data analysis is also come into being.
In the prior art, a large amount of data need to be entered into the table of standard, due to delay, typing unsuccessfully etc.,
It will lead to initial data and the target data in table be inconsistent, this is just to need to detect data, guarantees target data
With the consistency of initial data.
Summary of the invention
The purpose of the present invention one is to provide a kind of data consistency detection, realizes target data and initial data
Consistency detection.
Above-mentioned purpose one of the invention has the technical scheme that
A kind of data consistency detection, includes the following steps: S1: host receives initial data and target data；S2: host
Set detection node；S3: host extracts initial data and data corresponding in target data according to detection node, carries out data
Quantity compares, and judges data consistency, if quantity is consistent, the corresponding initial data of the detection node is consistent with target data,
If quantity is inconsistent, the corresponding initial data of the detection node and target data are inconsistent, return step S2.
Further, in the step S3, according to detection node, initial data and number corresponding in target data are extracted
According to later, further including step S301: host is according to the initial data of extraction and the size of target data, by initial data and target
Data are divided into multiple queues, and queue is respectively allocated to idle server and carries out data bulk calculating.
Further, after step S301, carry out step S302: host receives all original numbers under a certain detection node
According to compared with the quantity of the queue of target data, then counted.
Further, in the step S301, when server is when handling queue, no longer receiving queue, at queue
Continue to be in idle condition after having managed, receives next queue.
Further, the detection node is the period.
The purpose of the present invention two is to provide a kind of data consistency detection, realizes target data and initial data
Consistency detection.
Above-mentioned purpose two of the invention has the technical scheme that
A kind of data consistency detection system, comprising: host sets detection node for receiving initial data and target data,
Initial data and data corresponding in target data are extracted according to detection node；Server, for initial data and number of targets
According to progress data bulk comparison.
Further, the host is also used to the size of initial data and target data according to extraction, by initial data
Multiple queues are divided into target data, queue is respectively allocated to idle server and carries out data bulk calculating.
Further, host receives the quantity ratio of the queue of all initial data and target data under a certain detection node
Compared with, then counted.
Further, server is when handling queue, no longer receiving queue, and idle shape is kept after queue processing is complete
State receives next queue.
In conclusion the invention has the following advantages:
The quantity consistency for comparing initial data and target data by the tap node unit column to data is guaranteeing that data are quasi-
While true property, the efficiency of detection is also improved.
Detailed description of the invention
Fig. 1 is flow chart of the method for the present invention；
Fig. 2 is system block diagram of the invention.
Specific embodiment
Below in conjunction with attached drawing, the technical solution of the embodiment of the present invention is described.
Disclosed herein a kind of data consistency detection and systems, and method includes following step combined with Figure 1 and Figure 2,
It is rapid:
S1: host receives initial data and target data；
S2: host setting detection node；
S3: host extracts initial data and data corresponding in target data according to detection node, carries out data bulk comparison,
Judge data consistency, if quantity is consistent, the corresponding initial data of the detection node is consistent with target data, if quantity is different
It causes, then the corresponding initial data of the detection node and target data are inconsistent, return step S2.
As platform, the quantity of the database of access is very large.As using table as comparative unit, compare
Number be also very huge.For the timeliness and stability compared, serviced using task distribution processor mode multi-thread more
Journey processing data compare.
Therefore, in step s3, according to detection node, extract initial data with after data corresponding in target data,
Further include step S301: host is according to the initial data of extraction and the size of target data, by initial data and target data point
Multiple queues are cut into, queue is respectively allocated to idle server and carries out data bulk calculating.After step S301, walked
Rapid S302: host receives all initial data under a certain detection node compared with the quantity of the queue of target data, then carries out
Statistics.
Specifically, in data distribution, leader is obtained by zookeeper, zookeeper middleware is provided more
A primary server is chosen between a server, is used as by a wherein detection node service and is distributed all services, certain point
Hair service is also High Availabitity.In the case that i.e. delay machine occurs in some server, other servers, which are still capable of handling, to be distributed to
The task of delay machine server guarantees that the task of distribution is not in be stored in the table data compared the case where processing
In redis caching, detection node is waited to obtain data.Node serve mainly monitors answer redis buffer queue, handles in real time
Message content.Real-time Feedback processing result after the completion of processing.
By table data bulk relatively in, need to carry out by unified timing node quantity statistics then compare,
So needing to connect by both sides' table.In the complex data structures for dividing table for point library, it is unified that they are also required to configuration
The Detection task of management, finally summarizes and obtains comparison result.It is excessive that point library divides table mainly to solve single table data volume, to split
Data are dispersed in each table by tables of data.But it can need will be dispersed in each table with big data for us
Data concentrate in our table.
Claims (9)
1. a kind of data consistency detection, which comprises the steps of:
S1: host receives initial data and target data；
S2: host setting detection node；
S3: host extracts initial data and data corresponding in target data according to detection node, carries out data bulk comparison,
Judge data consistency, if quantity is consistent, the corresponding initial data of the detection node is consistent with target data, if quantity is different
It causes, then the corresponding initial data of the detection node and target data are inconsistent, return step S2.
2. a kind of data consistency detection according to claim 1, which is characterized in that in the step S3, according to
Detection node extracts initial data with after data corresponding in target data, and further include step S301: host is according to extraction
Initial data and target data are divided into multiple queues, queue are respectively allocated to by the size of initial data and target data
Idle server carries out data bulk calculating.
3. a kind of data consistency detection according to claim 2, which is characterized in that after step S301, carry out
Step S302: host receives all initial data under a certain detection node compared with the quantity of the queue of target data, then into
Row statistics.
4. a kind of data consistency detection according to claim 3 and system, which is characterized in that in the step
In S301, when server is when handling queue, no longer receiving queue continues to be in idle condition, receive after queue processing is complete
Next queue.
5. a kind of data consistency detection according to claim 3 and system, which is characterized in that the detection node
For the period.
6. a kind of data consistency detection system characterized by comprising
Host sets detection node, extracts initial data and mesh according to detection node for receiving initial data and target data
Mark corresponding data in data；
Server is used for initial data compared with target data carries out data bulk.
7. a kind of data consistency detection system according to claim 1, which is characterized in that the host is also used to basis
Initial data and target data are divided into multiple queues, queue are distinguished by the initial data of extraction and the size of target data
It distributes to idle server and carries out data bulk calculating.
8. a kind of data consistency detection system according to claim 1, which is characterized in that host receives a certain detection section
All initial data under point are compared with the quantity of the queue of target data, then are counted.
9. a kind of data consistency detection system according to claim 1, which is characterized in that server is in processing queue
When, no longer receiving queue continues to be in idle condition, receives next queue after queue processing is complete.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910100217.2A CN109828979A (en) | 2019-01-31 | 2019-01-31 | A kind of data consistency detection and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910100217.2A CN109828979A (en) | 2019-01-31 | 2019-01-31 | A kind of data consistency detection and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109828979A true CN109828979A (en) | 2019-05-31 |
Family
ID=66862035
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910100217.2A Pending CN109828979A (en) | 2019-01-31 | 2019-01-31 | A kind of data consistency detection and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109828979A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111666286A (en) * | 2020-05-21 | 2020-09-15 | 微民保险代理有限公司 | Method and device for detecting sub-warehouse and sub-table, computer equipment and storage medium |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101079683A (en) * | 2007-06-27 | 2007-11-28 | ***通信集团四川有限公司 | Data consistency processing method |
| US8498967B1 (en) * | 2007-01-30 | 2013-07-30 | American Megatrends, Inc. | Two-node high availability cluster storage solution using an intelligent initiator to avoid split brain syndrome |
| CN104036029A (en) * | 2014-06-27 | 2014-09-10 | 深圳市腾讯计算机***有限公司 | Big data consistency comparison method and system |
| CN104503845A (en) * | 2015-01-14 | 2015-04-08 | 北京邮电大学 | Task distributing method and system |
| CN105468718A (en) * | 2015-11-18 | 2016-04-06 | 腾讯科技（深圳）有限公司 | Data consistency processing method, device and system |
| CN106341454A (en) * | 2016-08-23 | 2017-01-18 | 世纪龙信息网络有限责任公司 | Across-room multiple-active distributed database management system and across-room multiple-active distributed database management method |
| CN106789095A (en) * | 2017-03-30 | 2017-05-31 | 腾讯科技（深圳）有限公司 | Distributed system and message treatment method |
| CN108280080A (en) * | 2017-01-06 | 2018-07-13 | 阿里巴巴集团控股有限公司 | A kind of method of data synchronization, device and electronic equipment |
| CN108833503A (en) * | 2018-05-29 | 2018-11-16 | 华南理工大学 | A kind of Redis cluster method based on ZooKeeper |
-
2019
- 2019-01-31 CN CN201910100217.2A patent/CN109828979A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8498967B1 (en) * | 2007-01-30 | 2013-07-30 | American Megatrends, Inc. | Two-node high availability cluster storage solution using an intelligent initiator to avoid split brain syndrome |
| CN101079683A (en) * | 2007-06-27 | 2007-11-28 | ***通信集团四川有限公司 | Data consistency processing method |
| CN104036029A (en) * | 2014-06-27 | 2014-09-10 | 深圳市腾讯计算机***有限公司 | Big data consistency comparison method and system |
| CN104503845A (en) * | 2015-01-14 | 2015-04-08 | 北京邮电大学 | Task distributing method and system |
| CN105468718A (en) * | 2015-11-18 | 2016-04-06 | 腾讯科技（深圳）有限公司 | Data consistency processing method, device and system |
| CN106341454A (en) * | 2016-08-23 | 2017-01-18 | 世纪龙信息网络有限责任公司 | Across-room multiple-active distributed database management system and across-room multiple-active distributed database management method |
| CN108280080A (en) * | 2017-01-06 | 2018-07-13 | 阿里巴巴集团控股有限公司 | A kind of method of data synchronization, device and electronic equipment |
| CN106789095A (en) * | 2017-03-30 | 2017-05-31 | 腾讯科技（深圳）有限公司 | Distributed system and message treatment method |
| CN108833503A (en) * | 2018-05-29 | 2018-11-16 | 华南理工大学 | A kind of Redis cluster method based on ZooKeeper |
Non-Patent Citations (1)
| Title |
|---|
| 青岛英谷教育科技股份有限公司: "《云计算与大数据概论》", 31 October 2017 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111666286A (en) * | 2020-05-21 | 2020-09-15 | 微民保险代理有限公司 | Method and device for detecting sub-warehouse and sub-table, computer equipment and storage medium |
| CN111666286B (en) * | 2020-05-21 | 2023-06-30 | 微民保险代理有限公司 | Method, device, computer equipment and storage medium for detecting database and table |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10891177B2 (en) | Message management method and device, and storage medium | |
| US10701012B2 (en) | Method, apparatus and system for customer service information forwarding | |
| EP4234060A1 (en) | Service processing method and apparatus, and storage medium | |
| US9286316B2 (en) | Enterprise level data collection systems and methodologies | |
| CN108712426B (en) | Crawler identification method and system based on user behavior buried points | |
| CN112860695B (en) | Monitoring data query method, device, equipment, storage medium and program product | |
| CN105577801B (en) | A kind of business accelerating method and device | |
| WO2020192649A1 (en) | Data center management system | |
| CN111813573B (en) | Communication method of management platform and robot software and related equipment thereof | |
| CN110493065A (en) | The alarm association degree analysis method and system of a kind of cloud center O&M | |
| CN110708256A (en) | CDN scheduling method, device, network equipment and storage medium | |
| CN110147470B (en) | Cross-machine-room data comparison system and method | |
| CN114567519B (en) | Method and device for managing instruction messages of multiple intelligent devices in multithreading parallel | |
| US20200004785A1 (en) | Automatic grouping based on user behavior | |
| CN109828979A (en) | A kind of data consistency detection and system | |
| CN105007200B (en) | The analysis method and system of network packet | |
| CN105162837B (en) | The method and system of I/O throughputs are promoted under mass data storage environment | |
| CN106487653B (en) | Message processing method and server | |
| CN110909072B (en) | Data table establishment method, device and equipment | |
| US20230063599A1 (en) | Edge computing network, data transmission method and apparatus, device and storage medium | |
| CN116521335A (en) | Distributed task scheduling method and system for inclined image model production | |
| KR102464688B1 (en) | Method and apparatus for detrmining event level of monitoring result | |
| CN110647448A (en) | Mobile application operation log data real-time analysis method, server and system | |
| CN105357317A (en) | Data uploading method and system based on multi-client polling queuing | |
| CN109542855A (en) | A kind of method and system of document sharing and document control |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190531 |