CN111563088A - Data consistency detection method and device - Google Patents
Data consistency detection method and device Download PDFInfo
- Publication number
- CN111563088A CN111563088A CN202010310383.8A CN202010310383A CN111563088A CN 111563088 A CN111563088 A CN 111563088A CN 202010310383 A CN202010310383 A CN 202010310383A CN 111563088 A CN111563088 A CN 111563088A
- Authority
- CN
- China
- Prior art keywords
- detected
- data
- data stream
- service
- detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 123
- 238000006243 chemical reaction Methods 0.000 claims abstract description 43
- 238000000034 method Methods 0.000 claims abstract description 30
- 238000012545 processing Methods 0.000 claims abstract description 25
- 238000005192 partition Methods 0.000 claims description 23
- 238000013480 data collection Methods 0.000 claims description 4
- 238000009825 accumulation Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000010223 real-time analysis Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data consistency detection method and a device, firstly, continuously generated service data streams are collected, then the service data streams are converted into data streams to be detected according to a preset conversion rule, if a group of mutually matched data streams to be detected can be obtained within a preset time, consistency detection is carried out on the mutually matched data streams to be detected according to a preset detection rule, and a detection result is obtained; if the detection result is that the data are inconsistent, sending alarm information; or if a matched group of data streams to be detected is not obtained within the preset time, sending alarm information. The scheme of the invention can process a plurality of groups of data streams to be detected in real time in parallel, and set the preset time, if the matched data streams to be detected can be acquired within the preset time, the acquisition operation is not overtime, and the consistency detection is carried out under the condition of not overtime, so that the trouble of data acquisition overtime can be quickly eliminated, and the processing efficiency is improved.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a data consistency detection method and device.
Background
The large distributed system comprises a plurality of subsystems, and under the normal condition, data among the subsystems in one distributed system are the same. However, if some subsystems are damaged in hardware, some subsystems are upgraded in software, or some subsystems are delayed in network, the problem of data inconsistency among subsystems can be caused.
Although the current method can detect whether data among subsystems of a distributed system are consistent, under the condition of rapid development of service, the system is iterated frequently, and if the problem of detecting data in a service flow can cause the problems of complicated service logic, large workload, increased processing delay and the like. In addition, the data volume in a large distributed system is often increased rapidly, the data problem is difficult to detect in a limited time by the current method, and a large amount of data is accumulated after a detection task is wrong, so that the data processing speed is slower than the data generation speed.
Therefore, how to quickly process the detection of data consistency in a large-scale distributed system and reduce the processing workload becomes a problem to be solved urgently by those skilled in the art.
Disclosure of Invention
The invention provides a data consistency detection method and a data consistency detection device, which are used for solving the problems of large workload and low processing speed when data in a large-scale distributed system is detected by the conventional data consistency detection method.
In one aspect, the present invention provides a method for detecting data consistency, including:
collecting a service data stream, wherein the service data stream consists of a plurality of fields, the content of each field is different, the service data stream is continuously generated by a data source, and the data source is from a plurality of different service systems;
converting the service data stream into a data stream to be detected according to a preset conversion rule, and performing conversion operation once every time one service data stream is collected, wherein the data stream to be detected consists of a plurality of fields, and the content of each field is different from each other;
if a group of mutually matched data streams to be detected can be obtained within a preset time, carrying out consistency detection on the mutually matched data streams to be detected according to a preset detection rule to obtain a detection result;
and if the detection result is that the data are inconsistent, sending alarm information.
With reference to the first aspect, in an implementation manner of the first aspect, the step of converting the service data stream into the data stream to be detected according to a preset conversion rule includes:
grouping fields in a service data stream according to a preset conversion rule;
and forming a data stream to be detected by using the fields belonging to the same group, wherein each service data stream can be converted into at least one data stream to be detected.
With reference to the first aspect, in an implementation manner of the first aspect, if a group of mutually matched data streams to be detected can be obtained within a preset time, consistency detection is performed on the mutually matched data streams to be detected according to a preset detection rule, and the step of obtaining a detection result includes:
after obtaining a data stream to be detected, judging whether another data stream to be detected matched with the data stream to be detected can be obtained within a preset time;
and if so, detecting whether the data of the corresponding fields in the matched data stream to be detected are consistent or not according to a preset detection rule.
With reference to the first aspect, in an implementation manner of the first aspect, after the converting the service data stream into the data stream to be detected according to a preset conversion rule, the method further includes:
one or more fields in the data stream to be detected are designated as the key value of the data stream to be detected;
and comparing the key values of all the data streams to be detected, and taking a plurality of data streams to be detected with the same key value as a matched group of data streams to be detected.
With reference to the first aspect, in an implementation manner of the first aspect, each service data stream is stored in a topic of the message queue, and the multiple data streams to be detected converted from each service data stream are respectively and correspondingly stored in multiple partitions of the topic.
In a second aspect, the present invention provides a data consistency detection apparatus, including:
the data collection module is used for collecting service data flow, the service data flow consists of a plurality of fields, the content of each field is different, the service data flow is continuously generated by a data source, and the data source is from a plurality of different service systems;
the data conversion module is used for converting the service data stream into a data stream to be detected according to a preset conversion rule and performing conversion operation once every time one service data stream is collected, wherein the data stream to be detected consists of a plurality of fields, and the content of each field is different;
the detection module is used for carrying out consistency detection on a group of mutually matched data streams to be detected according to a preset detection rule if the group of mutually matched data streams to be detected can be obtained within a preset time, so as to obtain a detection result;
and the alarm module is used for sending alarm information if the detection result is that the data are inconsistent.
With reference to the second aspect, in an implementation manner of the second aspect, the data conversion module includes:
the data grouping unit is used for grouping each field in a service data stream according to a preset conversion rule;
and the data stream forming unit is used for forming a data stream to be detected by using the fields belonging to the same group, and each service data stream can be converted into at least one data stream to be detected.
With reference to the second aspect, in an implementation manner of the second aspect, the detection module further includes:
the device comprises a to-be-detected data stream acquiring unit, a to-be-detected data stream acquiring unit and a data processing unit, wherein the to-be-detected data stream acquiring unit is used for judging whether another to-be-detected data stream matched with the to-be-detected data stream can be acquired within preset time after one to-be-detected data stream is acquired;
and the data detection unit is used for detecting whether the data of the corresponding fields in the matched data stream to be detected are consistent or not according to a preset detection rule if the other data stream to be detected matched with the data stream to be detected can be obtained within a preset time.
With reference to the second aspect, in an implementation manner of the second aspect, the apparatus further includes:
a key value determining module, configured to designate one or more fields in the data stream to be detected as a key value of the data stream to be detected;
and the data matching module is used for comparing the key values of all the data streams to be detected and taking a plurality of data streams to be detected with the same key values as a matched group of data streams to be detected.
With reference to the second aspect, in an implementation manner of the second aspect, each service data stream is stored in a topic of the message queue, and the multiple data streams to be detected converted from each service data stream are respectively and correspondingly stored in multiple partitions of the topic.
According to the technical scheme, the invention provides a data consistency detection method and a device, firstly, continuously generated service data streams are collected, then the service data streams are converted into data streams to be detected according to a preset conversion rule, if a group of mutually matched data streams to be detected can be obtained within a preset time, consistency detection is carried out on the mutually matched data streams to be detected according to a preset detection rule, and a detection result is obtained; if the detection result is that the data are inconsistent, sending alarm information; or if a matched group of data streams to be detected is not obtained within the preset time, sending alarm information. The scheme of the invention can process a plurality of groups of data streams to be detected in real time in parallel, and set the preset time, if the matched data streams to be detected can be acquired within the preset time, the acquisition operation is not overtime, and the consistency detection is carried out under the condition of not overtime, so that the trouble of data acquisition overtime can be quickly eliminated, and the processing efficiency is improved.
Drawings
In order to more clearly illustrate the technical solution of the present invention, the drawings needed to be used in the embodiment will be briefly described below, and it is obvious to those skilled in the art that other drawings can be obtained according to the drawings without any creative effort.
Fig. 1 is a flowchart of a data consistency detection method according to an embodiment of the present invention;
fig. 2 is a flowchart of a method for converting a service data stream into a data stream to be detected according to an embodiment of the present invention;
FIG. 3 is a flowchart of a method for consistency detection according to an embodiment of the present invention;
fig. 4 is a flowchart of a method for matching data streams to be detected according to an embodiment of the present invention;
fig. 5 is a block diagram of a data consistency detection apparatus according to an embodiment of the present invention.
Detailed Description
In order to make those skilled in the art better understand the technical solution of the present invention, the technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.
Fig. 1 is a flowchart of a data consistency detection method according to an embodiment of the present invention. As shown in fig. 1, the data consistency detection method provided in the embodiment of the present invention includes:
step S101, collecting service data flow, wherein the service data flow is composed of a plurality of fields, the content of each field is different, the service data flow is continuously generated by a data source, and the data source is from a plurality of different service systems.
In the embodiment of the invention, the data source comprises various service systems in a large-scale distributed system, the service systems can generate service data streams in real time when performing service operation, and the data in the service data streams are different according to different service contents. For example, in an e-commerce transaction service system, a service data stream generated by a transaction may include a plurality of data streams such as an order settlement data stream, a payment deduction data stream, and a revenue and account entry data stream, wherein the order settlement data stream includes a plurality of fields, such as an order settlement ID, a payment account ID, a revenue account ID, an order total amount, and the like, and a certain ID number or a specific amount is stored in different fields.
Step S102, the business data flow is converted into a data flow to be detected according to a preset conversion rule, and conversion operation is carried out once every time one business data flow is collected, wherein the data flow to be detected is composed of a plurality of fields, and the content of each field is different.
Still taking the above order settlement data stream as an example, the order settlement data stream may be converted into two data streams to be detected according to a preset conversion rule, which are an order expenditure data stream and an order income data stream respectively, where the order expenditure data stream includes fields such as an order settlement ID, an expenditure party account ID, and an order total, and the order income data stream includes fields such as an order settlement ID, an income party account ID, and an order total. Therefore, the preset conversion rule in the embodiment of the present invention may also be specifically set as a splitting rule, that is, one service data stream is split into multiple data streams to be detected, and since the data content included in one service data stream is of multiple types and quantities, only by splitting and converting the data content into a data stream to be detected of a single type, consistency detection can be effectively performed on one or some data in the specific data streams to be detected.
Since the data source generates data continuously in real time, the service data stream belongs to a stream data, and in the embodiment of the present invention, the processing of the stream data, such as the above conversion operation, should belong to stream computation, that is, a conversion operation is performed once when a service data stream is collected, and this operation is performed in real time. The accumulation of a large amount of service data is avoided, and the problems of low detection speed and inaccurate detection result of a detection device or a detection system caused by centralized processing of a large amount of service data are also avoided.
And because the data source for generating the service data stream comes from different service systems, the method in the embodiment of the invention can also process the service data stream and the data stream to be detected between any different service systems, and has high expansibility.
Step S103, if a group of mutually matched data streams to be detected can be obtained within a preset time, carrying out consistency detection on the mutually matched data streams to be detected according to a preset detection rule to obtain a detection result.
At present, for a method for detecting data consistency in a large-scale distributed system, when detection is performed, two matched data streams to be detected need to be associated first, and because data stream acquisition has a certain delay, a situation that one data stream arrives first and the other data stream does not arrive yet always occurs. For this situation, some processing methods wait for a period of time after one data stream arrives, and send a message that the association is successful if another data stream arrives. However, if another data stream is not reached during the waiting time, and the waiting time is exceeded, the processing modes can not send the result of association failure. Therefore, in the embodiment of the present invention, a preset time is set to determine whether the acquisition of the matched data stream to be detected is overtime. If a group of matched data streams to be detected can be obtained within the preset time, and the time is not overtime, the data consistency can be continuously detected; if the time is out, a notification of obtaining the time-out is sent to inform a technician of resetting the relevant parameters of the detection system or the detection device, wherein the relevant parameters specifically include preset time, preset conversion rules and the like, and the technician can also be informed of checking and reconfiguring the large-scale distributed system and the like.
The consistency detection in the embodiment of the invention refers to detecting the data in one field or a plurality of fields in the matched data stream to be detected according to a preset detection rule and judging whether the data are consistent. The preset detection rule can be specifically set in advance according to the service content.
And step S104, if the detection result is that the data are inconsistent, sending alarm information.
If the order expenditure data stream is matched with the account deduction data stream and the preset detection rule is detection amount data, the order settlement ID in the order expenditure data stream is the same as the business flow ID in the matched account deduction data stream, and whether the total amount of the order in the order expenditure data stream is the same as the expenditure amount in the account deduction data stream or not needs to be compared. If the data flow is the same as the data flow, the matched order expenditure data flow is consistent with the data flow of the account deduction data, and if the data flow is not the same as the data flow, the matched order expenditure data flow is inconsistent with the data flow of the account deduction data, and an alarm message is sent to a detection system or a detection device to prompt a technician to perform corresponding processing.
According to the scheme, the data consistency detection method provided by the embodiment of the invention can process a plurality of groups of data streams to be detected in real time in parallel, prevent the accumulation of service data and avoid the problems of large workload and low working efficiency caused by centralized processing of the service data. In addition, the method also sets the preset time, if the matched data stream to be detected can be acquired within the preset time, the acquisition operation is not overtime, and the consistency detection is carried out under the condition of not overtime, so that the trouble of overtime data acquisition can be quickly eliminated, and the processing efficiency is improved.
Fig. 2 is a flowchart of a method for converting a service data stream into a data stream to be detected according to an embodiment of the present invention. As shown in fig. 2, in the embodiment of the present invention, the step of converting the service data stream into the data stream to be detected according to the preset conversion rule includes:
step S201, grouping each field in a service data stream according to a preset conversion rule.
As described above, if a service data stream includes a plurality of fields, such as an order settlement ID, an account ID of an expenditure party, an account ID of an income party, and a total amount of orders, and these fields represent two services of expenditure and income, in order to facilitate final data consistency detection, it is necessary to simplify a service data stream including a plurality of services, so as to split data streams to be detected corresponding to different services. The preset conversion rule can be set according to specific service types, for example, set as a payment service and an income service, and at this time, the service data stream is converted into two groups of data which respectively correspond to an order payment data stream and an order income data stream, wherein the order payment data stream includes fields such as an order settlement ID, a payment party account ID and an order total amount, and the order income data stream includes fields such as an order settlement ID, an income party account ID and an order total amount.
Step S202, a data stream to be detected is formed by using the fields belonging to the same group, and each service data stream can be converted into at least one data stream to be detected.
Fig. 3 is a flowchart of a method for consistency detection according to an embodiment of the present invention. As shown in fig. 3, in the embodiment of the present invention, if a group of mutually matched data streams to be detected can be obtained within a preset time, consistency detection is performed on the mutually matched data streams to be detected according to a preset detection rule, and the step of obtaining a detection result includes:
step S301, after obtaining a data stream to be detected, determining whether another data stream to be detected matching the data stream to be detected can be obtained within a preset time.
Taking the above matched order expenditure data stream and account deduction data stream as an example, if the order expenditure data stream is received first, it needs to be determined whether the account deduction data stream is obtained within a preset time. In the embodiment of the present invention, there may be a plurality of mutually matched data streams to be detected, and when there may be a plurality of mutually matched data streams to be detected, it is further required to determine whether all other data streams to be detected matched with one of the data streams to be detected can be obtained within a preset time after one of the data streams to be detected is obtained first.
And step S302, if yes, detecting whether the data of the corresponding fields in the matched data stream to be detected are consistent or not according to a preset detection rule.
If the matched data to be detected can be obtained within the preset time, it indicates that the obtaining operation is not overtime, consistency detection can be continued, and if the operation is overtime, an alarm message should be sent to the detection system or the detection device to prompt a technician that the matched data to be detected is not obtained.
Fig. 4 is a flowchart of a method for matching data streams to be detected according to an embodiment of the present invention. As shown in fig. 4, in the embodiment of the present invention, after converting the service data stream into the data stream to be detected according to the preset conversion rule, the method further includes:
step S401, designating one or more fields in the data stream to be detected as a key value of the data stream to be detected.
Step S402, comparing the key values of all the data streams to be detected, and using a plurality of data streams to be detected with the same key values as a matched group of data streams to be detected.
In the embodiment of the present invention, all the above processes of data stream acquisition, data stream conversion, data stream matching, and data stream consistency detection are implemented in the message middleware. A message queue exists in the message middleware, an acquired service data stream is stored in a topic of the message queue, and each topic can be configured with a plurality of partitions. And when one service data stream is converted into different data streams to be detected, corresponding different key values are calculated, the hash value of the data stream to be detected is calculated according to the key values, then modular operation is carried out by utilizing the hash value and the partition number, and each data stream to be detected is distributed to each partition by utilizing the operation result. partition can be divided into different compute nodes so that parallel processing can be achieved. The data streams to be detected in the single partition are in a time guaranteed sequence; in addition, a task having no more than the number of partitions needs to be created, and data in partitions having the same key value will be processed by one task. For example, two partitions are configured, the data stream data to be detected is distributed into the two partitions according to the Key value, and one or two tasks can be configured to process the data on the two partitions.
The partial numbers corresponding to the data streams to be detected with the same key value are the same, but each partial exists in different topic. During consistency detection, if a group of data streams to be detected corresponds to two partitions, and the two partitions exist in topicA and topicB respectively, the task needs to process data in the two topiC. When one or more service data streams arrive in the topicA and the topicB respectively in real time, firstly converting the service data streams into data streams to be detected and storing the data streams to be detected in each partition, then storing the partition data to be detected in the topicA into a local state storage module storeA by a set detection Task module, storing the partition data to be detected in the topicB into a local state storage module storeB, and trying to match the partition data with the same key value from the storeB according to the key value of the partition data in the storeA within preset time. If the matching is achieved, whether the data of the corresponding fields in the two matched partition data are the same or not is further detected. And after the consistency detection is finished, deleting the data cached in the storeA and the storeB.
The local state storage module may be configured as a memory or other persisteable component. When data loss is allowed, the data loss can be configured as a memory, and the data consistency detection result is divided into the following types: and (3) deleting data in the topicA or the topicB, enabling the data in the topicA and the topicB to be consistent, or enabling the data in the topicA and the data in the topicB to be inconsistent, and storing a consistency detection result in a real-time analysis database so as to carry out alarm processing or other processing according to the result.
The data collection module stores the progress of synchronous data, and collection can be continued from the last collected progress after the collection task fails. After the data is stored in the message middleware, the data can be persisted in the hard disk, and the data consistency detection task can be recalculated and detected according to the stored data after failing. In the data processing process, the state of the data consistency detection task is subjected to incremental snapshot and persisted to a hard disk periodically, and all data do not need to be replayed after the data consistency detection task fails, so that the data consistency detection task can be quickly recovered.
Various business data streams can be uniformly collected into the message middleware through a database, a log synchronization tool and the like. The preset conversion rule of the service data stream and the preset detection rule of the data consistency detection are configured in a SQL (Structured Query Language) similar mode, and the use threshold is reduced. The alarm rule is simple, and usually alarms for detection results of data loss or data inconsistency.
The setting of the preset conversion rule and the preset detection rule can be realized by expanding the SQL API of KSQL or Flink of Kafka Stream.
The message middleware uses Kafka; for collecting Mysql service data flow to the message middleware, the method can be realized by using a binlog incremental subscription & consumption component such as Canal; for collecting log type service data flow to the message middleware, a data processing pipeline such as logstack can be used for implementation. The data conversion module and the detection module can realize specific functions by using a mainstream real-time computing framework such as Kafka Stream, Flink and the like.
It can be known from the above solutions that the embodiments of the present invention provide a data consistency detection method, which can utilize the function of a message middleware to achieve the purpose of processing multiple groups of data streams to be detected in parallel in real time, and can prevent business data from being stacked, thereby avoiding the problems of large workload and low work efficiency caused by centralized processing of business data.
In addition, the method also sets the preset time, if the matched data stream to be detected can be acquired within the preset time, the acquisition operation is not overtime, and the consistency detection is carried out under the condition of not overtime, so that the trouble of overtime data acquisition can be quickly eliminated, and the processing efficiency is improved.
Fig. 5 is a block diagram of a data consistency detection apparatus according to an embodiment of the present invention. As shown in fig. 5, the data consistency detection apparatus provided in the embodiment of the present invention includes:
a data collection module 501, configured to collect a service data stream, where the service data stream is composed of a plurality of fields, each of the fields has a different content, and the service data stream is continuously generated by a data source, where the data source is from a plurality of different service systems; a data conversion module 502, configured to convert the service data stream into a data stream to be detected according to a preset conversion rule, and perform a conversion operation once every time a service data stream is collected, where the data stream to be detected is composed of a plurality of fields, and the content of each field is different from each other; the detection module 503 is configured to, if a group of mutually matched data streams to be detected can be obtained within a preset time, perform consistency detection on the mutually matched data streams to be detected according to a preset detection rule to obtain a detection result; and an alarm module 504, configured to send alarm information if the detection result is that the data is inconsistent.
The data conversion module includes: the data grouping unit is used for grouping each field in a service data stream according to a preset conversion rule; and the data stream forming unit is used for forming a data stream to be detected by using the fields belonging to the same group, and each service data stream can be converted into at least one data stream to be detected.
The detection module further comprises: the device comprises a to-be-detected data stream acquiring unit, a to-be-detected data stream acquiring unit and a data processing unit, wherein the to-be-detected data stream acquiring unit is used for judging whether another to-be-detected data stream matched with the to-be-detected data stream can be acquired within preset time after one to-be-detected data stream is acquired; and the data detection unit is used for detecting whether the data of the corresponding fields in the matched data stream to be detected are consistent or not according to a preset detection rule if the other data stream to be detected matched with the data stream to be detected can be obtained within a preset time.
The device further comprises: a key value determining module, configured to designate one or more fields in the data stream to be detected as a key value of the data stream to be detected; and the data matching module is used for comparing the key values of all the data streams to be detected and taking a plurality of data streams to be detected with the same key values as a matched group of data streams to be detected.
Each service data stream is stored in a topic of the message queue, and the plurality of data streams to be detected converted from each service data stream are respectively and correspondingly stored in a plurality of partitions of the topic.
As can be seen from the above technical solutions, embodiments of the present invention provide a data consistency detection method and apparatus, first collect a continuously generated service data stream, then convert the service data stream into a data stream to be detected according to a preset conversion rule, and if a group of mutually matched data streams to be detected can be obtained within a preset time, perform consistency detection on the mutually matched data streams to be detected according to a preset detection rule, so as to obtain a detection result; if the detection result is that the data are inconsistent, sending alarm information; or if a matched group of data streams to be detected is not obtained within the preset time, sending alarm information. The scheme of the invention can process a plurality of groups of data streams to be detected in real time in parallel, and set the preset time, if the matched data streams to be detected can be acquired within the preset time, the acquisition operation is not overtime, and the consistency detection is carried out under the condition of not overtime, so that the trouble of data acquisition overtime can be quickly eliminated, and the processing efficiency is improved.
The present invention needs to be deployed in a general-purpose computer server that can be used to collect and detect a wide variety of general-purpose or special-purpose computing system data.
The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the invention and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention being indicated by the following claims.
It will be understood that the invention is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the invention is limited only by the appended claims.
Claims (10)
1. A data consistency detection method is characterized by comprising the following steps:
collecting a service data stream, wherein the service data stream consists of a plurality of fields, the content of each field is different, the service data stream is continuously generated by a data source, and the data source is from a plurality of different service systems;
converting the service data stream into a data stream to be detected according to a preset conversion rule, and performing conversion operation once every time one service data stream is collected, wherein the data stream to be detected consists of a plurality of fields, and the content of each field is different from each other;
if a group of mutually matched data streams to be detected can be obtained within a preset time, carrying out consistency detection on the mutually matched data streams to be detected according to a preset detection rule to obtain a detection result;
and if the detection result is that the data are inconsistent, sending alarm information.
2. The method according to claim 1, wherein the step of converting the service data stream into the data stream to be detected according to a preset conversion rule comprises:
grouping fields in a service data stream according to a preset conversion rule;
and forming a data stream to be detected by using the field data belonging to the same group, wherein each service data stream can be converted into at least one data stream to be detected.
3. The method according to claim 1, wherein if a group of mutually matched data streams to be detected can be obtained within a preset time, consistency detection is performed on the mutually matched data streams to be detected according to a preset detection rule, and the step of obtaining a detection result comprises:
after obtaining a data stream to be detected, judging whether another data stream to be detected matched with the data stream to be detected can be obtained within a preset time;
and if so, detecting whether the data of the corresponding fields in the matched data stream to be detected are consistent or not according to a preset detection rule.
4. The method according to claim 3, further comprising, after converting the service data stream into the data stream to be detected according to a preset conversion rule:
one or more fields in the data stream to be detected are designated as the key value of the data stream to be detected;
and comparing the key values of all the data streams to be detected, and taking a plurality of data streams to be detected with the same key value as a matched group of data streams to be detected.
5. The method according to claim 4, wherein each of the service data streams is stored in a topic of the message queue, and the plurality of data streams to be detected converted from each of the service data streams are respectively and correspondingly stored in a plurality of partitions of the topic.
6. A data consistency detection apparatus, comprising:
the data collection module is used for collecting service data flow, the service data flow consists of a plurality of fields, the content of each field is different, the service data flow is continuously generated by a data source, and the data source is from a plurality of different service systems;
the data conversion module is used for converting the service data stream into a data stream to be detected according to a preset conversion rule and performing conversion operation once every time one service data stream is collected, wherein the data stream to be detected consists of a plurality of fields, and the content of each field is different;
the detection module is used for carrying out consistency detection on a group of mutually matched data streams to be detected according to a preset detection rule if the group of mutually matched data streams to be detected can be obtained within a preset time, so as to obtain a detection result;
and the alarm module is used for sending alarm information if the detection result is that the data are inconsistent.
7. The apparatus of claim 6, wherein the data conversion module comprises:
the data grouping unit is used for grouping each field in a service data stream according to a preset conversion rule;
and the data stream forming unit is used for forming a data stream to be detected by using the fields belonging to the same group, and each service data stream can be converted into at least one data stream to be detected.
8. The apparatus of claim 6, wherein the detection module further comprises:
the device comprises a to-be-detected data stream acquiring unit, a to-be-detected data stream acquiring unit and a data processing unit, wherein the to-be-detected data stream acquiring unit is used for judging whether another to-be-detected data stream matched with the to-be-detected data stream can be acquired within preset time after one to-be-detected data stream is acquired;
and the data detection unit is used for detecting whether the data of the corresponding fields in the matched data stream to be detected are consistent or not according to a preset detection rule if the other data stream to be detected matched with the data stream to be detected can be obtained within a preset time.
9. The apparatus of claim 8, further comprising:
a key value determining module, configured to designate one or more fields in the data stream to be detected as a key value of the data stream to be detected;
and the data matching module is used for comparing the key values of all the data streams to be detected and taking a plurality of data streams to be detected with the same key values as a matched group of data streams to be detected.
10. The apparatus according to claim 9, wherein each of the service data streams is stored in a topic of a message queue, and the data streams to be detected converted from each of the service data streams are respectively and correspondingly stored in partitions of the topic.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010310383.8A CN111563088A (en) | 2020-04-20 | 2020-04-20 | Data consistency detection method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010310383.8A CN111563088A (en) | 2020-04-20 | 2020-04-20 | Data consistency detection method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111563088A true CN111563088A (en) | 2020-08-21 |
Family
ID=72071697
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010310383.8A Pending CN111563088A (en) | 2020-04-20 | 2020-04-20 | Data consistency detection method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111563088A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105447046A (en) * | 2014-09-02 | 2016-03-30 | 阿里巴巴集团控股有限公司 | Distributed system data consistency processing method, device and system |
| CN105989044A (en) * | 2015-02-04 | 2016-10-05 | 阿里巴巴集团控股有限公司 | Database verification method and system |
| CN105989089A (en) * | 2015-02-12 | 2016-10-05 | 阿里巴巴集团控股有限公司 | Data comparison method and device |
| CN107644077A (en) * | 2017-09-19 | 2018-01-30 | 杭州贝购科技有限公司 | Data consistency monitoring method, computer equipment and storage medium |
| CN108156141A (en) * | 2017-12-14 | 2018-06-12 | 北京奇艺世纪科技有限公司 | A kind of real time data recognition methods, device and electronic equipment |
| CN108733662A (en) * | 2017-04-13 | 2018-11-02 | 北京京东尚科信息技术有限公司 | Method, apparatus, electronic equipment and the readable storage medium storing program for executing of comparison of data consistency |
| CN110765091A (en) * | 2019-09-09 | 2020-02-07 | 上海陆家嘴国际金融资产交易市场股份有限公司 | Account checking method and system |
-
2020
- 2020-04-20 CN CN202010310383.8A patent/CN111563088A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105447046A (en) * | 2014-09-02 | 2016-03-30 | 阿里巴巴集团控股有限公司 | Distributed system data consistency processing method, device and system |
| CN105989044A (en) * | 2015-02-04 | 2016-10-05 | 阿里巴巴集团控股有限公司 | Database verification method and system |
| CN105989089A (en) * | 2015-02-12 | 2016-10-05 | 阿里巴巴集团控股有限公司 | Data comparison method and device |
| CN108733662A (en) * | 2017-04-13 | 2018-11-02 | 北京京东尚科信息技术有限公司 | Method, apparatus, electronic equipment and the readable storage medium storing program for executing of comparison of data consistency |
| CN107644077A (en) * | 2017-09-19 | 2018-01-30 | 杭州贝购科技有限公司 | Data consistency monitoring method, computer equipment and storage medium |
| CN108156141A (en) * | 2017-12-14 | 2018-06-12 | 北京奇艺世纪科技有限公司 | A kind of real time data recognition methods, device and electronic equipment |
| CN110765091A (en) * | 2019-09-09 | 2020-02-07 | 上海陆家嘴国际金融资产交易市场股份有限公司 | Account checking method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10747591B2 (en) | Endpoint process state collector | |
| US6584491B1 (en) | Arrangement for monitoring a progress of a message flowing through a distributed multiprocess system | |
| WO2019006654A1 (en) | Financial self-service equipment maintenance dispatch generation method, hand-held terminal and electronic device | |
| CN106815254B (en) | Data processing method and device | |
| CN110928851B (en) | Method, device and equipment for processing log information and storage medium | |
| CN110647447B (en) | Abnormal instance detection method, device, equipment and medium for distributed system | |
| CN114363042B (en) | Log analysis method, device, equipment and readable storage medium | |
| CN108140035B (en) | Database replication method and device for distributed system | |
| CN113760677A (en) | Abnormal link analysis method, device, equipment and storage medium | |
| CN111913824A (en) | Method for determining data link fault reason and related equipment | |
| CN105069029A (en) | Real-time ETL (extraction-transformation-loading) system and method | |
| CN113672452A (en) | Method and system for monitoring operation of data acquisition task | |
| CN117271177A (en) | Root cause positioning method and device based on link data, electronic equipment and storage medium | |
| CN112433757A (en) | Method and device for determining interface calling relationship | |
| CN111130882A (en) | Monitoring system and method of network equipment | |
| CN111563088A (en) | Data consistency detection method and device | |
| CN112579552A (en) | Log storage and calling method, device and system | |
| CN115664992A (en) | Network operation data processing method and device, electronic equipment and medium | |
| CN115269519A (en) | Log detection method and device and electronic equipment | |
| CN111813833B (en) | Real-time two-degree communication relation data mining method | |
| CN107766442B (en) | A kind of mass data association rule mining method and system | |
| CN113114612B (en) | Determination method and device for distributed system call chain | |
| CN115033457B (en) | Multi-source data real-time acquisition method and system capable of monitoring and early warning | |
| CN112596887A (en) | Analysis method for batch delay and related device | |
| CN114691639A (en) | Real-time streaming data processing method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: Floor 1, No.81 Xinle Road, high tech Zone, Chengdu, Sichuan 610041 Applicant after: Chengdu Cooper Innovation Technology Co.,Ltd. Address before: Floor 1, No.81 Xinle Road, high tech Zone, Chengdu, Sichuan 610041 Applicant before: Chengdu Cooper blockchain Technology Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200821 |
|
| RJ01 | Rejection of invention patent application after publication |