WO2019178772A1 - Database master-slave block consistency check method, device and system - Google Patents

Database master-slave block consistency check method, device and system Download PDF

Info

Publication number
WO2019178772A1
WO2019178772A1 PCT/CN2018/079861 CN2018079861W WO2019178772A1 WO 2019178772 A1 WO2019178772 A1 WO 2019178772A1 CN 2018079861 W CN2018079861 W CN 2018079861W WO 2019178772 A1 WO2019178772 A1 WO 2019178772A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
block
database
hash tree
hash
Prior art date
Application number
PCT/CN2018/079861
Other languages
French (fr)
Chinese (zh)
Inventor
陈志新
Original Assignee
深圳蓝贝科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳蓝贝科技有限公司 filed Critical 深圳蓝贝科技有限公司
Priority to PCT/CN2018/079861 priority Critical patent/WO2019178772A1/en
Publication of WO2019178772A1 publication Critical patent/WO2019178772A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Definitions

  • the present invention relates to the field of data storage and data verification technologies, and in particular, to a data consistency verification method, apparatus and system, and more particularly to a database master-slave block consistency verification method, apparatus and system.
  • the object of the present application is to overcome the above problems or at least partially solve or alleviate the above problems.
  • a data consistency verification method is provided, which is applied to a primary database, and includes:
  • Hash tree data sending step hashing the block data in the first block set in the primary database, obtaining first hash tree data, and the first hash tree data Sent to the slave database;
  • a verification result receiving step after comparing the first hash tree data and the second hash tree data by the slave database, receiving the verification result sent from the database, wherein the second hash The tree is obtained by hashing the block data in the second block set corresponding to the first block set in the database, and the check result includes: the first block set a block number that is inconsistent with data in the second block set; and
  • a block data sending step transmitting block data corresponding to the block number in the first block set to the slave database according to the block number inconsistent with the data, so as to correspond to the slave block according to the block number
  • the block data updates the block data of the corresponding block in the second block set.
  • the method can quickly calculate the consistency of data between the primary database and the secondary database, and update the data from the database according to the data of the primary database to achieve the function of data verification and data backup.
  • the first hash tree data and the second hash tree data are: constructing a leaf node by using the block data in the first block set and the second block set respectively All the hash data obtained by the tree.
  • the block is obtained by storing data of the primary database or the data of the secondary database in blocks of fixed time units.
  • the block data is an operation log or service data.
  • the operation log is a binary log (Binlog).
  • a data consistency verification method is also provided, which is applied to a slave database, including:
  • a hash tree data receiving step performing hashing on the block data in the first block set in the primary database in the primary database to obtain the first hash tree data, and receiving the First hash tree data;
  • a verification result generating step comparing the first hash tree data with the second hash tree data, generating a verification result and transmitting the verification result to the primary database, wherein the second hash tree is for the And performing a hash operation on the data of the block in the second block set corresponding to the first block set in the database, where the check result includes: the first block set and the first The block number in which the data in the second block set is inconsistent;
  • a block data receiving step receiving, by the primary database, block data corresponding to the block number in the first block set sent according to the block number inconsistent with the data;
  • the data updating step updating the block data of the corresponding block in the second block set according to the block data corresponding to the block number.
  • the device realizes fast consistency check and synchronization of the main database based on the hash tree and the slave database, and can also implement the checksum correction for the silent error data.
  • the block data is an operation log, and preferably, the operation log is a binary log.
  • the data update step includes:
  • Data parsing step parsing an operation log corresponding to the block number
  • Data execution step performing a database operation according to the operation log
  • Log update step update the operation log of the slave database.
  • the device implements a hash tree-based master database and a slave database for fast consistency checksum synchronization.
  • the block data is service data.
  • the data update step includes:
  • Data parsing step parsing the business data corresponding to the block number
  • Data execution step performing a B+Tree (B+Tree) operation or a Skip List operation according to the service data.
  • the device is based on the hash tree of the leaf node data block, and implements fast silent error data consistency check and correction of the primary database and the secondary database.
  • a data consistency verification apparatus applied to a primary database, comprising:
  • a hash tree data sending module configured to perform hash operation on the block data in the first block set in the primary database, to obtain first hash tree data, and to use the first hash Tree data is sent to the slave database;
  • a verification result receiving module configured to receive the verification result sent by the slave database after the first hash tree data and the second hash tree data are compared by the slave database, where The second hash tree is obtained by hashing the block data in the second block set corresponding to the first block set in the database, and the check result includes: a block number in which the data in the first block set and the second block set are inconsistent; and
  • a block data sending module configured to send, according to the block number of the data inconsistency, block data corresponding to the block number in the first block set to the slave database, so as to be based on the database
  • the block data corresponding to the block number updates the block data of the corresponding block in the second block set.
  • the device divides the data of the primary database into blocks, constructs a hash tree, and implements fast consistency check of the primary database and the secondary database based on the hash tree, thereby quickly backing up the data of the primary database to the secondary database.
  • the first hash tree data and the second hash tree data are: constructing a leaf node by using the block data in the first block set and the second block set respectively All the hash data obtained by the tree.
  • the block is obtained by storing data of the primary database or the data of the secondary database in blocks of fixed time units.
  • the block data is an operation log or service data.
  • the operation log is a binary log.
  • a data consistency check device which is applied to a slave database, comprising:
  • a hash tree data receiving module configured to perform hash operation on the block data in the first block set in the primary database in the primary database, to obtain the first hash tree data, and receive the primary The first hash tree data sent by the database;
  • a verification result generating module configured to compare the first hash tree data and the second hash tree data, generate a verification result, and send the verification result to the primary database, where the second hash The tree is obtained by hashing data of the block in the second block set corresponding to the first block set in the database, and the check result includes: the first block a block number that is inconsistent with the data in the set of the second block;
  • a block data receiving module configured to receive block data corresponding to the block number in the first block set sent by the primary database according to the block number inconsistent with the data
  • a data update module configured to update the block data of the corresponding block in the second block set according to the block data corresponding to the block number.
  • the device enables the slave database to quickly obtain inconsistent data from the primary database, thereby enabling backup of the primary database data.
  • the block data is an operation log, and preferably, the operation log is a binary log.
  • the data update module includes:
  • a data parsing module configured to parse an operation log corresponding to the block number
  • a data execution module configured to perform a database operation based on the operation log
  • a log update module configured to update an operation log of the slave database.
  • the block data is service data.
  • the data update module includes:
  • a data parsing module configured to parse the service data corresponding to the block number
  • a data execution module configured to perform a B+ tree operation or a table jump operation according to the service data.
  • a data consistency check system comprising a primary database and at least one secondary database, the primary database and the at least one secondary database being interconnected, wherein:
  • the primary database is configured to perform hash operation on the block data in the first block set in the primary database, obtain first hash tree data, and send the first hash tree data to the From the database;
  • the slave database is configured to compare the first hash tree data with the second hash tree data, generate a verification result, and send the verification result to the primary database, where the second hash tree is a
  • the hash data in the second block set corresponding to the first block set in the database is hashed, and the check result includes: the first block set and the first The block number in which the data in the second block set is inconsistent;
  • the primary database is further configured to send the block data corresponding to the block number in the first block set to the slave database according to the block number of the data inconsistency, so as to be based on the block number from the database.
  • the corresponding block data updates the block data of the corresponding block in the second block set.
  • the system can quickly calculate the consistency of data between the primary database and the secondary database, and achieve the role of data verification and data backup.
  • FIG. 1 is a flow chart of one embodiment of a data consistency check method applied to a primary database in accordance with the present application
  • FIG. 2 is a flow chart of one embodiment of a method of establishing a hash tree
  • FIG. 3 is a flow diagram of one embodiment of a data consistency verification method applied to a slave database in accordance with the present application
  • FIG. 4 is a flow chart of another embodiment of a data consistency check method in accordance with the present application.
  • Figure 5 is a flow diagram of one embodiment of a data update step from a database
  • FIG. 6 is a flow chart of another embodiment of a data consistency check method in accordance with the present application.
  • Figure 7 is a flow diagram of another embodiment of a data update step from a database
  • FIG. 8 is a flow diagram of one embodiment of a data consistency check device applied to a primary database in accordance with the present application
  • FIG. 9 is a flow diagram of one embodiment of a data consistency check device applied to a slave database in accordance with the present application.
  • FIG. 10 is a flow diagram of one embodiment of a data consistency check system in accordance with the present application.
  • a data consistency verification method for application to a primary database.
  • BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 is a flow diagram of one embodiment of a data consistency verification method applied to a primary database in accordance with the present application.
  • the method may include the step S11 hash tree data sending step: hashing the block data in the first block set in the primary database, obtaining the first hash tree data, and the first hash The hash tree data is sent to the slave database.
  • the block is obtained by storing the data of the primary database or the data in the database of the secondary database in blocks of fixed time units.
  • the Master and the Slave may agree to store the operation log of the database in a fixed time unit, for example, by dividing the block by one minute, thereby obtaining a plurality of blocks.
  • the number of blocks in the first block set can be selected according to the number of layers of the hash tree. For example, eight. 2 is a flow chart of one embodiment of a method of establishing a hash tree. The structure of a four-layer hash tree is shown in FIG. The bottom layer, the fourth layer, is the leaf node. The content of the leaf node may be block data.
  • the leaf node of the hash tree includes 8 blocks, which are respectively labeled as D1 block to D8 block. They form the first block set. Each of the two blocks in the first block set is hashed into a group to obtain a hash value of the third layer. For example, the data of the D1 block and the D2 block are hashed to obtain the result N1.
  • the penultimate layer obtains a total of four hash values N1 to N4.
  • the two hashes of the two hash values M1 and M2 are obtained by grouping each of the four hash values into a group.
  • the two hash values of the second layer are hashed again to obtain the hash value Root of the root node.
  • the number of layers of the hash value can be selected according to the computing power and the amount of data of the server.
  • the D1 block to the D8 block may be blocks sorted in chronological order, or may be arranged in other manners. In each layer, any two blocks or hash values may be selected for hashing, and the order of operations shown in FIG. 2 should not be construed as limiting the present application.
  • the first hash tree data is all hash data obtained by constructing a hash tree by using the block data in the first block set as a leaf node. That is to say, the first hash tree data may be the result of connecting the hash values of all layers in a certain order. For example, the hash values of each layer may be spliced in order from the first layer to the third layer, or spliced in the order from the third layer to the first layer. It can be understood that the first hash tree data may also be the result of splicing the results of some of the layers, or may only transmit the hash value of the root node.
  • the method may further include a step S12 verification result receiving step: after comparing the first hash tree data and the second hash tree data from the database, receiving the verification result sent from the database, wherein The second hash tree is obtained by hashing the block data in the second block set corresponding to the first block set in the database, and the check result may include : the block number of the first block set and the second block set inconsistent data.
  • the second hash tree data is all hash data obtained by constructing a hash tree as the leaf node by using the block data in the second block set.
  • data corresponding to the primary database is recorded from the database.
  • the data in the database is divided into the same block, for example, divided into D1' blocks to D8' blocks.
  • the hash value is calculated in the same way to obtain the second hash tree data. Comparing the second hash tree data with the first hash tree data transmitted by the primary database from the database, an inconsistent hash value can be obtained, and the block number of the data inconsistency can be determined by the position of the inconsistent hash value.
  • the block numbers D1 and D2 may be included in the verification result. If the primary database only transmits the data of the root node of each hash tree, if a certain root node data is inconsistent, the verification result may include all the block numbers below the root node.
  • the method may further include the step of transmitting a block data in the step S13: sending the block data corresponding to the block number in the first block set to the slave database according to the block number of the data inconsistency
  • the slave database updates the block data of the corresponding block in the second block set according to the block data corresponding to the block number.
  • the primary database after receiving the inconsistent block numbers D1 and D2, the primary database sends the data of the D1 block and the D2 block in the database to the slave database.
  • the block data D1' and D2' of the corresponding block in the second block set are updated from the database based on the block data corresponding to the block number.
  • the block data is an operation log.
  • the operation log is a binary log.
  • the block data is service data.
  • the method can quickly calculate the consistency of data between the primary database and the secondary database, and update the data from the database according to the data of the primary database to achieve the function of data verification and data backup.
  • a data consistency check method for applying to a slave database.
  • 3 is a flow diagram of one embodiment of a data consistency verification method applied to a slave database in accordance with the present application.
  • the method may include the step S21 hash tree data receiving step: performing hashing on the block data in the first block set in the primary database in the primary database to obtain the first hash tree data, and receiving the The first hash tree data sent by the primary database.
  • the method may further include a step S22 verification result generating step of comparing the first hash tree data with the second hash tree data, generating a verification result and transmitting the result to the primary database, wherein the second The hash tree is obtained by hashing data of the block in the second block set corresponding to the first block set in the database, and the check result may include: A block number in which a block set and the second block set are inconsistent in data.
  • the method may further include the step S23 block data receiving step of receiving the block data corresponding to the block number in the first block set sent by the primary database according to the block number inconsistent with the data.
  • the method may further include the step S24 data updating step of: updating the block data of the corresponding block in the second block set according to the block data corresponding to the block number.
  • the method is used to enable the database to check and update its own data according to the data of the primary database, thereby realizing rapid data consistency error correction and functioning as a data backup.
  • the primary database and the secondary database may be located in the same physical node, for example, in the same server. From the perspective of data reliability, the primary database and the secondary database can also be located in different physical nodes, for example, in the primary server and the secondary server.
  • the block data is an operational log, preferably the operational log is a binary log.
  • FIG. 4 is a flow chart of another embodiment of a data consistency check method in accordance with the present application.
  • Master and Slave agree to store the operation log of the database in blocks in a fixed time unit.
  • the Master transmits the operation log block hash tree of the N time units of T1 to TN to the slave.
  • Slave compares the H operation log block Hash trees in the local corresponding time period, and quickly finds the inconsistent block number and sends the block number to the master.
  • the Master sends the block number data to the slave database to synchronize the inconsistent operation log blocks.
  • Slave uses the corresponding protocol of the database in chronological order to parse the corresponding database operations from the block data and play them locally, that is, perform database operations on the block data records locally.
  • FIG. 5 is a flow diagram of one embodiment of a data update step from a database.
  • the S24 data update step may include:
  • Step S241 a data parsing step: parsing an operation log corresponding to the block number
  • Step S242 executing the data: performing a database operation according to the operation log;
  • Step S243 Log update step: update the operation log of the slave database.
  • the block data is business data.
  • 6 is a flow chart of another embodiment of a data consistency check method in accordance with the present application.
  • the primary database and the underlying database are stored in a B+Tree or Skip List or similar format.
  • B+Tree is a variant of the B-tree and has a higher query performance than the B-tree.
  • Skip List is a kind of linked list, which is an improvement on ordered singly linked list. The use of hop table can make the query operation more complex data.
  • the leaf node stores real business data. When the Master performs leaf block operations, it also uses a storage structure to store the operation time of the leaf nodes.
  • the Master selects the block whose operation time conforms to the policy according to a certain silent data error check strategy. For example, select the block whose latest operation time belongs to 7 days ago, or 1 month ago, to build a block Hash tree. Master sends the selected block hash tree to Slave. Slave compares the corresponding block hash tree and sends the inconsistent block number to the master. After the Master synchronizes the inconsistent blocks, the Slave parses the corresponding values and performs the update operation of the B+Tree or Skip List.
  • FIG. 7 is a flow diagram of another embodiment of a data update step from a database.
  • the S24 data update step may include:
  • Step S241' data parsing step parsing the service data corresponding to the block number
  • Step S242' a data execution step of: performing a B+ tree operation or a table jump operation according to the service data.
  • the method provides a method for direct and fast synchronization of the master-slave database, solves the fast consistency check based on the operation log, and solves the problem of rapid consistency synchronization of silent error data.
  • a data consistency check device for use in a primary database.
  • 8 is a flow diagram of one embodiment of a data consistency verification device applied to a primary database in accordance with the present application.
  • the device may include a hash tree data sending module 11 configured to perform hash operation on the block data in the first block set in the primary database to obtain the first hash tree data, and The first hash tree data is sent to the slave database.
  • the method may further include a verification result receiving module 12 configured to receive the school sent from the database after comparing the first hash tree data and the second hash tree data from the database a result, wherein the second hash tree is obtained by hashing the block data in the second block set corresponding to the first block set in the database, the school
  • the result of the verification may include: a block number of the first block set and the second block set inconsistent data.
  • the method may further include a block data sending module 13 configured to send the block data corresponding to the block number in the first block set to the slave according to the block number of the data inconsistency And a database, so that the block data of the corresponding block in the second block set is updated from the database according to the block data corresponding to the block number.
  • a block data sending module 13 configured to send the block data corresponding to the block number in the first block set to the slave according to the block number of the data inconsistency And a database, so that the block data of the corresponding block in the second block set is updated from the database according to the block data corresponding to the block number.
  • the device divides the data of the primary database into blocks, constructs a hash tree, and implements fast consistency check of the primary database and the secondary database based on the hash tree, thereby quickly backing up the data of the primary database to the secondary database.
  • the first hash tree data and the second hash tree data are: constructing a leaf node by using the block data in the first block set and the second block set respectively All the hash data obtained by the tree.
  • the block is obtained by storing data of the primary database or the data of the secondary database in blocks of fixed time units.
  • the block data is an operation log or service data.
  • the operation log is a binary log.
  • a data consistency check device for use in a slave database.
  • 9 is a flow diagram of one embodiment of a data consistency verification device applied to a slave database in accordance with the present application.
  • the apparatus may include a hash tree data receiving module 21 configured to perform a hash operation on the block data in the first block set in the primary database in the primary database to obtain the first hash tree data. Receiving the first hash tree data sent by the primary database.
  • the apparatus may further include a verification result generating module 22 configured to compare the first hash tree data and the second hash tree data, generate a verification result, and send the verification result to the primary database, where
  • the second hash tree is obtained by hashing data of the block in the second block set corresponding to the first block set in the database, and the check result may include : the block number of the first block set and the second block set inconsistent data.
  • the device may further include a block data receiving module 23 configured to receive the block data corresponding to the block number in the first block set sent by the primary database according to the block number inconsistent with the data. .
  • the apparatus may further include a data update module 24 configured to update the block data of the corresponding block in the second set of blocks according to the block data corresponding to the block number.
  • the device enables the slave database to quickly obtain inconsistent data from the primary database, thereby enabling backup of the primary database data.
  • the block data is an operation log, and preferably, the operation log is a binary log.
  • the data update module 24 may include:
  • a data parsing module 241 configured to parse an operation log corresponding to the block number
  • a data execution module 242 configured to perform a database operation in accordance with the operational log
  • a log update module 243 is configured to update an operational log of the slave database.
  • the block data is business data.
  • the data update module 24 can include:
  • a data parsing module 241' configured to parse the service data corresponding to the block number
  • a data execution module 242' is configured to perform a B+ tree operation or a table jump operation based on the service data.
  • FIG. 10 is a flow diagram of one embodiment of a data consistency check system in accordance with the present application.
  • the system can include a primary database and at least one secondary database, the primary database and at least one secondary database connection.
  • the primary database may include any of the above-described data consistency check devices applied to the primary database.
  • the slave database may include any of the above-described data consistency check devices applied to the slave database.
  • the primary database is configured to perform hash operation on the block data in the first block set in the primary database to obtain the first Generating the data and transmitting the first hash tree data to the slave database;
  • the slave database is configured to compare the first hash tree data with the second hash tree data to generate a checksum And sending the result to the primary database, wherein the second hash tree is to hash the block data in the second block set corresponding to the first block set in the database
  • the check result may include: a block number in which the data in the first block set and the second block set are inconsistent; and the primary database is further used in an area according to the data inconsistency a block number, the block data corresponding to the block number in the first block set is sent to the slave database, so that the second block set is updated from the database according to the block data corresponding to the block number.
  • Block data for the corresponding block may include: a block number in which the data in the first block set and the second block set are inconsistent; and the primary database is further used in an area according to the
  • the system can quickly calculate the consistency of data between the primary database and the secondary database, and update the data from the database according to the data of the primary database to achieve data verification and data backup.
  • the number of slave databases is three.
  • the primary database and each secondary database are connected through a wired network or a wireless network.
  • the computer program product includes one or more computer instructions.
  • the computer loads and executes the computer program instructions, the processes or functions described in accordance with embodiments of the present application are generated in whole or in part.
  • the computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable device.
  • the computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be from a website site, computer, server or data center Transfer to another website site, computer, server, or data center by wire (eg, coaxial cable, fiber optic, digital subscriber line (DSL), or wireless (eg, infrared, wireless, microwave, etc.).
  • the computer readable storage medium can be any available media that can be accessed by a computer or a data storage device such as a server, data center, or the like that includes one or more available media.
  • the usable medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (such as a solid state disk (SSD)).

Abstract

Disclosed in the present application are a database master-slave block consistency check method, device and system, the data consistency check method being applied to a master database and comprising: performing a hash operation on block data in a first block set in the master database to obtain first hash tree data, and sending the data to a slave database; after comparing the first hash tree data and second hash tree data in the slave database, receiving a check result sent from the slave database; and according to a block number having data inconsistency in the check result, transmitting block data corresponding to said block number in the first block set to the slave database, so that the slave database updates block data of a corresponding block in a second block set according to the block data corresponding to the block number. By using the described method, the data consistency between the master database and the slave database may be quickly calculated, and data in the slave database may be updated according to the data of the master database, thereby achieving the functions of data check and data backup.

Description

数据库主从区块一致性校验方法、装置和***Database master-slave block consistency verification method, device and system 技术领域Technical field
本申请涉及数据存储和数据校验技术领域，特别是涉及一种数据一致性校验方法、装置和***，更具体地，涉及一种数据库主从区块一致性校验方法、装置和***。The present invention relates to the field of data storage and data verification technologies, and in particular, to a data consistency verification method, apparatus and system, and more particularly to a database master-slave block consistency verification method, apparatus and system.
背景技术Background technique
在现有的数据库存储***中，数据一致性问题是极为重要和复杂的。一旦***出现数据一致性问题，会带来十分严重的后果。在主数据库(Master)和从数据库(Slave)之间出现数据不一致的情况时，现有技术中通常采用的方法是分别将主数据库和从数据库的区块逐块进行对比，这种方法非常浪费带宽、耗费时间长，并且缺乏快速一致性状态检查机制，导致数据库运维工程师无法评估数据库之间的同步状态和数据的丢失率。In existing database storage systems, data consistency issues are extremely important and complex. Once the data consistency problem occurs in the system, it will have very serious consequences. In the case of data inconsistency between the primary database (Master) and the secondary database (Slave), the method commonly used in the prior art is to compare the primary database and the secondary database block by block, which is very wasteful. Bandwidth, long time, and lack of a fast consistency state checking mechanism, resulting in database operation and maintenance engineers unable to assess the synchronization status and data loss rate between databases.
发明内容Summary of the invention
本申请的目的在于克服上述问题或者至少部分地解决或缓减解决上述问题。The object of the present application is to overcome the above problems or at least partially solve or alleviate the above problems.
根据本申请的一个方面，提供了一种数据一致性校验方法，应用于主数据库，包括：According to an aspect of the present application, a data consistency verification method is provided, which is applied to a primary database, and includes:
哈希(hash)树数据发送步骤：对所述主数据库中的第一区块集合中的区块数据进行哈希运算，得到第一哈希树数据，并将所述第一哈希树数据发送给从数据库；Hash tree data sending step: hashing the block data in the first block set in the primary database, obtaining first hash tree data, and the first hash tree data Sent to the slave database;
校验结果接收步骤：在所述从数据库将所述第一哈希树数据和第二哈希树数据进行对比后，接收所述从数据库发送的校验结果，其中，所述第二哈希树是对所述从数据库中与所述第一区块集合相对应的第二区块集合中的区块数据进行哈希运算得到的，所述校验结果包括：所述第一区块集合和所述第二区块集合中数据不一致的区块号；和a verification result receiving step: after comparing the first hash tree data and the second hash tree data by the slave database, receiving the verification result sent from the database, wherein the second hash The tree is obtained by hashing the block data in the second block set corresponding to the first block set in the database, and the check result includes: the first block set a block number that is inconsistent with data in the second block set; and
区块数据发送步骤：根据所述数据不一致的区块号，将所述第一区块集合中该区块号对应的区块数据发送给所述从数据库，以便从数据库根据该区块号对应的区块数据更新所述第二区块集合中相应区块的区块数据。a block data sending step: transmitting block data corresponding to the block number in the first block set to the slave database according to the block number inconsistent with the data, so as to correspond to the slave block according to the block number The block data updates the block data of the corresponding block in the second block set.
该方法能够快速计算主数据库和从数据库之间的数据的一致性，并根据主 数据库的数据更新从数据库的数据，达到数据校验和数据备份的作用。The method can quickly calculate the consistency of data between the primary database and the secondary database, and update the data from the database according to the data of the primary database to achieve the function of data verification and data backup.
可选地，所述第一哈希树数据和所述第二哈希树数据为：分别以所述第一区块集合和所述第二区块集合中的区块数据作为叶子节点构建哈希树所得到的所有的哈希数据。Optionally, the first hash tree data and the second hash tree data are: constructing a leaf node by using the block data in the first block set and the second block set respectively All the hash data obtained by the tree.
可选地，所述区块是将所述主数据库的数据或者所述从数据库的数据以固定的时间单元分块存储得到的。Optionally, the block is obtained by storing data of the primary database or the data of the secondary database in blocks of fixed time units.
可选地，所述区块数据为操作日志或者业务数据。可选地，所述操作日志为二进制日志(Binlog)。Optionally, the block data is an operation log or service data. Optionally, the operation log is a binary log (Binlog).
根据本申请的另一个方面，还提供了一种数据一致性校验方法，应用于从数据库，包括：According to another aspect of the present application, a data consistency verification method is also provided, which is applied to a slave database, including:
哈希树数据接收步骤：在主数据库对所述主数据库中的第一区块集合中的区块数据进行哈希运算，得到第一哈希树数据后，接收所述主数据库发送的所述第一哈希树数据；a hash tree data receiving step: performing hashing on the block data in the first block set in the primary database in the primary database to obtain the first hash tree data, and receiving the First hash tree data;
校验结果生成步骤：将所述第一哈希树数据和第二哈希树数据进行对比，生成校验结果并发送给所述主数据库，其中，所述第二哈希树是对所述从数据库中与所述第一区块集合相对应的第二区块集合中的区块的数据进行哈希运算得到的，所述校验结果包括：所述第一区块集合和所述第二区块集合中数据不一致的区块号；a verification result generating step: comparing the first hash tree data with the second hash tree data, generating a verification result and transmitting the verification result to the primary database, wherein the second hash tree is for the And performing a hash operation on the data of the block in the second block set corresponding to the first block set in the database, where the check result includes: the first block set and the first The block number in which the data in the second block set is inconsistent;
区块数据接收步骤：接收所述主数据库根据所述数据不一致的区块号发送的所述第一区块集合中该区块号对应的区块数据；和a block data receiving step: receiving, by the primary database, block data corresponding to the block number in the first block set sent according to the block number inconsistent with the data; and
数据更新步骤：根据该区块号对应的区块数据更新所述第二区块集合中相应区块的区块数据。The data updating step: updating the block data of the corresponding block in the second block set according to the block data corresponding to the block number.
该装置实现了基于哈希树的主数据库和从数据库快速一致性校验和同步，对于静默出错数据，也能实现快递校验和更正。The device realizes fast consistency check and synchronization of the main database based on the hash tree and the slave database, and can also implement the checksum correction for the silent error data.
可选地，所述区块数据为操作日志，优选地，所述操作日志为二进制日志。Optionally, the block data is an operation log, and preferably, the operation log is a binary log.
可选地，所述数据更新步骤包括：Optionally, the data update step includes:
数据解析步骤：解析该区块号对应的操作日志；Data parsing step: parsing an operation log corresponding to the block number;
数据执行步骤：根据所述操作日志执行数据库操作；Data execution step: performing a database operation according to the operation log;
日志更新步骤：更新所述从数据库的操作日志。Log update step: update the operation log of the slave database.
该装置实现了基于哈希树的主数据库和从数据库快速一致性校验和同步。The device implements a hash tree-based master database and a slave database for fast consistency checksum synchronization.
可选地，所述区块数据为业务数据。Optionally, the block data is service data.
可选地，所述数据更新步骤包括：Optionally, the data update step includes:
数据解析步骤：解析该区块号对应的业务数据；和Data parsing step: parsing the business data corresponding to the block number; and
数据执行步骤：根据所述业务数据执行B+树(B+Tree)操作或者跳表(Skip List)操作。Data execution step: performing a B+Tree (B+Tree) operation or a Skip List operation according to the service data.
该装置基于叶子节点数据块的Hash树，实现了主数据库和从数据库的快速静默出错数据一致性校验和更正。The device is based on the hash tree of the leaf node data block, and implements fast silent error data consistency check and correction of the primary database and the secondary database.
根据本申请的另一个方面，还提供了一种数据一致性校验装置，应用于主数据库，包括：According to another aspect of the present application, there is also provided a data consistency verification apparatus, applied to a primary database, comprising:
哈希树数据发送模块，其配置为用于对所述主数据库中的第一区块集合中的区块数据进行哈希运算，得到第一哈希树数据，并将所述第一哈希树数据发送给从数据库；a hash tree data sending module, configured to perform hash operation on the block data in the first block set in the primary database, to obtain first hash tree data, and to use the first hash Tree data is sent to the slave database;
校验结果接收模块，其配置为用于在所述从数据库将所述第一哈希树数据和第二哈希树数据进行对比后，接收所述从数据库发送的校验结果，其中，所述第二哈希树是对所述从数据库中与所述第一区块集合相对应的第二区块集合中的区块数据进行哈希运算得到的，所述校验结果包括：所述第一区块集合和所述第二区块集合中数据不一致的区块号；和a verification result receiving module, configured to receive the verification result sent by the slave database after the first hash tree data and the second hash tree data are compared by the slave database, where The second hash tree is obtained by hashing the block data in the second block set corresponding to the first block set in the database, and the check result includes: a block number in which the data in the first block set and the second block set are inconsistent; and
区块数据发送模块，其配置为用于根据所述数据不一致的区块号，将所述第一区块集合中该区块号对应的区块数据发送给所述从数据库，以便从数据库根据该区块号对应的区块数据更新所述第二区块集合中相应区块的区块数据。a block data sending module, configured to send, according to the block number of the data inconsistency, block data corresponding to the block number in the first block set to the slave database, so as to be based on the database The block data corresponding to the block number updates the block data of the corresponding block in the second block set.
该装置将主数据库的数据分成区块，构建哈希树，并基于哈希树实现了主数据库和从数据库的数据快速一致性校验，从而将主数据库的数据快速备份到从数据库。The device divides the data of the primary database into blocks, constructs a hash tree, and implements fast consistency check of the primary database and the secondary database based on the hash tree, thereby quickly backing up the data of the primary database to the secondary database.
可选地，所述第一哈希树数据和所述第二哈希树数据为：分别以所述第一区块集合和所述第二区块集合中的区块数据作为叶子节点构建哈希树所得到的所有的哈希数据。Optionally, the first hash tree data and the second hash tree data are: constructing a leaf node by using the block data in the first block set and the second block set respectively All the hash data obtained by the tree.
可选地，所述区块是将所述主数据库的数据或者所述从数据库的数据以固定的时间单元分块存储得到的。Optionally, the block is obtained by storing data of the primary database or the data of the secondary database in blocks of fixed time units.
可选地，所述区块数据为操作日志或者业务数据。Optionally, the block data is an operation log or service data.
可选地，所述操作日志为二进制日志。Optionally, the operation log is a binary log.
根据本申请的另一个方面，还提供了一种数据一致性校验装置，应用于从数据库，包括：According to another aspect of the present application, there is also provided a data consistency check device, which is applied to a slave database, comprising:
哈希树数据接收模块，其配置为用于在主数据库对所述主数据库中的第一区块集合中的区块数据进行哈希运算，得到第一哈希树数据后，接收所述主数 据库发送的所述第一哈希树数据；a hash tree data receiving module, configured to perform hash operation on the block data in the first block set in the primary database in the primary database, to obtain the first hash tree data, and receive the primary The first hash tree data sent by the database;
校验结果生成模块，其配置为用于将所述第一哈希树数据和第二哈希树数据进行对比，生成校验结果并发送给所述主数据库，其中，所述第二哈希树是对所述从数据库中与所述第一区块集合相对应的第二区块集合中的区块的数据进行哈希运算得到的，所述校验结果包括：所述第一区块集合和所述第二区块集合中数据不一致的区块号；a verification result generating module configured to compare the first hash tree data and the second hash tree data, generate a verification result, and send the verification result to the primary database, where the second hash The tree is obtained by hashing data of the block in the second block set corresponding to the first block set in the database, and the check result includes: the first block a block number that is inconsistent with the data in the set of the second block;
区块数据接收模块，其配置为用于接收所述主数据库根据所述数据不一致的区块号发送的所述第一区块集合中该区块号对应的区块数据；和a block data receiving module, configured to receive block data corresponding to the block number in the first block set sent by the primary database according to the block number inconsistent with the data; and
数据更新模块，其配置为用于根据该区块号对应的区块数据更新所述第二区块集合中相应区块的区块数据。And a data update module configured to update the block data of the corresponding block in the second block set according to the block data corresponding to the block number.
该装置使得从数据库能够从主数据库快速获取不一致数据，从而实现对主数据库数据的备份。The device enables the slave database to quickly obtain inconsistent data from the primary database, thereby enabling backup of the primary database data.
可选地，所述区块数据为操作日志，优选地，所述操作日志为二进制日志。Optionally, the block data is an operation log, and preferably, the operation log is a binary log.
可选地，所述数据更新模块包括：Optionally, the data update module includes:
数据解析模块，其配置为用于解析该区块号对应的操作日志；a data parsing module configured to parse an operation log corresponding to the block number;
数据执行模块，其配置为用于根据所述操作日志执行数据库操作；和a data execution module configured to perform a database operation based on the operation log; and
日志更新模块，其配置为用于更新所述从数据库的操作日志。A log update module configured to update an operation log of the slave database.
可选地，所述区块数据为业务数据。Optionally, the block data is service data.
可选地，所述数据更新模块包括：Optionally, the data update module includes:
数据解析模块，其配置为用于解析该区块号对应的业务数据；a data parsing module configured to parse the service data corresponding to the block number;
数据执行模块，其配置为用于根据所述业务数据执行B+树操作或者跳表操作。And a data execution module configured to perform a B+ tree operation or a table jump operation according to the service data.
根据本申请的另一个方面，还提供了一种数据一致性校验***，包括主数据库和至少一个从数据库，所述主数据库和至少一个从数据库之间相互连接，其中：According to another aspect of the present application, there is also provided a data consistency check system comprising a primary database and at least one secondary database, the primary database and the at least one secondary database being interconnected, wherein:
所述主数据库用于对所述主数据库中的第一区块集合中的区块数据进行哈希运算，得到第一哈希树数据，并将所述第一哈希树数据发送给所述从数据库；The primary database is configured to perform hash operation on the block data in the first block set in the primary database, obtain first hash tree data, and send the first hash tree data to the From the database;
所述从数据库用于将所述第一哈希树数据和第二哈希树数据进行对比后，生成校验结果并发送给所述主数据库，其中，所述第二哈希树是对所述从数据库中与所述第一区块集合相对应的第二区块集合中的区块数据进行哈希运算得到的，所述校验结果包括：所述第一区块集合和所述第二区块集合中数据不 一致的区块号；The slave database is configured to compare the first hash tree data with the second hash tree data, generate a verification result, and send the verification result to the primary database, where the second hash tree is a The hash data in the second block set corresponding to the first block set in the database is hashed, and the check result includes: the first block set and the first The block number in which the data in the second block set is inconsistent;
所述主数据库还用于根据所述数据不一致的区块号，将所述第一区块集合中该区块号对应的区块数据发送给所述从数据库，以便从数据库根据该区块号对应的区块数据更新所述第二区块集合中相应区块的区块数据。The primary database is further configured to send the block data corresponding to the block number in the first block set to the slave database according to the block number of the data inconsistency, so as to be based on the block number from the database. The corresponding block data updates the block data of the corresponding block in the second block set.
该***能够快速计算主数据库和从数据库之间的数据的一致性，达到数据校验和数据备份的作用。根据下文结合附图对本申请具体实施例的详细描述，本领域技术人员将会更加明了本申请的上述以及其他目的、优点和特征。The system can quickly calculate the consistency of data between the primary database and the secondary database, and achieve the role of data verification and data backup. The above and other objects, advantages and features of the present application will become apparent to those skilled in the <RTI
附图说明DRAWINGS
后文将参照附图以示例性而非限制性的方式详细描述本申请的一些具体实施例。附图中相同的附图标记标示了相同或类似的部件或部分。本领域技术人员应该理解，这些附图未必是按比例绘制的。附图中：Some specific embodiments of the present application will be described in detail below by way of example and not limitation. The same reference numbers in the drawings identify the same or similar parts. Those skilled in the art should understand that the drawings are not necessarily drawn to scale. In the figure:
图1是根据本申请的应用于主数据库的数据一致性校验方法的一个实施例的流程图；1 is a flow chart of one embodiment of a data consistency check method applied to a primary database in accordance with the present application;
图2是哈希树的建立方法的一个实施例的流程图；2 is a flow chart of one embodiment of a method of establishing a hash tree;
图3是根据本申请的应用于从数据库的数据一致性校验方法的一个实施例的流程图；3 is a flow diagram of one embodiment of a data consistency verification method applied to a slave database in accordance with the present application;
图4是根据本申请的数据一致性校验方法的另一个实施例的流程图；4 is a flow chart of another embodiment of a data consistency check method in accordance with the present application;
图5是从数据库的数据更新步骤的一个实施例的流程图；Figure 5 is a flow diagram of one embodiment of a data update step from a database;
图6是根据本申请的数据一致性校验方法的另一个实施例的流程图；6 is a flow chart of another embodiment of a data consistency check method in accordance with the present application;
图7是从数据库的数据更新步骤的另一个实施例的流程图；Figure 7 is a flow diagram of another embodiment of a data update step from a database;
图8是根据本申请的应用于主数据库的数据一致性校验装置的一个实施例的流程图；8 is a flow diagram of one embodiment of a data consistency check device applied to a primary database in accordance with the present application;
图9是根据本申请的应用于从数据库的数据一致性校验装置的一个实施例的流程图；9 is a flow diagram of one embodiment of a data consistency check device applied to a slave database in accordance with the present application;
图10是根据本申请的数据一致性校验***的一个实施例的流程图。10 is a flow diagram of one embodiment of a data consistency check system in accordance with the present application.
具体实施方式detailed description
根据下文结合附图对本申请的具体实施例的详细描述，本领域技术人员将会更加明了本申请的上述以及其他目的、优点和特征。The above and other objects, advantages and features of the present application will become apparent to those skilled in the <RTI
根据本申请的一个方面，提供了一种数据一致性校验方法，应用于主数据库。图1是根据本申请的应用于主数据库的数据一致性校验方法的一个实施例 的流程图。该方法可以包括步骤S11哈希树数据发送步骤：对所述主数据库中的第一区块集合中的区块数据进行哈希运算，得到第一哈希树数据，并将所述第一哈希树数据发送给从数据库。According to an aspect of the present application, a data consistency verification method is provided for application to a primary database. BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 is a flow diagram of one embodiment of a data consistency verification method applied to a primary database in accordance with the present application. The method may include the step S11 hash tree data sending step: hashing the block data in the first block set in the primary database, obtaining the first hash tree data, and the first hash The hash tree data is sent to the slave database.
可选地，所述区块是将所述主数据库的数据库或者所述从数据库的数据库中的数据以固定的时间单元分块存储得到的。在一种可选的实施方案中，Master和Slave可以约定以固定的时间单元，将数据库的操作日志分块存储，比如按一分钟为基本单位进行分块，从而得到若干个区块。在构建哈希树时，可以根据哈希树的层数选择第一区块集合中的区块数目。例如，8个。图2是哈希树的建立方法的一个实施例的流程图。在图2中示出了一个四层哈希树的结构。最下面一层，即第四层，是叶子节点。该叶子节点的内容可以是区块数据。该哈希树的叶子节点包括8个区块，分别标为D1区块至D8区块。它们构成了第一区块集合。将第一区块集合中每两个区块分为一组进行哈希运算，得到第三层的哈希值。例如，将D1区块和D2区块的数据进行哈希运算，得到结果N1。以此类推，倒数第二层共得到4个哈希值N1至N4。将这四个哈希值的每两个分为一组继续进行哈希运算，得到第二层的哈希值M1和M2。将第二层的两个哈希值再次经哈希运算，得到根节点的哈希值Root。可以理解的是，哈希值数的层数可以根据服务器的运算能力和数据量进行选择。D1区块至D8区块可以是按照时间的顺序进行排序的区块，也可以按照其他方式进行排列。在每一层中，可以选择任意两个区块或者哈希值进行哈希运算，图2中所示的运算顺序不应理解为对本申请的限制。Optionally, the block is obtained by storing the data of the primary database or the data in the database of the secondary database in blocks of fixed time units. In an optional implementation, the Master and the Slave may agree to store the operation log of the database in a fixed time unit, for example, by dividing the block by one minute, thereby obtaining a plurality of blocks. When constructing a hash tree, the number of blocks in the first block set can be selected according to the number of layers of the hash tree. For example, eight. 2 is a flow chart of one embodiment of a method of establishing a hash tree. The structure of a four-layer hash tree is shown in FIG. The bottom layer, the fourth layer, is the leaf node. The content of the leaf node may be block data. The leaf node of the hash tree includes 8 blocks, which are respectively labeled as D1 block to D8 block. They form the first block set. Each of the two blocks in the first block set is hashed into a group to obtain a hash value of the third layer. For example, the data of the D1 block and the D2 block are hashed to obtain the result N1. By analogy, the penultimate layer obtains a total of four hash values N1 to N4. The two hashes of the two hash values M1 and M2 are obtained by grouping each of the four hash values into a group. The two hash values of the second layer are hashed again to obtain the hash value Root of the root node. It can be understood that the number of layers of the hash value can be selected according to the computing power and the amount of data of the server. The D1 block to the D8 block may be blocks sorted in chronological order, or may be arranged in other manners. In each layer, any two blocks or hash values may be selected for hashing, and the order of operations shown in FIG. 2 should not be construed as limiting the present application.
可选地，所述第一哈希树数据是以所述第一区块集合中的区块数据作为叶子节点构建哈希树所得到的所有的哈希数据。也就是说，第一哈希树数据可以是将所有层的哈希值按照一定的顺序进行连接得到的结果。例如，可以按照从第一层到第三层的顺序将每一层的哈希值进行拼接，或者按照从第三层到第一层的顺序进行拼接。可以理解的是，第一哈希树数据也可以是将其中某些层的结果进行拼接得到的结果，也可以只传送根节点的哈希值。Optionally, the first hash tree data is all hash data obtained by constructing a hash tree by using the block data in the first block set as a leaf node. That is to say, the first hash tree data may be the result of connecting the hash values of all layers in a certain order. For example, the hash values of each layer may be spliced in order from the first layer to the third layer, or spliced in the order from the third layer to the first layer. It can be understood that the first hash tree data may also be the result of splicing the results of some of the layers, or may only transmit the hash value of the root node.
该方法还可以包括步骤S12校验结果接收步骤：在所述从数据库将所述第一哈希树数据和第二哈希树数据进行对比后，接收所述从数据库发送的校验结果，其中，所述第二哈希树是对所述从数据库中与所述第一区块集合相对应的第二区块集合中的区块数据进行哈希运算得到的，所述校验结果可以包括：所述第一区块集合和所述第二区块集合中数据不一致的区块号。The method may further include a step S12 verification result receiving step: after comparing the first hash tree data and the second hash tree data from the database, receiving the verification result sent from the database, wherein The second hash tree is obtained by hashing the block data in the second block set corresponding to the first block set in the database, and the check result may include : the block number of the first block set and the second block set inconsistent data.
可选地，所述第二哈希树数据是以所述第二区块集合中的区块数据作为叶 子节点构建哈希树所得到的所有的哈希数据。Optionally, the second hash tree data is all hash data obtained by constructing a hash tree as the leaf node by using the block data in the second block set.
在一个可选的实施方案中，从数据库中记录有与主数据库对应一致的数据。按照主数据库的区块划分方法，将从数据库中的数据划分成相同的区块，例如，分为D1’区块至D8’区块。在从数据库中，按照同样的方法计算哈希值，得到第二哈希树数据。从数据库将第二哈希树数据与主数据库传送的第一哈希树数据进行对比，能够得到不一致的哈希值，通过不一致的哈希值的位置能够判断出数据不一致的区块号。例如，如果从数据库通过对比发现，第二哈希树中与第一哈希树中N1值对应的哈希值N1’与N1不一致，则验证结果中可以包括区块号D1和D2。如果主数据库仅传送每个哈希树的根节点的数据，则如果某个根节点数据不一致，则验证结果中可以包括该根节点下方所有的区块号。In an alternative embodiment, data corresponding to the primary database is recorded from the database. According to the block division method of the main database, the data in the database is divided into the same block, for example, divided into D1' blocks to D8' blocks. In the slave database, the hash value is calculated in the same way to obtain the second hash tree data. Comparing the second hash tree data with the first hash tree data transmitted by the primary database from the database, an inconsistent hash value can be obtained, and the block number of the data inconsistency can be determined by the position of the inconsistent hash value. For example, if it is found from the comparison by the database that the hash value N1' corresponding to the N1 value in the first hash tree in the second hash tree does not coincide with N1, the block numbers D1 and D2 may be included in the verification result. If the primary database only transmits the data of the root node of each hash tree, if a certain root node data is inconsistent, the verification result may include all the block numbers below the root node.
该方法还可以包括步骤S13区块数据发送步骤：根据所述数据不一致的区块号，将所述第一区块集合中该区块号对应的区块数据发送给所述从数据库，以便所述从数据库根据该区块号对应的区块数据更新所述第二区块集合中相应区块的区块数据。The method may further include the step of transmitting a block data in the step S13: sending the block data corresponding to the block number in the first block set to the slave database according to the block number of the data inconsistency The slave database updates the block data of the corresponding block in the second block set according to the block data corresponding to the block number.
例如，主数据库在收到不一致的区块号D1和D2后，则将该数据库中的D1区块和D2区块的数据发送给从数据库。从数据库根据该区块号对应的区块数据更新所述第二区块集合中相应区块的区块数据D1’和D2’。For example, after receiving the inconsistent block numbers D1 and D2, the primary database sends the data of the D1 block and the D2 block in the database to the slave database. The block data D1' and D2' of the corresponding block in the second block set are updated from the database based on the block data corresponding to the block number.
可选地，所述区块数据为操作日志。进一步地，所述操作日志为二进制日志。可选地，所述区块数据为业务数据。Optionally, the block data is an operation log. Further, the operation log is a binary log. Optionally, the block data is service data.
该方法能够快速计算主数据库和从数据库之间的数据的一致性，并根据主数据库的数据更新从数据库的数据，达到数据校验和数据备份的作用。The method can quickly calculate the consistency of data between the primary database and the secondary database, and update the data from the database according to the data of the primary database to achieve the function of data verification and data backup.
根据本申请的另一个方面，还提供了一种数据一致性校验方法，应用于从数据库。图3是根据本申请的应用于从数据库的数据一致性校验方法的一个实施例的流程图。该方法可以包括步骤S21哈希树数据接收步骤：在主数据库对所述主数据库中的第一区块集合中的区块数据进行哈希运算，得到第一哈希树数据后，接收所述主数据库发送的所述第一哈希树数据。According to another aspect of the present application, there is also provided a data consistency check method for applying to a slave database. 3 is a flow diagram of one embodiment of a data consistency verification method applied to a slave database in accordance with the present application. The method may include the step S21 hash tree data receiving step: performing hashing on the block data in the first block set in the primary database in the primary database to obtain the first hash tree data, and receiving the The first hash tree data sent by the primary database.
该方法还可以包括步骤S22校验结果生成步骤：将所述第一哈希树数据和第二哈希树数据进行对比，生成校验结果并发送给所述主数据库，其中，所述第二哈希树是对所述从数据库中与所述第一区块集合相对应的第二区块集合中的区块的数据进行哈希运算得到的，所述校验结果可以包括：所述第一区块集合和所述第二区块集合中数据不一致的区块号。The method may further include a step S22 verification result generating step of comparing the first hash tree data with the second hash tree data, generating a verification result and transmitting the result to the primary database, wherein the second The hash tree is obtained by hashing data of the block in the second block set corresponding to the first block set in the database, and the check result may include: A block number in which a block set and the second block set are inconsistent in data.
该方法还可以包括步骤S23区块数据接收步骤：接收所述主数据库根据所 述数据不一致的区块号发送的所述第一区块集合中该区块号对应的区块数据。The method may further include the step S23 block data receiving step of receiving the block data corresponding to the block number in the first block set sent by the primary database according to the block number inconsistent with the data.
该方法还可以包括步骤S24数据更新步骤：根据该区块号对应的区块数据更新所述第二区块集合中相应区块的区块数据。The method may further include the step S24 data updating step of: updating the block data of the corresponding block in the second block set according to the block data corresponding to the block number.
利用该方法，使得从数据库能够根据主数据库的数据对自身数据进行校验和更新，从而实现数据快速一致性纠错，起到数据备份的作用。The method is used to enable the database to check and update its own data according to the data of the primary database, thereby realizing rapid data consistency error correction and functioning as a data backup.
应当理解的是，主数据库和从数据库可以位于同一个物理节点中，例如，位于同一台服务器中。从数据的可靠性角度来看，主数据库和从数据库也可以位于不同的物理节点中，例如，分别位于主服务器和从服务器中。It should be understood that the primary database and the secondary database may be located in the same physical node, for example, in the same server. From the perspective of data reliability, the primary database and the secondary database can also be located in different physical nodes, for example, in the primary server and the secondary server.
在一个可选的实施方案中，所述区块数据为操作日志，优选地，所述操作日志为二进制日志。In an alternative embodiment, the block data is an operational log, preferably the operational log is a binary log.
图4是根据本申请的数据一致性校验方法的另一个实施例的流程图。在该方法中，Master和Slave约定以固定的时间单元，将数据库的操作日志分块存储。Master将T1至TN的N个时间单元的操作日志区块hash树传输给Slave。Slave对比本地相应时间段的N个操作日志区块Hash树，并快速找到不一致的区块号，并将该区块号发送给Master。Master将该区块号的数据发送给从数据库，以同步不一致的操作日志区块。Slave按照时间顺序利用数据库相应的协议，从区块数据中解析相应的数据库操作，在本地进行回放，也就是说，在本地执行区块数据记录的数据库操作。4 is a flow chart of another embodiment of a data consistency check method in accordance with the present application. In this method, Master and Slave agree to store the operation log of the database in blocks in a fixed time unit. The Master transmits the operation log block hash tree of the N time units of T1 to TN to the slave. Slave compares the H operation log block Hash trees in the local corresponding time period, and quickly finds the inconsistent block number and sends the block number to the master. The Master sends the block number data to the slave database to synchronize the inconsistent operation log blocks. Slave uses the corresponding protocol of the database in chronological order to parse the corresponding database operations from the block data and play them locally, that is, perform database operations on the block data records locally.
可选地，图5是从数据库的数据更新步骤的一个实施例的流程图。该S24数据更新步骤可以包括：Optionally, Figure 5 is a flow diagram of one embodiment of a data update step from a database. The S24 data update step may include:
步骤S241数据解析步骤：解析该区块号对应的操作日志；Step S241: a data parsing step: parsing an operation log corresponding to the block number;
步骤S242数据执行步骤：根据所述操作日志执行数据库操作；和Step S242: executing the data: performing a database operation according to the operation log; and
步骤S243日志更新步骤：更新所述从数据库的操作日志。Step S243: Log update step: update the operation log of the slave database.
在另一个可选的实施方案中，所述区块数据为业务数据。图6是根据本申请的数据一致性校验方法的另一个实施例的流程图。在该实施方案中，主数据库和从数据库的底层按照B+Tree或者Skip List或者类似的格式进行数据存储。B+Tree是B树的变种，有着比B树更高的查询性能。Skip List是链表的一种，是对有序单向链表的一种改进，使用跳表可以将查询操作的复杂度更高的数据。在这两种格式下，叶子节点存储的是真实的业务数据。Master进行叶子块操作的时候，同时利用一个存储结构存储叶子节点的操作时间。Master根据一定的静默数据出错校验策略，选取操作时间符合该策略的区块。例如，选择最新操作时间属于7天前，或者1个月之前的区块，构建区块Hash树。 Master将选择的区块Hash树发送给Slave。Slave对比相应的区块Hash树，并将不一致的区块号发送给Master。Master同步完不一致的区块后，Slave解析相应值，进行B+Tree或者Skip List的更新操作。In another optional embodiment, the block data is business data. 6 is a flow chart of another embodiment of a data consistency check method in accordance with the present application. In this embodiment, the primary database and the underlying database are stored in a B+Tree or Skip List or similar format. B+Tree is a variant of the B-tree and has a higher query performance than the B-tree. Skip List is a kind of linked list, which is an improvement on ordered singly linked list. The use of hop table can make the query operation more complex data. In both formats, the leaf node stores real business data. When the Master performs leaf block operations, it also uses a storage structure to store the operation time of the leaf nodes. The Master selects the block whose operation time conforms to the policy according to a certain silent data error check strategy. For example, select the block whose latest operation time belongs to 7 days ago, or 1 month ago, to build a block Hash tree. Master sends the selected block hash tree to Slave. Slave compares the corresponding block hash tree and sends the inconsistent block number to the master. After the Master synchronizes the inconsistent blocks, the Slave parses the corresponding values and performs the update operation of the B+Tree or Skip List.
图7是从数据库的数据更新步骤的另一个实施例的流程图。在该实施方式中，S24数据更新步骤可以包括：Figure 7 is a flow diagram of another embodiment of a data update step from a database. In this embodiment, the S24 data update step may include:
步骤S241’数据解析步骤：解析该区块号对应的业务数据；Step S241' data parsing step: parsing the service data corresponding to the block number;
步骤S242’数据执行步骤：根据所述业务数据执行B+树操作或者跳表操作。Step S242': a data execution step of: performing a B+ tree operation or a table jump operation according to the service data.
该方法提供了一种主从数据库直接快速同步的方法，解决了基于操作日志的快速一致性校验，并解决了静默出错数据快速一致性同步问题。The method provides a method for direct and fast synchronization of the master-slave database, solves the fast consistency check based on the operation log, and solves the problem of rapid consistency synchronization of silent error data.
根据本申请的另一个方面，还提供了一种数据一致性校验装置，应用于主数据库。图8是根据本申请的应用于主数据库的数据一致性校验装置的一个实施例的流程图。该装置可以包括哈希树数据发送模块11，其配置为用于对所述主数据库中的第一区块集合中的区块数据进行哈希运算，得到第一哈希树数据，并将所述第一哈希树数据发送给从数据库。According to another aspect of the present application, there is also provided a data consistency check device for use in a primary database. 8 is a flow diagram of one embodiment of a data consistency verification device applied to a primary database in accordance with the present application. The device may include a hash tree data sending module 11 configured to perform hash operation on the block data in the first block set in the primary database to obtain the first hash tree data, and The first hash tree data is sent to the slave database.
该方法还可以包括校验结果接收模块12，其配置为用于在所述从数据库将所述第一哈希树数据和第二哈希树数据进行对比后，接收所述从数据库发送的校验结果，其中，所述第二哈希树是对所述从数据库中与所述第一区块集合相对应的第二区块集合中的区块数据进行哈希运算得到的，所述校验结果可以包括：所述第一区块集合和所述第二区块集合中数据不一致的区块号。The method may further include a verification result receiving module 12 configured to receive the school sent from the database after comparing the first hash tree data and the second hash tree data from the database a result, wherein the second hash tree is obtained by hashing the block data in the second block set corresponding to the first block set in the database, the school The result of the verification may include: a block number of the first block set and the second block set inconsistent data.
该方法还可以包括区块数据发送模块13，其配置为用于根据所述数据不一致的区块号，将所述第一区块集合中该区块号对应的区块数据发送给所述从数据库，以便从数据库根据该区块号对应的区块数据更新所述第二区块集合中相应区块的区块数据。The method may further include a block data sending module 13 configured to send the block data corresponding to the block number in the first block set to the slave according to the block number of the data inconsistency And a database, so that the block data of the corresponding block in the second block set is updated from the database according to the block data corresponding to the block number.
该装置将主数据库的数据分成区块，构建哈希树，并基于哈希树实现了主数据库和从数据库的数据快速一致性校验，从而将主数据库的数据快速备份到从数据库。The device divides the data of the primary database into blocks, constructs a hash tree, and implements fast consistency check of the primary database and the secondary database based on the hash tree, thereby quickly backing up the data of the primary database to the secondary database.
可选地，所述第一哈希树数据和所述第二哈希树数据为：分别以所述第一区块集合和所述第二区块集合中的区块数据作为叶子节点构建哈希树所得到的所有的哈希数据。Optionally, the first hash tree data and the second hash tree data are: constructing a leaf node by using the block data in the first block set and the second block set respectively All the hash data obtained by the tree.
可选地，所述区块是将所述主数据库的数据或者所述从数据库的数据以固定的时间单元分块存储得到的。Optionally, the block is obtained by storing data of the primary database or the data of the secondary database in blocks of fixed time units.
可选地，所述区块数据为操作日志或者业务数据。进一步地，所述操作日 志为二进制日志。Optionally, the block data is an operation log or service data. Further, the operation log is a binary log.
根据本申请的另一个方面，还提供了一种数据一致性校验装置，应用于从数据库。图9是根据本申请的应用于从数据库的数据一致性校验装置的一个实施例的流程图。该装置可以包括哈希树数据接收模块21，其配置为用于在主数据库对所述主数据库中的第一区块集合中的区块数据进行哈希运算，得到第一哈希树数据后，接收所述主数据库发送的所述第一哈希树数据。According to another aspect of the present application, there is also provided a data consistency check device for use in a slave database. 9 is a flow diagram of one embodiment of a data consistency verification device applied to a slave database in accordance with the present application. The apparatus may include a hash tree data receiving module 21 configured to perform a hash operation on the block data in the first block set in the primary database in the primary database to obtain the first hash tree data. Receiving the first hash tree data sent by the primary database.
该装置还可以包括校验结果生成模块22，其配置为用于将所述第一哈希树数据和第二哈希树数据进行对比，生成校验结果并发送给所述主数据库，其中，所述第二哈希树是对所述从数据库中与所述第一区块集合相对应的第二区块集合中的区块的数据进行哈希运算得到的，所述校验结果可以包括：所述第一区块集合和所述第二区块集合中数据不一致的区块号。The apparatus may further include a verification result generating module 22 configured to compare the first hash tree data and the second hash tree data, generate a verification result, and send the verification result to the primary database, where The second hash tree is obtained by hashing data of the block in the second block set corresponding to the first block set in the database, and the check result may include : the block number of the first block set and the second block set inconsistent data.
该装置还可以包括区块数据接收模块23，其配置为用于接收所述主数据库根据所述数据不一致的区块号发送的所述第一区块集合中该区块号对应的区块数据。The device may further include a block data receiving module 23 configured to receive the block data corresponding to the block number in the first block set sent by the primary database according to the block number inconsistent with the data. .
该装置还可以包括数据更新模块24，其配置为用于根据该区块号对应的区块数据更新所述第二区块集合中相应区块的区块数据。The apparatus may further include a data update module 24 configured to update the block data of the corresponding block in the second set of blocks according to the block data corresponding to the block number.
该装置使得从数据库能够从主数据库快速获取不一致数据，从而实现对主数据库数据的备份。The device enables the slave database to quickly obtain inconsistent data from the primary database, thereby enabling backup of the primary database data.
可选地，所述区块数据为操作日志，优选地，所述操作日志为二进制日志。Optionally, the block data is an operation log, and preferably, the operation log is a binary log.
可选地，所述数据更新模块24可以包括：Optionally, the data update module 24 may include:
数据解析模块241，其配置为用于解析该区块号对应的操作日志；和a data parsing module 241 configured to parse an operation log corresponding to the block number; and
数据执行模块242，其配置为用于根据所述操作日志执行数据库操作；和a data execution module 242 configured to perform a database operation in accordance with the operational log; and
日志更新模块243，其配置为用于更新所述从数据库的操作日志。A log update module 243 is configured to update an operational log of the slave database.
在一个可选的实施方案中，所述区块数据为业务数据。In an alternative embodiment, the block data is business data.
在一个可选的实施方案中，所述数据更新模块24可以包括：In an optional implementation, the data update module 24 can include:
数据解析模块241’，其配置为用于解析该区块号对应的业务数据；和a data parsing module 241' configured to parse the service data corresponding to the block number; and
数据执行模块242’，其配置为用于根据所述业务数据执行B+树操作或者跳表操作。A data execution module 242' is configured to perform a B+ tree operation or a table jump operation based on the service data.
根据本申请的另一个方面，还提供了一种数据一致性校验***，图10是根据本申请的数据一致性校验***的一个实施例的流程图。该***可以包括主数据库和至少一个从数据库，所述主数据库和和至少一个从数据库连接。主数据库可以包括上述应用于主数据库的数据一致性校验装置中的任意一个。从数 据库可以包括上述应用于从数据库的数据一致性校验装置中的任意一个。In accordance with another aspect of the present application, a data consistency check system is also provided, and FIG. 10 is a flow diagram of one embodiment of a data consistency check system in accordance with the present application. The system can include a primary database and at least one secondary database, the primary database and at least one secondary database connection. The primary database may include any of the above-described data consistency check devices applied to the primary database. The slave database may include any of the above-described data consistency check devices applied to the slave database.
在一个可选的实施方案中，在数据一致性校验***中，所述主数据库用于对所述主数据库中的第一区块集合中的区块数据进行哈希运算，得到第一哈希树数据，并将所述第一哈希树数据发送给所述从数据库；所述从数据库用于将所述第一哈希树数据和第二哈希树数据进行对比后，生成校验结果并发送给所述主数据库，其中，所述第二哈希树是对所述从数据库中与所述第一区块集合相对应的第二区块集合中的区块数据进行哈希运算得到的，所述校验结果可以包括：所述第一区块集合和所述第二区块集合中数据不一致的区块号；并且，所述主数据库还用于根据所述数据不一致的区块号，将所述第一区块集合中该区块号对应的区块数据发送给所述从数据库，以便从数据库根据该区块号对应的区块数据更新所述第二区块集合中相应区块的区块数据。In an optional implementation, in the data consistency check system, the primary database is configured to perform hash operation on the block data in the first block set in the primary database to obtain the first Generating the data and transmitting the first hash tree data to the slave database; the slave database is configured to compare the first hash tree data with the second hash tree data to generate a checksum And sending the result to the primary database, wherein the second hash tree is to hash the block data in the second block set corresponding to the first block set in the database The check result may include: a block number in which the data in the first block set and the second block set are inconsistent; and the primary database is further used in an area according to the data inconsistency a block number, the block data corresponding to the block number in the first block set is sent to the slave database, so that the second block set is updated from the database according to the block data corresponding to the block number. Block data for the corresponding block.
该***能够快速计算主数据库和从数据库之间的数据的一致性，并根据主数据库的数据更新从数据库的数据，达到数据校验和数据备份的作用。The system can quickly calculate the consistency of data between the primary database and the secondary database, and update the data from the database according to the data of the primary database to achieve data verification and data backup.
在一个可选实施方案中，从数据库的数量为3个。在主数据库和从数据库均位于不同物理节点的情况下，主数据库和每个从数据库中间通过有线网络或者无线网络连接。In an alternative embodiment, the number of slave databases is three. In the case where both the primary database and the secondary database are located on different physical nodes, the primary database and each secondary database are connected through a wired network or a wireless network.
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机指令。在计算机加载和执行所述计算机程序指令时，全部或部分地产生按照本申请实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、获取其他可编程装置。所述计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，所述计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线(DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可用介质集成的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质，(例如，软盘、硬盘、磁带)、光介质(例如，DVD)、或者半导体介质(例如固态硬盘Solid State Disk(SSD))等。In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer loads and executes the computer program instructions, the processes or functions described in accordance with embodiments of the present application are generated in whole or in part. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be from a website site, computer, server or data center Transfer to another website site, computer, server, or data center by wire (eg, coaxial cable, fiber optic, digital subscriber line (DSL), or wireless (eg, infrared, wireless, microwave, etc.). The computer readable storage medium can be any available media that can be accessed by a computer or a data storage device such as a server, data center, or the like that includes one or more available media. The usable medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (such as a solid state disk (SSD)).
专业人员应该还可以进一步意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、计算机软件或者二者的结合来实现， 为了清楚地说明硬件和软件的可互换性，在上述说明中已经按照功能一般性地描述了各示例的组成及步骤。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。A person skilled in the art should further appreciate that the elements and algorithm steps of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software or a combination of both, in order to clearly illustrate hardware and software. Interchangeability, the composition and steps of the various examples have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the solution. A person skilled in the art can use different methods to implement the described functions for each particular application, but such implementation should not be considered to be beyond the scope of the present application.
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分步骤是可以通过程序来指令处理器完成，所述的程序可以存储于计算机可读存储介质中，所述存储介质是非短暂性(英文：non-transitory)介质，例如随机存取存储器，只读存储器，快闪存储器，硬盘，固态硬盘，磁带(英文：magnetic tape)，软盘(英文：floppy disk)，光盘(英文：optical disc)及其任意组合。It will be understood by those skilled in the art that all or part of the steps of implementing the above embodiments may be performed by a program, and the program may be stored in a computer readable storage medium, which is non-transitory ( English: non-transitory) media, such as random access memory, read-only memory, flash memory, hard disk, solid state disk, magnetic tape (English: magnetic tape), floppy disk (English: floppy disk), CD (English: optical disc) And any combination thereof.
至此，本领域技术人员应认识到，虽然本文已详尽示出和描述了本申请的多个示例性实施例，但是，在不脱离本申请精神和范围的情况下，仍可根据本申请公开的内容直接确定或推导出符合本申请原理的许多其他变型或修改。因此，本申请的范围应被理解和认定为覆盖了所有这些其他变型或修改。In this regard, those skilled in the art will recognize that the present invention has been described and illustrated in detail herein. The content directly determines or derives many other variations or modifications consistent with the principles of the application. Therefore, the scope of the present application should be understood and construed as covering all such other variations or modifications.

Claims (10)

  1. 一种数据一致性校验方法，应用于主数据库，包括：A data consistency check method applied to a primary database, including:
    哈希树数据发送步骤：对所述主数据库中的第一区块集合中的区块数据进行哈希运算，得到第一哈希树数据，并将所述第一哈希树数据发送给从数据库；Hash tree data sending step: hashing block data in the first block set in the primary database, obtaining first hash tree data, and transmitting the first hash tree data to the slave database;
    校验结果接收步骤：在所述从数据库将所述第一哈希树数据和第二哈希树数据进行对比后，接收所述从数据库发送的校验结果，其中，所述第二哈希树是对所述从数据库中与所述第一区块集合相对应的第二区块集合中的区块数据进行哈希运算得到的，所述校验结果包括：所述第一区块集合和所述第二区块集合中数据不一致的区块号；和a verification result receiving step: after comparing the first hash tree data and the second hash tree data by the slave database, receiving the verification result sent from the database, wherein the second hash The tree is obtained by hashing the block data in the second block set corresponding to the first block set in the database, and the check result includes: the first block set a block number that is inconsistent with data in the second block set; and
    区块数据发送步骤：根据所述数据不一致的区块号，将所述第一区块集合中该区块号对应的区块数据发送给所述从数据库，以便从数据库根据该区块号对应的区块数据更新所述第二区块集合中相应区块的区块数据。a block data sending step: transmitting block data corresponding to the block number in the first block set to the slave database according to the block number inconsistent with the data, so as to correspond to the slave block according to the block number The block data updates the block data of the corresponding block in the second block set.
  2. 根据权利要求1所述的方法，其特征在于，所述区块是将所述主数据库的数据或者所述从数据库的数据以固定的时间单元分块存储得到的。The method according to claim 1, wherein said block is obtained by storing data of said primary database or said data of said secondary database in blocks of fixed time units.
  3. 一种数据一致性校验方法，应用于从数据库，包括：A data consistency check method applied to a database, including:
    哈希树数据接收步骤：在主数据库对所述主数据库中的第一区块集合中的区块数据进行哈希运算，得到第一哈希树数据后，接收所述主数据库发送的所述第一哈希树数据；a hash tree data receiving step: performing hashing on the block data in the first block set in the primary database in the primary database to obtain the first hash tree data, and receiving the First hash tree data;
    校验结果生成步骤：将所述第一哈希树数据和第二哈希树数据进行对比，生成校验结果并发送给所述主数据库，其中，所述第二哈希树是对所述从数据库中与所述第一区块集合相对应的第二区块集合中的区块的数据进行哈希运算得到的，所述校验结果包括：所述第一区块集合和所述第二区块集合中数据不一致的区块号；a verification result generating step: comparing the first hash tree data with the second hash tree data, generating a verification result and transmitting the verification result to the primary database, wherein the second hash tree is for the And performing a hash operation on the data of the block in the second block set corresponding to the first block set in the database, where the check result includes: the first block set and the first The block number in which the data in the second block set is inconsistent;
    区块数据接收步骤：接收所述主数据库根据所述数据不一致的区块号发送的所述第一区块集合中该区块号对应的区块数据；和a block data receiving step: receiving, by the primary database, block data corresponding to the block number in the first block set sent according to the block number inconsistent with the data; and
    数据更新步骤：根据该区块号对应的区块数据更新所述第二区块集合中相应区块的区块数据。The data updating step: updating the block data of the corresponding block in the second block set according to the block data corresponding to the block number.
  4. 根据权利要求3所述的方法，其特征在于，所述区块数据为操作日志，优选地，所述操作日志为二进制日志。The method according to claim 3, wherein the block data is an operation log, and preferably the operation log is a binary log.
  5. 根据权利要求3所述的方法，其特征在于，所述数据更新步骤包括：The method of claim 3 wherein said step of updating data comprises:
    数据解析步骤：解析该区块号对应的操作日志；Data parsing step: parsing an operation log corresponding to the block number;
    数据执行步骤：根据所述操作日志执行数据库操作；和Data execution step: performing a database operation according to the operation log; and
    日志更新步骤：更新所述从数据库的操作日志。Log update step: update the operation log of the slave database.
  6. 根据权利要求3所述的方法，其特征在于，所述区块数据为业务数据。The method of claim 3 wherein said block data is business data.
  7. 根据权利要求6所述的方法，其特征在于，所述数据更新步骤包括：The method of claim 6 wherein said step of updating data comprises:
    数据解析步骤：解析该区块号对应的业务数据；和Data parsing step: parsing the business data corresponding to the block number; and
    数据执行步骤：根据所述业务数据执行B+树(B+Tree)操作或者跳表(Skip List)操作。Data execution step: performing a B+Tree (B+Tree) operation or a Skip List operation according to the service data.
  8. 一种数据一致性校验装置，应用于主数据库，包括：A data consistency verification device is applied to a primary database, including:
    哈希树数据发送模块，其配置为用于对所述主数据库中的第一区块集合中的区块数据进行哈希运算，得到第一哈希树数据，并将所述第一哈希树数据发送给从数据库；a hash tree data sending module, configured to perform hash operation on the block data in the first block set in the primary database, to obtain first hash tree data, and to use the first hash Tree data is sent to the slave database;
    校验结果接收模块，其配置为用于在所述从数据库将所述第一哈希树数据和第二哈希树数据进行对比后，接收所述从数据库发送的校验结果，其中，所述第二哈希树是对所述从数据库中与所述第一区块集合相对应的第二区块集合中的区块数据进行哈希运算得到的，所述校验结果包括：所述第一区块集合和所述第二区块集合中数据不一致的区块号；和a verification result receiving module, configured to receive the verification result sent by the slave database after the first hash tree data and the second hash tree data are compared by the slave database, where The second hash tree is obtained by hashing the block data in the second block set corresponding to the first block set in the database, and the check result includes: a block number in which the data in the first block set and the second block set are inconsistent; and
    区块数据发送模块，其配置为用于根据所述数据不一致的区块号，将所述第一区块集合中该区块号对应的区块数据发送给所述从数据库，以便从数据库根据该区块号对应的区块数据更新所述第二区块集合中相应区块的区块数据。a block data sending module, configured to send, according to the block number of the data inconsistency, block data corresponding to the block number in the first block set to the slave database, so as to be based on the database The block data corresponding to the block number updates the block data of the corresponding block in the second block set.
  9. 一种数据一致性校验装置，应用于从数据库，包括：A data consistency check device, applied to a slave database, comprising:
    哈希树数据接收模块，其配置为用于在主数据库对所述主数据库中的第一区块集合中的区块数据进行哈希运算，得到第一哈希树数据后，接收所述主数据库发送的所述第一哈希树数据；a hash tree data receiving module, configured to perform hash operation on the block data in the first block set in the primary database in the primary database, to obtain the first hash tree data, and receive the primary The first hash tree data sent by the database;
    校验结果生成模块，其配置为用于将所述第一哈希树数据和第二哈希树数据进行对比，生成校验结果并发送给所述主数据库，其中，所述第二哈希树是对所述从数据库中与所述第一区块集合相对应的第二区块集合中的区块的数 据进行哈希运算得到的，所述校验结果包括：所述第一区块集合和所述第二区块集合中数据不一致的区块号；a verification result generating module configured to compare the first hash tree data and the second hash tree data, generate a verification result, and send the verification result to the primary database, where the second hash The tree is obtained by hashing data of the block in the second block set corresponding to the first block set in the database, and the check result includes: the first block a block number that is inconsistent with the data in the set of the second block;
    区块数据接收模块，其配置为用于接收所述主数据库根据所述数据不一致的区块号发送的所述第一区块集合中该区块号对应的区块数据；和a block data receiving module, configured to receive block data corresponding to the block number in the first block set sent by the primary database according to the block number inconsistent with the data; and
    数据更新模块，其配置为用于根据该区块号对应的区块数据更新所述第二区块集合中相应区块的区块数据。And a data update module configured to update the block data of the corresponding block in the second block set according to the block data corresponding to the block number.
  10. 一种数据一致性校验***，包括主数据库和至少一个从数据库，所述主数据库与所述至少一个从数据库相连接，其中：A data consistency verification system includes a primary database and at least one secondary database, the primary database being coupled to the at least one secondary database, wherein:
    所述主数据库用于对所述主数据库中的第一区块集合中的区块数据进行哈希运算，得到第一哈希树数据，并将所述第一哈希树数据发送给所述从数据库；The primary database is configured to perform hash operation on the block data in the first block set in the primary database, obtain first hash tree data, and send the first hash tree data to the From the database;
    所述从数据库用于将所述第一哈希树数据和第二哈希树数据进行对比后，生成校验结果并发送给所述主数据库，其中，所述第二哈希树是对所述从数据库中与所述第一区块集合相对应的第二区块集合中的区块数据进行哈希运算得到的，所述校验结果包括：所述第一区块集合和所述第二区块集合中数据不一致的区块号；并且The slave database is configured to compare the first hash tree data with the second hash tree data, generate a verification result, and send the verification result to the primary database, where the second hash tree is a The hash data in the second block set corresponding to the first block set in the database is hashed, and the check result includes: the first block set and the first a block number in which the data in the second block set is inconsistent; and
    所述主数据库还用于根据所述数据不一致的区块号，将所述第一区块集合中该区块号对应的区块数据发送给所述从数据库，以便从数据库根据该区块号对应的区块数据更新所述第二区块集合中相应区块的区块数据。The primary database is further configured to send the block data corresponding to the block number in the first block set to the slave database according to the block number of the data inconsistency, so as to be based on the block number from the database. The corresponding block data updates the block data of the corresponding block in the second block set.
PCT/CN2018/079861 2018-03-21 2018-03-21 Database master-slave block consistency check method, device and system WO2019178772A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/CN2018/079861 WO2019178772A1 (en) 2018-03-21 2018-03-21 Database master-slave block consistency check method, device and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2018/079861 WO2019178772A1 (en) 2018-03-21 2018-03-21 Database master-slave block consistency check method, device and system

Publications (1)

Publication Number Publication Date
WO2019178772A1 true WO2019178772A1 (en) 2019-09-26

Family

ID=67988034

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/079861 WO2019178772A1 (en) 2018-03-21 2018-03-21 Database master-slave block consistency check method, device and system

Country Status (1)

Country Link
WO (1) WO2019178772A1 (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111694852A (en) * 2020-05-28 2020-09-22 平安科技（深圳）有限公司 Data processing method, device, terminal and storage medium based on distributed transaction
CN113297235A (en) * 2020-11-03 2021-08-24 阿里巴巴集团控股有限公司 Data consistency checking method and device for distributed database cluster
WO2023091203A1 (en) * 2021-11-19 2023-05-25 Marqeta, Inc. Generating cryptographic proof of a series of transactions
CN117251460A (en) * 2023-08-10 2023-12-19 上海栈略数据技术有限公司 Data consistency check system for graph database and relational database

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102054035A (en) * 2010-12-29 2011-05-11 北京播思软件技术有限公司 Data range-based method for synchronizing data in database
CN103164523A (en) * 2013-03-19 2013-06-19 华为技术有限公司 Inspection method, device and system of data consistency inspection
CN103902617A (en) * 2012-12-28 2014-07-02 华为技术有限公司 Distributed database synchronization method and system
CN104731792A (en) * 2013-12-19 2015-06-24 ***股份有限公司 Method and system for verifying database consistency and method and system for positioning data difference

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102054035A (en) * 2010-12-29 2011-05-11 北京播思软件技术有限公司 Data range-based method for synchronizing data in database
CN103902617A (en) * 2012-12-28 2014-07-02 华为技术有限公司 Distributed database synchronization method and system
CN103164523A (en) * 2013-03-19 2013-06-19 华为技术有限公司 Inspection method, device and system of data consistency inspection
CN104731792A (en) * 2013-12-19 2015-06-24 ***股份有限公司 Method and system for verifying database consistency and method and system for positioning data difference

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111694852A (en) * 2020-05-28 2020-09-22 平安科技（深圳）有限公司 Data processing method, device, terminal and storage medium based on distributed transaction
CN113297235A (en) * 2020-11-03 2021-08-24 阿里巴巴集团控股有限公司 Data consistency checking method and device for distributed database cluster
CN113297235B (en) * 2020-11-03 2023-12-19 阿里巴巴集团控股有限公司 Data consistency checking method and device for distributed database cluster
WO2023091203A1 (en) * 2021-11-19 2023-05-25 Marqeta, Inc. Generating cryptographic proof of a series of transactions
CN117251460A (en) * 2023-08-10 2023-12-19 上海栈略数据技术有限公司 Data consistency check system for graph database and relational database
CN117251460B (en) * 2023-08-10 2024-04-05 上海栈略数据技术有限公司 Data consistency check system for graph database and relational database

Similar Documents

Publication Publication Date Title
WO2019178772A1 (en) Database master-slave block consistency check method, device and system
US10185497B2 (en) Cluster federation and trust in a cloud environment
US9405781B2 (en) Virtual multi-cluster clouds
CN102098342B (en) Transaction level-based data synchronizing method, device thereof and system thereof
EP2919130A1 (en) Method and system for synchronizing distributed database
US10140351B2 (en) Method and apparatus for processing database data in distributed database system
US11354065B2 (en) Cloud edition and retrieve
US20130061084A1 (en) Massively Scalable Object Storage
CN102012944B (en) Distributed NOSQL (not only structured query language) database capable of providing replication property
JP5548829B2 (en) Computer system, data management method, and data management program
US11068499B2 (en) Method, device, and system for peer-to-peer data replication and method, device, and system for master node switching
US10712964B2 (en) Pre-forking replicas for efficient scaling of a distributed data storage system
US11663192B2 (en) Identifying and resolving differences between datastores
CN106326239A (en) Distributed file system and file meta-information management method thereof
JP2015035020A (en) Storage system, storage control device, and control program
KR101527634B1 (en) Method and apparatus for providing sharding service
CN113656384A (en) Data processing method, distributed database system, electronic device and storage medium
KR101748913B1 (en) Cluster management method and data storage system for selecting gateway in distributed storage environment
JP2013134522A (en) Data management system, data management method, and data management program
CN117478300B (en) Cross-chain consensus method and device based on node determination and computer equipment
CN114185715A (en) Data recovery method and device
CN109302326A (en) A kind of configuration method and device of qos policy
JP2019003584A (en) Database server, database management method, and program
JP2018163572A (en) Storage system and communication method

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18911099

Country of ref document: EP

Kind code of ref document: A1