CN117332009A

CN117332009A - Data processing method and related equipment

Info

Publication number: CN117332009A
Application number: CN202311331328.7A
Authority: CN
Inventors: 秦波; 蓝虎; 张慧; 梁作栋; 黄桂鸿; 卢江玮; 陈自民; 莫洋; 高聪; 范滔; 佘仕好
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2023-10-13
Filing date: 2023-10-13
Publication date: 2024-01-02

Abstract

The embodiment of the application provides a data processing method and related equipment, wherein the method is applied to a block chain data synchronization system, N data reading components are arranged in the block chain data synchronization system, and N is a positive integer; each data reading component is for reading data from a blockchain of one chain type, the method comprising: determining a target block chain to be synchronized; determining a target data reading component which is matched with the target blockchain from N data reading components according to the chain type of the target blockchain; invoking a target data reading component to read data to be synchronized from a target block chain; and writing the read data into a database. The complexity of the data synchronization system can be effectively reduced through the method, so that the synchronization cost and the maintenance cost are reduced, and the stability of the block chain data synchronization is improved.

Description

Data processing method and related equipment

Technical Field

The present invention relates to the field of computer technology, and in particular, to a data processing method and related device, a data processing apparatus, a computer device, a computer readable storage medium, and a computer program product.

Background

The data can be processed by each node in the blockchain network to generate the blockchain and realize the safe storage of the data. With the operation of the blockchain network, the data volume of each node in the blockchain network is larger and larger, and for some high-frequency use scenes of data and scenes with higher requirements for inquiry delay, such as exchanges, browsing of blocks and transactions, data statistical analysis and the like, the cost of inquiring data through the whole node of the operation blockchain network is higher and higher, and the requirements cannot be well met. To improve ease and efficiency of querying, the blockchain network may interact with a database (a centralized storage) to synchronize data on the blockchain to the database, thereby querying the data directly from the database. However, the types of blockchains and databases are very numerous, and when the types of blockchains or databases are more and more accessed, the complexity of the whole data synchronization system is higher and higher, the stability of the blockchain data synchronization is affected, and the synchronization cost and the maintenance cost are also increased.

Disclosure of Invention

The embodiment of the application provides a data processing method and related equipment, which can reduce the complexity of a data synchronization system, thereby reducing the synchronization cost and the maintenance cost and improving the stability of block chain data synchronization.

In one aspect, an embodiment of the present application provides a data processing method, where the method is applied to a blockchain data synchronization system, where N data reading units are provided in the blockchain data synchronization system, where N is a positive integer; each data reading component is for reading data from a blockchain of one chain type; the method comprises the following steps:

determining a target block chain to be synchronized;

determining a target data reading component which is matched with the target blockchain from N data reading components according to the chain type of the target blockchain;

invoking a target data reading component to read data to be synchronized from a target blockchain

In one aspect, an embodiment of the present application provides a data processing apparatus, including:

a determining unit for determining a target blockchain to be synchronized;

the determining unit is further used for determining target data reading components which are matched with the target block chain from N data reading components according to the chain type of the target block chain;

the processing unit is used for calling a target data reading component and reading data to be synchronized from a target block chain; the method comprises the steps of,

and the processing unit is also used for writing the read data into the database.

In one aspect, embodiments of the present application provide a computer device comprising:

A processor adapted to execute a computer program;

a computer readable storage medium having a computer program stored therein, which when executed by a processor, implements a data processing method as described above.

Accordingly, embodiments of the present application provide a computer readable storage medium having a computer program stored therein, the computer program being loaded by a processor and performing a data processing method as described above.

Accordingly, embodiments of the present application provide a computer program product comprising a computer program or computer instructions which, when executed by a processor, implement the above-described data processing method.

In the embodiment of the application, N data reading components are arranged in the block chain data synchronization system, wherein N is a positive integer; each data reading component is configured to read data from a blockchain of one chain type, and it can be seen that the blockchain data synchronization system has a data reading function. In the method, the target block chain to be synchronized can be determined, and the target data reading component matched with the target block chain is determined from N data reading components according to the chain type of the target block chain, so that a special data reading component is determined for the block chain of the chain type, and the target data reading component can be called to read the data to be synchronized from the target block chain and write the read data into a database. The data to be synchronized in the target blockchain is now synchronized into the database. The block chain and the database are respectively accessed through the block chain data synchronization system, the block chain does not need to be directly in butt joint with the database, but the block chain data synchronization system is used as a middleware between the block chain and the database to realize data synchronization, and the data synchronization system can be greatly simplified, so that the maintenance cost of the system and the synchronization cost of the data are effectively reduced. By means of the data reading components arranged in the block chain data synchronization system, for the block chains to be synchronized with the same chain type, the same data reading component can be determined from the block chain data synchronization system to read data, and for the block chains to be synchronized with different chain types, different data reading components can be determined and independently read data. Thus, the block chain data synchronization system can cope with block chains of various chain types, data synchronization of the block chains of various chain types is not interfered due to the quantity change of the block chains, and the data synchronization efficiency and the stability of the block chain data synchronization can be improved.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1a is a block diagram of a data processing system according to one illustrative embodiment of the present application;

FIG. 1b is an architecture diagram of a distributed system provided in an exemplary embodiment of the present application;

FIG. 1c is an alternative schematic diagram of a block structure provided in an exemplary embodiment of the present application;

FIG. 2 is a flow chart of a method of data processing according to an exemplary embodiment of the present application;

FIG. 3 is a schematic diagram of a data processing flow provided in an exemplary embodiment of the present application;

FIG. 4 is a flow chart of a method for processing data according to another exemplary embodiment of the present application;

FIG. 5 is a flow diagram of blockchain data synchronization provided by an exemplary embodiment of the present application;

FIG. 6 is a flow chart of a method of data processing according to another exemplary embodiment of the present application;

FIG. 7a is a schematic diagram of an interactive flow of data synchronization provided in an exemplary embodiment of the present application;

FIG. 7b is a schematic diagram of an application of a data synchronization system according to an exemplary embodiment of the present application;

FIG. 8 is a block chain data synchronization system according to one exemplary embodiment of the present application;

FIG. 9a is a schematic diagram of a block chain data synchronization system according to another exemplary embodiment of the present application;

FIG. 9b is a schematic diagram of a blockchain data synchronization system provided in accordance with another exemplary embodiment of the present application;

FIG. 10 is a schematic diagram of a data processing apparatus according to an exemplary embodiment of the present application;

fig. 11 is a schematic structural diagram of a computer device according to an exemplary embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.

The present application provides a data processing scheme, which relates to a data processing system, a method and related equipment, and the scheme aims at a target blockchain to be synchronized, and can determine a target data reading component adapted to the target blockchain from N (N is a positive integer) data reading components arranged in the blockchain data synchronizing system according to the chain type of the target blockchain. Therefore, the data on the target blockchain is synchronized into the database, so that the data in the blockchain can be directly inquired from the database under the high-frequency use scene of the data and the scene with higher inquiry delay requirement, and the requirements of low delay and high-efficiency inquiry can be better met.

The block chain data synchronization system has the function of data reading and serves as a middleware for synchronizing the block chain to the database, so that direct butt joint between the block chain and the database can be avoided, and the overall complexity of the system is reduced. For any chain type of blockchain, the data to be synchronized can be read through a data reading component which is matched in a blockchain data synchronization system; for blockchains of different chain types, different data reading components in the blockchain data synchronization system are adopted to read data in corresponding blockchains. The block chain data synchronization system can cope with block chains of various chain types, and realizes the stable and efficient data synchronization function from the block chains of different chain types to the database.

The architecture of a data processing system provided in embodiments of the present application will be described below with reference to the accompanying drawings.

With reference now to FIG. 1a, an architecture diagram of a data processing system is presented in accordance with one illustrative embodiment of the present application. As shown in FIG. 1a, the data processing system includes a computer device 101, a blockchain network 102, and a database 103. The computer device 101 may be one or both of a terminal device and a server. Terminal devices include, but are not limited to: smart phone, tablet computer, intelligent wearable equipment, intelligent voice interaction equipment, intelligent household electrical appliances, notebook computer, vehicle-mounted terminal, intelligent camera scheduling equipment, this application does not limit in this. The number of terminal devices is not limited in this application. The server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network ), basic cloud computing services such as big data and artificial intelligent platform, but is not limited thereto. The present application is not limited with respect to the number of servers.

A blockchain data synchronization system is deployed in the computer device 101. The blockchain data synchronization system may be considered a blockchain data synchronization tool (which may be referred to as SPV-X) that may be used to synchronize data in a blockchain to be synchronized into a database. In one possible implementation, the blockchain data synchronization system belongs to a system with an embedded development structure, and can be run in a computer device in a process mode, and the blockchain data synchronization system can be constructed by adopting a plug-in development architecture, namely a Framework plus plug in. By means of the advantages of low development difficulty of the plug-in development architecture, low coupling degree inside the system module and the like, various components arranged in the block chain data synchronization system can be flexibly added and deleted according to specific service requirements, the block chain data synchronization system is easy to expand, and the maintenance cost is low.

N data reading components are arranged in the block chain data synchronization system, wherein N is a positive integer. Each data reading component is for reading data from a blockchain of one chain type, and a different data reading component is for reading data in a blockchain of a different chain type. Illustratively, the data reading component sp1 is for reading data in the blockchain of the chain type1, and the data reading component sp2 is for reading data in the blockchain of the chain type 2. The data reading component may also be referred to as a data reading module or a data reading plug-in (SPV-plug in), which is abstracted from a blockchain data source, and illustratively, any data reading component may be specifically a light node (SPV) in a blockchain network to which a blockchain of an adapted chain type belongs, such as a coalition chain SPV, based on the chain type to which the data reading component is adapted. The data reading component may read data in the target blockchain from any of the identified nodes of the blockchain network.

The data processing flow performed by computer device 101 may generally include: the method comprises the steps of firstly determining a target blockchain to be synchronized, wherein the target blockchain is one blockchain accessed by a blockchain data synchronization system, then determining a target data reading component which is matched with the target blockchain from N data reading components based on the chain type of the target blockchain, and calling the target data reading component to read data to be synchronized in the target blockchain. Specifically, data to be synchronized in the target blockchain may be read from the blockchain network. Thus, for the target blockchain, the exclusive target data reading component can be adopted to read the data in the target blockchain, so that for blockchains of different chain types, the data in the blockchains of the corresponding chain types can be respectively read based on the exclusive data reading components, and the independent processing of the blockchains of the various chain types can be realized. And then, the read data can be written into a database to realize the synchronization of the data in the target block chain. In the embodiments of the present application, the database may also be referred to as a destination or centralized storage. The number of databases written may be one or more.

In one possible implementation, the blockchain-based data synchronization system may provide data synchronization configuration functionality to the object to meet the synchronization requirements of the object personalization. For example, the blockchain to be synchronized, the database to which the data is to be written, the data read rate, the data write rate, and some general processing (e.g., filtering) may be specified by the data synchronization configuration functions provided. Based on this, the manner in which the computer device determines the target blockchain to synchronize may include the following: (1) acquiring synchronous configuration information, and determining a blockchain identified by a chain identification in the synchronous configuration information as a target blockchain; (2) and determining a target block chain to be synchronized according to the service requirement. For example, where business requirements indicate a transaction requiring a query for a digital asset, the blockchain used to record the transaction data for the digital asset may be the target blockchain. For another example, if the service requirement indicates that the blockchain with larger data size is synchronized, the data amount of each accessed blockchain can be compared, and the blockchain with the largest data amount can be determined to be the target blockchain, so that the blockchain with the large data amount can be synchronized by the centralized resource priority.

As an implementation manner, after receiving the configured instruction, the computer device may start the blockchain data synchronization system, and then may establish communication connection between the blockchain data synchronization system and each blockchain network and each database, or after determining the target blockchain to be synchronized, establish communication connection between the blockchain data synchronization system and the blockchain network where the target blockchain is located and the designated target database.

The blockchain network 102 may be a distributed system formed by clients, multiple nodes (any form of computing device in an access network, such as servers, terminals) connected by a form of network communication. Taking the distributed system as an example of a blockchain network, referring to fig. 1b, fig. 1b is a schematic structural diagram of an alternative application of the distributed system 100 provided in the embodiment of the present application to the blockchain network, where the network is formed by a plurality of nodes (any type of computing devices in an access network, such as servers and terminals) and clients, and a peer-to-peer network formed by the nodes is formed, and a peer-to-peer protocol is an application layer protocol running on top of a transmission control protocol (TCP, transmission Control Protocol) protocol. In a distributed system, any machine, such as a server, a terminal, may join to become a node, including a hardware layer, an intermediate layer, an operating system layer, and an application layer.

Referring to the functionality of each node in the blockchain network shown in fig. 1b, the functionality involved includes:

1) The routing, the node has basic functions for supporting communication between nodes.

Besides the routing function, the node can also have the following functions:

2) The application is used for being deployed in a blockchain to realize specific service according to actual service requirements, recording data related to the realization function to form recorded data, carrying a digital signature in the recorded data to represent the source of task data, sending the recorded data to other nodes in the blockchain network, and adding the recorded data into a temporary block when the source and the integrity of the recorded data are verified by the other nodes.

For example, the services implemented by the application include:

2.1 A resource client for implementing a resource management business function that may include functions of transacting digital assets, including initiating transactions (i.e., sending a transaction record of a current transaction to other nodes in the blockchain network, the other nodes, after verification, storing record data of the transaction in a temporary block of the blockchain in response to acknowledging that the transaction is valid; of course, the resource client also supports querying the digital assets remaining in the digital asset address;

2.2 The shared account book is used for providing the functions of storing, inquiring, modifying and the like of account data, sending record data of the operation on the account data to other nodes in the blockchain network, and after the other nodes verify validity, storing the record data into a temporary block as a response for acknowledging that the account data is valid, and can also send confirmation to the node initiating the operation.

2.3 A computerized agreement that can execute the terms of a contract, implemented by code deployed on a shared ledger for execution when certain conditions are met, for completing automated transactions based on actual business demand codes, such as querying the physical distribution status of the goods purchased by the buyer, transferring the buyer's digital asset to the merchant's address after the buyer signs for the goods; of course, the smart contract is not limited to executing the contract for the transaction, and may execute a contract that processes the received information.

3) The blockchain comprises a series of blocks (blocks) which are mutually connected according to the generated sequence time, the new blocks can not be removed once being added into the blockchain, and record data submitted by nodes in the blockchain network are recorded in the blocks. Blockchains (or blockchain) are concatenated literal records (also known as blocks) that are cryptographically concatenated and protected from content. Each block includes a cryptographic hash of the previous block, a corresponding time stamp, and transaction data (typically represented by a hash value calculated using the Merkle tree algorithm), which is designed to make the contents of the block difficult to tamper with, and the distributed ledgers concatenated using the blockchain technique enable both parties to record transactions efficiently and to permanently verify the transactions.

Referring to fig. 1c, fig. 1c is an optional schematic diagram of a Block Structure (Block Structure) provided in the embodiment of the present application, where each Block includes a hash value of a transaction record stored in the Block (hash value of the Block) and a hash value of a previous Block, and the blocks are connected by the hash values to form a blockchain. In addition, the block may include information such as a time stamp at the time of block generation. The Blockchain (Blockchain), which is essentially a de-centralized database, is a string of data blocks that are generated in association using cryptographic methods, each of which contains associated information that is used to verify the validity (anti-counterfeiting) of its information and to generate the next block.

The blockchain may be generated and maintained through a blockchain network. The number of the blockchain networks is not limited, if a plurality of blockchain networks are included, blockchains with different chain types can be generated in different blockchain networks, and one blockchain network can generate blockchains with one chain type. Database 103 is used to store written data into which data read from the blockchain may be written. The number of databases is not limited herein, and one database may be used to store data in blockchains of one or more chain types, and a different database may be used to store data in blockchains of the same chain type.

According to the data processing system provided by the embodiment of the application, the computer equipment can determine the target block chain to be synchronized, and determine the target data reading component matched with the target block chain from N data reading components arranged in the block chain data synchronizing system according to the chain type of the target block chain, and because each data reading component can be used for reading data from the block chain of one chain type, the target data reading component can be called to read the data to be synchronized from the target block chain, and the read data is written into the database. In this way, synchronization of data on the target blockchain into the database is completed. The data reading component based on the block chain data synchronization system can support the reading of the data of the block chains of various chain types, and the data reading among the block chains of different chain types is not interfered with each other, so that the stable and efficient synchronization of the block chains of different chain types to the database is realized. The blockchain network and the database in the data processing system do not need direct communication, but the blockchain data synchronization system is used as a middleware to read the data in each blockchain and orderly write the data into the database, so that the complexity of the system can be reduced, the maintenance cost and the synchronization cost are reduced, and the computer equipment invokes the blockchain data system to perform the synchronization of the blockchain data in an overall mode, so that the stability of the data synchronization can be improved.

Next, a description will be given of a data processing method provided in an embodiment of the present application.

Referring to fig. 2, fig. 2 is a flow chart of a data processing method according to an exemplary embodiment of the present application. The data processing method can be applied to a block chain data synchronization system, wherein N data reading components are arranged in the block chain data synchronization system, and N is a positive integer; each data reading component is for reading data from a blockchain of one chain type. The data processing method may be performed by a computer device, such as the computer device 101 in fig. 1a, and may include what is described in the following S201-S204.

S201, determining a target blockchain to be synchronized.

The target blockchain to be synchronized refers to any one of at least one blockchain accessed by the blockchain data synchronization system. The blockchain data synchronization system accessing the blockchain is understood to be that the blockchain data synchronization system establishes communication connection with a blockchain network where the blockchain is located, and data in each blockchain supports synchronization to a database through the blockchain data synchronization system. At least one blockchain refers to one or more chain types, one chain type may correspond to one or more blockchains, and the chain type of each blockchain. In one embodiment, the chain type is determined from the composition of data on the blockchain, illustratively, different chain types correspond to different encrypted assets. In another possible implementation, the chain type is determined according to the disclosure level of the blockchain, and illustratively private chains, public chains, federated chains, and semi-federated chains, etc., belong to blockchains of different chain types. The private chain refers to a blockchain which is opened to an independent object, the public chain refers to a blockchain which is opened to each object, the alliance chain refers to a blockchain which is opened to a specific object, and compared with the data disclosure of the public chain and the free joining and exiting of nodes, the alliance chain is that the inside members of the alliance have authority to access the data, the nodes are permitted to join the network, the faster transaction speed can be realized based on the alliance chain, and a plurality of independent account books can be established.

In one implementation, each blockchain has a chain identification, one for uniquely identifying one blockchain, which may be generated using a hash algorithm. Illustratively, the chain is identified as the address of the blockchain. When determining a target blockchain to be synchronized, the computer equipment can firstly acquire synchronization configuration information, wherein the synchronization configuration information refers to information configured by the synchronization blockchain, the target blockchain is determined according to a chain identifier in the synchronization configuration information, and the blockchain identified by the chain identifier in the synchronization configuration information can be determined to be the target blockchain. That is, the computer device may determine the object-specified blockchain as the target blockchain. If the synchronization configuration information includes a plurality of chain identifications, the blockchain identified by each chain identification can be used as a target blockchain, and the synchronization processing is performed according to the processing flow shown in the embodiment of the application. In another implementation, the computer device may determine a target blockchain to synchronize according to traffic requirements. For example, where business requirements indicate a transaction requiring a query for a digital asset, the blockchain used to record the transaction data for the digital asset may be the target blockchain. For another example, where the traffic demand indicates a blockchain with a largest amount of synchronized data, the computer device may detect the amount of data for each blockchain and may determine the blockchain with the largest amount of data corresponding to the detected data to be synchronized among the plurality of blockchains as the target blockchain. In this way, the computer device can synchronize data in the blockchain that has a large amount of data first.

S202, determining a target data reading component which is matched with the target blockchain from N data reading components according to the chain type of the target blockchain.

In a specific implementation, based on the function of the data reading units, the N data reading units may correspond to N chain types, where the chain type of the target blockchain is one of the N chain types, so that the computer device may find a data reading unit corresponding to the chain type of the target blockchain from the N data reading units according to the chain type of the target blockchain, and determine the data reading unit as a target data reading unit adapted to the target blockchain, where the target data reading unit refers to one data reading unit of the N data reading units, and the adapting refers to that the chain type of the blockchain supported by the target data reading unit is the same as the chain type of the target blockchain.

It can be appreciated that, for each blockchain to be synchronized, the respective adapted data reading component can be determined in the above manner, so that, for blockchains belonging to the same chain type, the same data reading component is determined, and for blockchains belonging to different chain types, different data reading components are determined, so that blockchains of various chain types can be handled by the data reading component specific to various chain types can be provided through the blockchain data synchronization system, the reading efficiency is improved, and the data synchronization process is accelerated.

S203, calling a target data reading component, reading data to be synchronized from a target blockchain, and writing the read data into a database.

The data to be synchronized may include at least one of: the data which is not synchronized needs to be synchronized again. The data to be re-synchronized may be history data that is not successfully synchronized, such as data that has been historically read but not successfully written into the database. The data to be synchronized may be data in individual blocks on the target blockchain, including block header and data in the block. In one implementation, if the computer device first invokes the target data reading component to read data to be synchronized from the target blockchain, the read data is all data in the target blockchain, and if the computer device does not first invoke the target data reading component to read data to be synchronized from the target blockchain, the read data may include data in the target blockchain that has not been synchronized.

The computer device may write the read data to one or more databases, which may be a database determined based on a pre-configured library identification, a database adapted to the read data, or a database adapted to the target blockchain, as the application is not limited in this respect. From the library types of the databases, either database may be a relational database or a non-relational database. The relational database is a database which adopts a relational model (such as a two-dimensional table model) to organize data. Such as Mysql (a relational database management system), SQL Server (an extensible, high-performance, database management system designed for distributed client/Server computing), oracle (an efficient, reliable, high-throughput-compliant database), and PostgreSQL (an object-relational database management system that can be used to securely store data); non-relational databases (i.e., noSQL databases) refer to data storage systems such as MongoDB (a database based on distributed file storage), dis (Remote Dictionary Server, remote dictionary services, a key-value (value) storage system), and HBase (a distributed, array-oriented open source database), etc., that are non-relational, distributed, and generally do not guarantee the nature of ACID (atomic, or insertibility), consistency, isolation, and persistence that is required to write data. The database may also be a distributed database, such as HDFS (Hadoop Distributed File System), which is a file system for storing files, locating files by directory trees; secondly, based on the distributed characteristic, the database is combined by a plurality of servers to realize the functions, and the servers in the cluster have respective roles. Also, for example, HBase, which is a distributed, multi-version-supported NoSQL database built on top of HDFS, appears as a short board that complements the ability to quickly interrogate data in large data scenarios, is suitable for storing hot data in a platform and provides interrogation functionality. By hot data is meant data that is accessed frequently, such as: data having an access frequency greater than a preset access frequency, which refers to the number of accesses in between units, illustratively 1 second for example, is considered hot data, and 100 times/second for a preset access frequency, if the data is accessed 1000 times in 1 second, that is, it is indicated that the access frequency is 1000 times/second, it can be determined that the data is hot data. Such data is business critical data that needs to be accessed and processed quickly and efficiently and therefore needs to be stored in a high performance, low latency database such as the HDFS database described above.

In one implementation, when the computer device writes the read data into the plurality of databases, as a specific implementation, the computer device may divide the read data into a plurality of corresponding data according to the capacity of each database, and write the plurality of corresponding data into the plurality of databases, so that the read data is written into the plurality of databases in a scattered manner, and in a query scenario, the required data may be queried only by querying more than two databases, but based on the portion of the read data stored in each database, the required data may be queried without traversing all the data, thereby improving the query efficiency. As another specific implementation, the computer device may write the read data into multiple databases, respectively. Thus, the plurality of databases store the same read data, and the complete data can be queried only by accessing one database under the query scene, so that the plurality of databases are not needed to be accessed; and the multiple databases can realize the backup of the read data, so that other databases can be used as backup databases corresponding to the target blockchain under the condition that any database fails, and the normal operation of the data query service based on the target blockchain is ensured. Therefore, the data on the target blockchain can be directly inquired from the database in some business scenes by synchronizing the data on the target blockchain into the database, and the data on the target blockchain can be acquired without running each node in the blockchain network, so that the efficiency of data inquiry can be improved.

It will be appreciated that for any of the blockchains to be synchronized, the synchronization of the data in the blockchain to the database may be implemented as described in the foregoing description of S201-S203, see a schematic flow chart of an exemplary data process shown in fig. 3. As shown in fig. 3, synchronization of blockchains of different chain types is involved, and each blockchain corresponds to a data reading component for processing, if the target blockchain to be synchronized is a blockchain chain1, the determined target data reading component is a data reading component SP2, so that the data reading component SP2 can be called to read data in the blockchain chain1, the read data can be written into a database D1, a database D2 and a database D3, and if the target blockchain to be synchronized is a blockchain chain2, the determined target data reading component is a data reading component SP1, so that the data reading component SP1 can be called to read data in the blockchain chain2, and the read data can be written into the database D1 and the database D2.

The data processing method provided by the embodiment of the application can be applied to a block chain data synchronization system, wherein N data reading components are arranged in the block chain data synchronization system, and N is a positive integer; each data reading component is configured to read data from a blockchain of one chain type, and it can be seen that the blockchain data synchronization system has a data reading function. In the method, the target block chain to be synchronized can be determined, and the target data reading component matched with the target block chain is determined from N data reading components according to the chain type of the target block chain, so that a special data reading component is determined for the block chain of the chain type, and the target data reading component can be called to read the data to be synchronized from the target block chain and write the read data into a database. The data to be synchronized in the target blockchain is now synchronized into the database. The block chain and the database are respectively accessed through the block chain data synchronization system, the block chain does not need to be directly in butt joint with the database, but the block chain data synchronization system is used as a middleware between the block chain and the database to realize data synchronization, and the data synchronization system can be greatly simplified, so that the maintenance cost of the system and the synchronization cost of the data are effectively reduced. By means of the data reading components arranged in the block chain data synchronization system, for the block chains to be synchronized with the same chain type, the same data reading component can be determined from the block chain data synchronization system to read data, and for the block chains to be synchronized with different chain types, different data reading components can be determined and independently read data. Thus, the block chain data synchronization system can cope with block chains of various chain types, data synchronization of the block chains of various chain types is not interfered due to the quantity change of the block chains, and the data synchronization efficiency and the stability of the block chain data synchronization can be improved.

Fig. 4 is a flowchart of a data processing method according to another exemplary embodiment of the present application. The data processing method can be applied to a block chain data synchronization system, wherein N data reading components are arranged in the block chain data synchronization system, and N is a positive integer; each data reading component is for reading data from a blockchain of one chain type. The block chain data synchronization system is also provided with M data writing-in components, wherein M is a positive integer; each data writing component is for writing data read from a blockchain into a database of one library type, and a different data writing component is for writing data read from a corresponding blockchain into a database of a different library type. The data writing component in this application, which may also be referred to as a data writing module or a data writing plug-in (Writer-plug in), abstracts database writing and is incorporated into a blockchain data synchronization system. Thus, the blockchain data synchronization system not only has a data reading function, but also has a data writing function. For data that is called up to be read from different blockchains by the data reading component, if it is written to the same database, the data writing component employed is the same, illustratively, data read from blockchain chain1 and data read from blockchain chain2 are both written to the HBase database by the data writing component HBase-Writer and to the HDFS database by the data writing component HDFS-Writer. The data processing method may be performed by a computer device, such as the computer device 101 in fig. 1a, and may include the following S401-S406.

S401, determining a target blockchain to be synchronized.

S402, determining a target data reading component which is matched with the target blockchain from N data reading components according to the chain type of the target blockchain.

S403, calling a target data reading component to read the data to be synchronized from the target block chain.

In one possible implementation, when the computer device invokes the target data reading component to read the data to be synchronized from the target blockchain, the computer device specifically reads the data to be synchronized in the target blockchain according to the following steps 1.1 to 1.3.

Step 1.1, acquiring historical synchronous data of a target block chain, wherein the historical synchronous data corresponds to a historical maximum block height. In a specific implementation, if the computer device is to first call the target data reading component to read the data to be synchronized in the target blockchain, the obtained historical synchronization data is empty, and the maximum historical height may be recorded as a value smaller than 0, such as-1, so that the maximum historical height. If the computer device has read data from the target blockchain at a time other than the first time the target data reading component was called, i.e., the target data reading component has been called at a historical time (i.e., a time prior to the time the target data reading component was called last time), then the historical synchronization data includes data that was historically read from the target blockchain and written to the database, and these data are data in blocks on the target blockchain. Because each block on the block chain has a block height, and the history synchronization data includes data in at least one block, a maximum block height can be determined from the block heights of at least one block corresponding to the history synchronization data, and the maximum block height is used as a history maximum block height.

Step 1.2, monitoring the current maximum block height corresponding to the target block chain, and comparing the current maximum block height with the historical maximum block height.

The computer device may monitor the target blockchain in real time or periodically to obtain a current maximum block height corresponding to the target blockchain, where the current maximum block height is a maximum block height of the block heights of the blocks currently included in the target blockchain. For example, the block height of the created block in the target blockchain is 0, the target blockchain currently includes 11 blocks, and then the current maximum block height is 10 can be monitored. Further, the computer device may compare the sizes between the current maximum block height and the historical maximum block height to determine whether to invoke the target data reading component to read the data to be synchronized from the target blockchain based on a size relationship of the two. As one possible implementation, the target data reading component may also be responsible for monitoring the function of the target blockchain, i.e., the computer device may invoke the target data reading component to monitor the current maximum blockheight of the target blockchain, and as another possible implementation, the computer device may employ a proprietary blockchain monitoring component (e.g., a blockchain monitoring plug-in or a blockchain monitoring program) to monitor the current maximum blockheight of the target blockchain, which is a component independent of the target data reading component, may be provided in or independent of the blockchain data synchronization system.

Step 1.3, if the current maximum block height is greater than the historical maximum block height, determining the data in the blocks with the block heights in the target block chain greater than the historical maximum block height as the data to be synchronized; and calling a target data reading component to read the data to be synchronized.

If the current maximum block height is greater than the historical maximum block height, it is indicated that a block is newly added in the target blockchain (may be referred to as a newly added block), and the block height of the newly added block is greater than the historical maximum block height, so that data in the newly added block (i.e., a block with a block height greater than the historical maximum block height) can be determined as data to be synchronized, and the target data reading component is invoked to read the data in the newly added block in the target blockchain. Otherwise, if the current maximum block height is less than or equal to the historical maximum block height, which indicates that the target blockchain is not currently added with blocks, and the data in the target blockchain is read at the historical moment, the computer device can continuously monitor the current maximum block height corresponding to the target blockchain, and when the monitored current maximum block height is greater than the historical maximum block height, the target data reading component can be called to read the data in the blocks with the block heights greater than the historical maximum block height in the target blockchain.

The processing logic for calling the data reading component to read the data to be synchronized in the target blockchain is shown in the steps 1.1-1.3, and whether the target data reading component is called to read the data to be synchronized in the target blockchain is automatically determined by monitoring the block height of the target blockchain and comparing the historical block height corresponding to the historical synchronization data. After the data to be synchronized in the target blockchain is read, the target blockchain is updated, so that the reading of the data on the chain can be automatically triggered by the maximum blockheight corresponding to the monitored target blockchain, the latest data on the target blockchain is synchronized timely, and the integrity of the data synchronized from the target blockchain is ensured.

In another implementation, if the computer device invokes the target data reading component for the first time to read the data to be synchronized from the target blockchain, all the data in the target blockchain can be directly read, thereby improving the synchronization efficiency. If the computer equipment does not call the target data reading component for the first time to read the data to be synchronized from the target block chain, the data can be read according to the steps 1.1-1.3, so that the synchronized data is prevented from being repeatedly read, resources required by the synchronization process are saved, and the synchronization cost is reduced.

S404, determining a target database to which the read data needs to be written.

In a specific implementation, the number of target databases to which the read data is to be written, which is determined by the computer device, includes one or more, and in order to better understand the flow of writing the read data into the databases, taking one target database as an example to describe the flow of writing the read data into the databases, for each target database, the data can be written according to the content shown in S405-S406.

In one embodiment, the blockchain data synchronization system may access M databases, each of which corresponds to a data writing component. The computer device may determine a target database to which read data needs to be written from the M databases according to a chain type, usage scenario, or business requirements of the target blockchain. Specifically, determining the target database may include several ways, as shown in (1) - (3) below.

(1) And acquiring synchronous configuration information aiming at the target block chain, and determining a target database to which the read data needs to be written according to a library identifier preset in the synchronous configuration information.

Each database has a library identity, such as a string, or a combination of database name and value, and the library identity is unique. The blockchain data synchronization system can provide a configuration function, and synchronous configuration information comprising a preset database identifier can be generated through the preset database identifier, so that the computer equipment can acquire the synchronous configuration information aiming at a target blockchain, search a database identified by the database identifier from various databases according to the preset database identifier in the synchronous configuration information, and determine the searched database as a target database to which read data needs to be written. In this way, the object can specify the database to which the data read from the target blockchain needs to be written, and flexibly specify one or more databases for the blockchains needing to be synchronized, so as to meet the personalized requirements of the object.

(2) A database that is adapted to the chain type of the target blockchain is determined as the target database to which the read data needs to be written.

Because the chain type of the target blockchain can be determined based on the data composition in the target blockchain or based on the openness degree of the target blockchain, the data composition and the openness degree are related to the data in the target blockchain, and different databases have different storage characteristics, when the storage characteristics of the databases can be better compatible with the data of the target blockchain, the databases can be considered to be matched with the chain type of the target blockchain, so that the databases can be determined as the target databases to which the read data need to be written. And the adaptive database is used as a target database, so that the data can be written into the database normally. As an alternative implementation manner, databases adapted to different chain types may be different, and the library types of databases adapted to the same chain type may be the same, so that data in blockchains of different chain types may be written into different databases respectively, so that the corresponding databases may be accessed based on the chain type in a query scenario.

(3) And determining the database which is matched with the data type of the read data as a target database to which the read data needs to be written.

The data type of the read data may be determined according to the data function or the data source. Illustratively, the data type of the read data may be any one or more of the following: invoice data, payment data or logistical data, etc. The record formats of the data with different data types are different, the database matched with the data type is determined as the target database to which the read data needs to be written, the data with the same data type can be matched with the database with the same database type, and the data with different data types can be matched with the database with different database types, so that when the data read from the target blockchain relates to multiple data types, the data read from the same blockchain can be classified and stored into the different databases, and then classified and queried based on the data types.

S405, determining a target data writing component which is matched with the target database from M data writing components according to the library type of the target database.

In one implementation, the library types may be determined based on the functionality of the database, illustratively, the library types include HDFS for storing files, HBase clusters for large data storage, and the like. The different databases have different functions, and the library types of the different databases can be different, and one library type corresponds to one database. In another implementation, the library types may be determined based on storage characteristics of the databases, e.g., the library type of any one database may be a relational database or a non-relational database or an object-oriented database, one library type may correspond to one or more databases, and the library types of different databases may be the same, e.g., both databases are relational databases.

Based on the purpose of the data writing units, one data writing unit corresponds to one library type, and the computer equipment can determine the data writing unit corresponding to the library type from M data writing units according to the library type of the target database and determine the data writing unit as the target data writing unit matched with the target database. The adaptation is that the library type of the database which is supported by the data reading part to be written is the same as the library type of the target database, and the target data writing part is one data writing part in M data writing parts and can be used for writing the data read from the target blockchain into the target database.

S406, calling a target data writing component to write the read data into a target database.

In one embodiment, if the read data is excessive, for example, the data amount is greater than a preset data amount, the computer device may invoke the target data writing component to write the read data to the target database in batches. If it is determined that the target database to which the read data is to be written includes a plurality of target data writing units, the computer device may call each target data writing unit to write the read data into each target database, so that the same data may be written into different destination terminals to perform data backup in the different destination terminals. Or the computer equipment can call each target data writing component, select data which needs to be written into each database respectively from the read data, and write the selected data into the corresponding target database. Therefore, the same data can be written into different destination terminals in a scattered way, so that the writing speed can be increased, and the data storage efficiency can be improved.

To facilitate a more intuitive understanding of the above data processing flow, an exemplary illustration can be made in connection with fig. 5, as shown in fig. 5, in which the blockchain data synchronization system further includes a data processing component that can act as a transmission channel between the data reading component and the data writing component. The data read from the target blockchain by the target data reading component may be sent to the data processing component, and the target data writing component may continually pull data from the data processing component, thereby writing the read data into the target database.

According to the data processing method provided by the embodiment of the application, through the data reading part and the data writing part which are arranged in the block chain data synchronization system, the computer equipment can determine the target data reading part which is matched with the target block chain and the target data writing part which needs to write the read data from the block chain data synchronization system; and the adaptive target data reading part is called to read the data on the target block chain, and the target data writing part is called to write the read data into the database, so that the efficient reading and writing of the data on the target block chain can be realized, and the data on the target block chain can be synchronized into the target database. The block chain data synchronization system integrates a data reading function and a data writing function, can read data on a chain through the data reading component provided by the block chain data synchronization system for block chains of different chain types, can write data through the data writing component provided by the block chain data synchronization system for databases of different library types to be written, realizes orderly synchronization of the block chain data through cooperation between the data reading component and the data writing component, and improves stability of data synchronization.

Fig. 6 is a flowchart of a data processing method according to another exemplary embodiment of the present application. The data processing method can be applied to a block chain data synchronization system, wherein N data reading components are arranged in the block chain data synchronization system, and N is a positive integer; each data reading component is for reading data from a blockchain of one chain type. The blockchain data synchronization system is also provided with a data processing component (conversion-frame) that can be used to perform some critical processing on the read data to control the writing of the data or to control the reading of the data. If the blockchain data synchronization system is further provided with a data writing component, the data processing component can serve as a transmission channel between the data reading component and the data writing component. The data processing method may be performed by a computer device, such as computer device 101 in fig. 1a, and may include the following S601-S605.

S601, determining a target blockchain to be synchronized.

S602, determining a target data reading component which is matched with the target blockchain from N data reading components according to the chain type of the target blockchain.

S603, calling a target data reading component to read data to be synchronized from the target block chain.

S604, calling a data processing component to perform transmission processing on the read data.

In a specific implementation, the data processing component integrates some general functions, and processes the read data by using the general functions, so that on one hand, the efficiency of block chain data synchronization is improved, and on the other hand, the synchronization requirement of objects can be met to a certain extent. The computer device calls the data processing component to transmit the read data, and the transmission processing can comprise the following (one) - (three) specific implementations.

And (one) buffering.

In some cases, the computer device may call the data read by the target data reading component and be limited by the data processing resources, so that the data cannot be immediately processed, thereby causing congestion of data synchronization and bringing a certain data processing pressure. Therefore, the read data can be buffered firstly, and the data can be buffered specifically to reduce the data processing pressure so as to improve the fluency of data synchronization on the block chain, thereby improving the data synchronization efficiency. Specifically, the method comprises the following steps (1-1) -step (1-2).

And (2) calling a data processing component to judge whether the read data meets the caching condition.

As a possible implementation manner, when the computer device performs the step (1-1), the following may be specifically performed: invoking a data processing part to detect the data quantity of the data read by the target data reading part in unit time; if the detected data amount of the data read by the target data reading component in the unit time is larger than the preset data amount, determining that the read data meets the caching condition.

The unit time mentioned in the embodiments of the present application may be 1 millisecond (ms), 1 second(s), 1 minute (min), 1 hour (h), etc., and the present application is not limited thereto. The data amount of the data read by the target data reading unit in the unit time may be regarded as a data reading rate of the target data reading unit, and the preset data amount is a data amount corresponding to a preset data reading rate, and the preset data reading rate may be an upper limit value of a data reading rate determined based on the configuration of the object, or may be an upper limit value set by the computer device based on a maximum data writing rate of the target database, at which the target data reading unit may read the data of the preset data amount in the unit time. Therefore, the comparison between the data amount of the data to be read in the unit time and the preset data amount can represent the comparison between the actual data reading rate and the preset data reading data. Further, when the computer device detects that the data amount of the data read by the target data reading component in unit time is larger than the preset data amount, the actual data reading rate exceeds the preset data reading rate, the computer device calls that the data read by the target data reading component may not be processed in time, and then it can be determined that the read data meets the buffering condition, and the read data is buffered first.

If the detected data amount of the data read by the target data reading component in unit time is smaller than or equal to the preset data amount, the actual data reading rate is smaller than the preset data reading rate, and the computer equipment calls the data read by the target data reading component to be processed in time, so that the read data can be determined to not meet the caching condition, and the read data does not need to be cached.

In this way, the computer device may invoke the data processing section to detect the data amount of the data read by the target data reading section in a unit time, and further compare the detected data amount with a preset data amount, so as to determine whether the read data satisfies the buffering condition. In the mode, the data quantity of the data read in the unit time is detected to be the actual data quantity, and the judgment result can be more effective and reliable based on the actual data quantity; on the other hand, as only one data reading rate is needed to be detected, the result can be judged by comparing the actual data quantity with the preset data quantity, so that the time can be saved, and the processing efficiency can be improved.

As another possible implementation manner, the read data is written into the database through the target data writing component; the target data writing means is for writing the read data into the target database, and thus the writing database is specifically the writing target database. The computer device, when executing the step (1-1), may specifically execute the following: invoking a data processing part, and detecting the data reading rate of a target data reading part and the data writing rate of a target data writing part; if the data reading rate is greater than the data writing rate, determining that the read data meets the caching condition.

The data read rate of the target data read section refers to the data amount of the target data read section to read data in a unit time. Such as 1kb/s, means that the target data reading component reads 1 kilobyte (kb) of data from the target blockchain per second. The data writing rate of the target data writing section refers to the data amount of the data written into the database by the target data reading section per unit time. Such as 1Mb/s, means that the target data writing means can write 1Mb (1 mbit) of data read into the database every second. The amount of data written at the data read rate may match the throughput of the target database, and in particular may be less than or equal to the throughput. A balance point is needed between the data reading rate and the data writing rate, so that the read data can be processed in time. If the data reading rate is greater than the data writing rate, which means that the data amount of the data read by the calling target data reading part in the unit time is greater than the data amount of the data written by the calling target data writing part in the unit time, the read data may not be written into the database in time, so that the read data can be determined to meet the buffering condition, and the read data can be buffered and queued for processing.

If the detected data reading rate is smaller than or equal to the data writing rate, the data quantity of the data read by the calling target data reading component in unit time is smaller than or equal to the data quantity of the data written by the calling target data writing component in unit time to the database, and the read data can be written into the database in time, so that the fact that the read data does not meet the caching condition can be determined, and the read data does not need to be cached.

In this way, the computer device may invoke the data processing unit to detect the data reading rate of the target data reading unit and the data writing rate of the target data writing unit, and compare the sizes between the detected two rates, so as to determine whether the read data meets the buffering condition. By comparing the processing speeds between the source end of data reading and the destination end of data writing, whether the read data needs to be cached or not can be accurately and effectively judged.

In one possible implementation manner, if the determined number of the target databases includes a plurality of target databases, the data read from the target blockchain is written into the corresponding target databases through the plurality of data writing units, and in the process of determining whether the read data needs to be cached, the target data writing rate can be specifically determined according to the respective data writing rate of each data writing unit, and the data reading rate of the target data reading unit is compared with the target data writing rate to determine whether the caching condition is met. In one implementation, the maximum data writing rate may be determined from the data writing rates, and the maximum data writing rate is used as the target data writing rate, or an average data writing rate obtained by averaging the data writing rates is used as the target data writing rate. If the read data are uniformly written into a plurality of target databases, the target data writing rate can be the maximum data writing rate in the data writing rates, so that when the maximum data writing rate is smaller than the data reading rate, the read data can be determined to meet the caching condition; if the data of the data is written in the plurality of databases in a scattered manner, the target data writing rate may take an average value of the respective data writing rates, so that when the average data writing rate is smaller than the data reading rate, it may be determined that the read data satisfies the buffering condition.

Step (1-2), if the caching condition is met, calling a data processing component to cache the read data into a cache space; the data after transmission processing refers to cache data in a cache space, and the cache data in the cache space supports batch writing into a database.

Specifically, in the case where the read data satisfies the buffering condition, the computer device may call the data processing section to buffer the data read from the target blockchain to a buffer space, which may be a local space of the computer device or a cloud database connected to the computer device. The cache data in the cache space includes data read from the target blockchain. Since the buffer space is used for buffering data read from the blockchain and satisfying the buffering condition. For blockchains of other chain types than the chain type of the target blockchain, if the data read from the blockchain satisfies the cache condition, the data may also be cached in the cache space.

The cache data in the cache space can be used as data after transmission processing, the computer equipment can write the cache data into the database in batches, and the data volume of the data of each batch can be equal or unequal. According to the first-in first-out principle, according to the caching time of the cached data, the data batch where the data with the earliest caching time is located can be obtained from the caching space, so that the cached data of the batch can be written into the database. Thus, the data buffered in the buffer space can be written into the database first, thereby ensuring the orderly processing of the buffered data.

If the cache condition is not met, the data read by the computer equipment calling the target database can be processed in time, and the computer equipment does not need to execute the step of calling the data processing component to cache the read data into the cache space. As one implementation, the computer device may perform other transmission processes on the read data, such as a filtering process or a data conversion process.

In one implementation manner, if the computer device reads data from the blockchains of multiple chain types and concurrently executes writing of the data read from different blockchains into the same database, the computer device may count the total data amount of the read data and acquire the throughput of the database, and when the total data amount is greater than the throughput of the database, buffer the read data into the buffer space and wait for writing processing, so as to avoid failure caused by congestion of data writing, and improve the smoothness of data synchronization.

In the process of synchronizing the data in the blockchain, the data processing component not only can judge whether the data can be cached, but also can cache the data meeting the caching condition into the caching space to finish the buffering of the read data, thereby improving the smoothness of data synchronization and avoiding the congestion of data writing. For the data which does not meet the buffer condition, the calling data processing component can directly send the data to the data writing component or execute other transmission processing, and the whole flow of data synchronization can be advanced.

And (II) filtering.

Since the read data may contain useless data, it may not be used in some query scenarios, and thus the data processing component may be invoked to filter the read data before it is written to the database to write useful data to the database. Specifically comprises the following steps (2-1) and (2-2).

And (2-1) obtaining data filtering conditions.

In a specific implementation, since the blockchain-based data synchronization system can provide a function of synchronization configuration, it is used for pre-configuring data filtering conditions by the function of synchronization configuration, and thus can synchronize specified data. The computer device may obtain synchronization configuration information and obtain data filtering conditions from the synchronization configuration information. If the data filtering condition is not configured, then all data on the synchronous blockchain may be configured by a system default, such as a system default configuration. The data filtering condition is a condition for screening out required data, and may specifically be a condition related to an attribute of the blockchain data, for example, the data filtering condition is: a contract is specified. In this way, the computer device can synchronize data of a specified contract. For another example, the data filtering conditions are: the block height range is preset. In this way, the computer device can synchronize block data having a block height within a specified block height range.

Step (2-2), calling a data processing component, and screening the read data according to the data filtering condition to obtain target data meeting the data filtering condition; wherein the data after the transmission processing refers to target data.

In a specific implementation, the computer device may compare the data filtering condition with the attribute of the read data, and use the data whose attribute meets the condition of the data as the target data that meets the data filtering condition. The smart contract a is determined based on the data filtering condition, so that the data of the smart contract a can be selected as target data based on the smart contract to which the read data belongs, and the target data is the data after the transmission processing.

The filtering processing shown in the steps (2-1) - (2-2) is performed on the read data through the data filtering conditions, so that on one hand, the designated data can be screened out to better meet the personalized requirements of the object, and on the other hand, the read data is filtered before the data is written into the database, so that part of the data which does not meet the conditions can be removed, the data volume written into the database is reduced, and the data writing pressure is reduced to a certain extent.

And (III) data conversion.

In order to better write data into the database, the computer device may call the data processing component to convert the read data into a data format adapted to the database to be written, thereby improving the write rate. Specifically, the method comprises the following two modes (3.1) and (3.2).

And (3.1) calling a data processing component, and performing format conversion processing on the read data according to the data format of the database to obtain converted data.

The data format of the database refers to the data format of the database to which the read data is written, such as the target database mentioned above. The data format is a rule describing that data is stored in a database, such as JSON (JavaScript Object Notation, a lightweight data exchange format) format or Key-Value (i.e., key-Value) format. The data formats of different databases are different, for example, the data formats supported by MySQL database include JSON format, and the data formats supported by HBase database are: key-value format. In a specific implementation, if the read data needs to be written into a database, the computer device may call the data processing unit to convert the read data into data matching the data format of the database, so as to obtain converted data, that is, the data format of the converted data conforms to the data format required by the database. If the read data is required to be written into a plurality of databases, the computer equipment can call the data processing component to respectively convert the read data into data which are respectively matched with the data formats of the databases, so as to obtain a plurality of converted data, wherein the data format of each converted data accords with the data format required by one database. In this way, the same data can be converted into data with different formats, so that the data is matched with the data formats of different databases, and the converted data can be written into the databases by directly calling the data writing component by the subsequent computer equipment, thereby improving the efficiency of writing the data into the databases.

And (3.2) calling a data processing component, selecting the data with the target quantity from the read data, and performing format conversion processing on the data with the target quantity to obtain converted data.

Specifically, the computer device may also call the data processing unit to screen out part of the data, so as to convert part of the data into data in a certain data format, and write the data into an adapted database. Wherein the target number may be a pre-configured amount of data or determined based on the capacity of the database. The computer device may invoke the data processing component to perform a format conversion process on the target amount of data in accordance with the data format of the database. When the target number is the preset data amount, the format conversion shown in the above (3.2) can be executed every time the read data is written into one database, so that the read data can be divided into equal data, and the equal data can be written into the corresponding database after the format conversion processing. When determining the target number from the capacity of the database, the target number may be less than or equal to the capacity of the database, and the resulting data format of the converted data matches the data format required for determining the database to which the target number relates. In this way, the computer device can convert the read data into data with different data formats, so that the converted data can be conveniently and quickly written into the database, namely, the data writing component does not need to be responsible for data format conversion, and the data processing component uniformly performs conversion processing, so that the consistency of data conversion is ensured.

In the modes (3.1) and (3.2), the transmission processed data refers to the converted data, and the computer device can directly write the converted data into the database. As a possible implementation manner, the cached data in the cache space may be written into the database after format conversion processing, or the target data may be written into the database after format conversion.

S605, writing the data after the transmission processing into a database.

The read data is transmitted to obtain data after transmission processing, and the data after transmission processing comprises any one or more of the following: the method comprises the steps of caching data in a cache space, target data meeting data filtering conditions and converted data. Based on different transmission processes, the data after the transmission process may be the complete read data (such as the cache data in the cache space), or may be the part of the read data (such as the target data) which is left after the screening, or may be new data (such as the converted data) with a different data format from the read data. Depending on the specific processing content of the transmission processing performed on the read data, and which transmission processing the invoking data processing component specifically performs may be specified by the object or decided by the device on its own based on usage scenario/traffic requirements. For example, both the data filtering process and the format conversion process are performed on the read data, so that the resulting transmission processed data is partial data having a different data format.

In one possible implementation, the blockchain data synchronization system includes M data writing units, and when executing S604, the computer device may specifically determine the target database first, determine the target data writing unit from the M data writing units, and then call the target data writing unit to write the data after the transmission processing into the target database.

In one implementation, the computer device may call the data processing component to perform the flow control processing on the read data; wherein, the flow control treatment includes: one or both of the data read rate and the data write rate are controlled. The data read rate refers to the data amount of read data per unit time, and the data write rate refers to the data amount of data written to the database per unit time. In one specific implementation, the computer device may compare the data read rate and the data write rate, and when the data read rate is greater than the data write rate, adjust the rate by any one of the following ways to realize control of the data flow: (1) step by step, the data reading rate is reduced until the adjusted data reading rate is smaller than or equal to the data writing rate; (2) the data writing rate is increased until the adjusted data writing rate is greater than or equal to the data writing rate; (3) and simultaneously, the data writing rate is increased and the data reading rate is decreased until the data writing rate and the data reading rate are equal or the difference value of the data writing rate and the data reading rate is within a preset range. The read data is subjected to flow control treatment, so that the size of the data volume of the data ingested by the block chain data synchronization system or the size of the data volume of the data output by the block chain data synchronization system can be controlled, the balance of the data volume between data reading and data writing is realized, unnecessary congestion caused by too fast data reading or too slow data writing is avoided, the fluency of data synchronization is ensured, and the efficiency of data synchronization is improved to a certain extent.

In yet another implementation, the computer device may also invoke the data processing component to process the read data concurrently, such as concurrently reading data or concurrently writing data to a database, or concurrently filtering the data read by the respective data reading components. In summary, the data processing unit may be used for performing transmission processing on the read data, performing flow control processing on the read data, and performing concurrency processing on the read data. Therefore, the data processing component integrates rich functions of buffering, flow control, concurrency, filtering, data conversion and the like, and is beneficial to carrying out general processing on read data. The sequence of the various processes is not limited in this application, for example: and performing flow control treatment on the read data and then performing filtering treatment.

According to the data processing method provided by the embodiment of the application, the read data can be written into the database after being transmitted, and some general functions of data synchronization, such as buffering, flow control, concurrency, filtering, data conversion and the like, can be multiplexed through the data processing component in the block chain data synchronization system, so that the read data can be processed by adopting a unified processing standard, and the consistency of data processing can be ensured.

In connection with the description of the embodiments of FIGS. 2, 4 and 6 above, the following may provide the overall flow of interactions involved in blockchain data synchronization through a blockchain data synchronization system as shown in FIG. 7 a. The method specifically comprises the following steps:

1. the object configures the synchronization information through a synchronization client for providing a visual configuration interface, designates a blockchain to be synchronized and designates a database for writing data in the blockchain through configuration in the configuration interface, and starts the blockchain data synchronization system.

2. Data in the specified blockchain is read from the blockchain network through a blockchain data synchronization system.

3. The specified blockchain is looked up from the blockchain network according to the configured chain identification.

4. The data to be synchronized is read from the specified blockchain by a data reading component in the blockchain data synchronization system. The data processing component in the blockchain data synchronization system is then invoked to process the read data, such as filtering, buffering, flow control, data conversion, and the like.

5. And calling a data writing component in the block chain data synchronization system to write the data into the database. Optionally, after the data writing is completed, the blockchain data synchronization system may also feed back a message of completion of the blockchain data synchronization to the synchronization client.

6. The blockchain data synchronization system can monitor whether the state of the blockchain read before is updated or not in a timing or real-time manner; specifically, it can be monitored whether the current maximum block height and the historical maximum block height of the blockchain are the same.

7. If the state of the blockchain is monitored to be updated (namely, the current maximum blockheight is greater than the historical maximum blockheight), the blockchain data synchronization system requests to read the latest data on the blockchain through the step 8, searches the blockchain with updated state through the step 9, reads the data on the blockchain through the step 10, performs filtering, buffering and other processes on the read data, writes the data into a database through the step 11, and completes synchronization of the latest data on the blockchain. Optionally, if the synchronized blockchain is not specified, the system can also monitor whether all blockchain states are updated according to the default configuration of the system, and synchronize the data in the blockchain with the updated states into the database.

An exemplary application scenario for synchronizing multiple types of blockchains into multiple databases using a blockchain data synchronization system may be shown in fig. 7b, as described above for each blockchain accessed by the system, by synchronizing the data on the blockchain into the databases according to the interaction procedure described above. The system comprises a block chain data synchronization system, a centralized storage system and a storage system, wherein the block chain comprises a block chain1, a block chain2, a block chain3 and the like, the centralized storage system comprises a MySQL database, an HBase database, an HDFS and the like, and each block chain completes data synchronization through the block chain data synchronization system. It will be appreciated that the above-described blockchain and database are merely examples, and that if the blockchain data synchronization system is connected to other blockchains or other databases, the data synchronization may be accomplished through the above-described interaction flow.

In one embodiment, the description of the blockchain data synchronization system in connection with the embodiments of fig. 2, 4, and 6 may provide a schematic structural diagram of the blockchain data synchronization system shown in fig. 8, where N (where N is 3) data reading units, M (where M is 3) data writing units, and one data processing unit are provided. The data reading component comprises an A chain-SPV, a B chain-SPV and a C chain-SPV, data can be read from blockchains of different chain types respectively, and the data writing component comprises an HBase-writing component, a MySQL writing component and an HDFS writing component which are used for writing the data read from the blockchains into an HBase database, a MySQL database and an HDFS database respectively. The interactive flow among various components of the block chain data synchronization system comprises the following contents: firstly, a data reading component (such as an A chain-SPV) is responsible for reading data to be synchronized in an accessed blockchain (such as an A blockchain) and sending the read data to a data processing component, the data processing component can perform transmission processing on the read data, the transmission processing comprises one or more of buffering, filtering and data conversion, and the data processing component can perform flow control, concurrency and other processing, so that the data synchronization efficiency is improved. The data processing unit may send the data after the transmission processing to the data writing unit (such as MySQL writing unit and HDFS writing unit), and the data writing unit may write the data after the transmission processing into the database (such as MySQL database and HDFS database).

Based on the above-mentioned blockchain data synchronization system shown in fig. 8, the blockchain data of the same chain type can be read by the same data reading component, and the blockchain data of different chain types can be independently processed by different data reading components, so that the data reading efficiency can be improved; databases of different library types may be matched to different data writing components to thereby independently process data to be written to the corresponding databases. The data processing unit can be understood as a framework for connecting the data reading unit and the data writing unit, and the data can be uniformly processed by adopting a uniform framework, so that the consistency of data processing is ensured, and the maintenance cost can be reduced.

In one embodiment, the data reading components in the blockchain data synchronization system may also be flexibly pluggable as the number of chain types of the accessed blockchains varies. Comprising the following steps: (1) when the chain type of the block chain to be synchronized accessed by the block chain data synchronization system is newly increased, deploying a data reading component matched with the newly increased chain type in the block chain data synchronization system; (2) when the chain type of the blockchain to be synchronized that has been accessed by the blockchain data synchronization system is reduced, a data reading component adapted to the reduced chain type is disabled in the blockchain data synchronization system.

Specifically, when the blockchain data synchronization system accesses a blockchain of a newly added chain type, in order to synchronize data in the newly added blockchain, the computer device may deploy a data reading component adapted to the newly added chain type in the blockchain data synchronization system, thereby facilitating reading of data in the blockchain of the newly added chain type. When the chain type of the blockchain to be synchronized that has been accessed in the blockchain data synchronization system is reduced, the computer device may disable the data reading component in the blockchain data synchronization system that is adapted to the reduced chain type in order to avoid unnecessary scheduling of resources. The disabling may be either a delete or a shut down, if the previously reduced chain type blockchain is subsequently added, or the original data reading element may be re-enabled without re-deployment if the data reading element is shut down. Under the condition that the original data reading component is deleted, the data reading component can be redeployed, and the flexibility is high. Thus, the number of data reading components provided in the blockchain data synchronization system may change with the number of the chain types of the blockchains to be synchronized that are accessed, for example, based on the blockchain data synchronization system shown in fig. 8, the blockchain data synchronization system shown in fig. 9a is provided, the chain types of the blockchains to be synchronized that are accessed in the blockchain data synchronization system are newly increased to 4 types (i.e. newly increased D blockchains), then the number of data reading components provided may also be newly increased to 4 types (i.e. newly increased data reading components: D chains-SPV), the number of the blockchains to be synchronized that are accessed in the blockchain data synchronization system is reduced by C blockchains, as shown in (2) in fig. 9a, then the original data reading components C chains-SPV are disabled (identified by the disable identifier 910). Each data reading element is adapted to the blockchain so that one data reading element can read data to be synchronized from the blockchain of one chain type. Based on the method, no matter how many kinds of blockchains of the chain types are newly added, corresponding data reading components can be deployed to read the newly added blockchains, and the expansibility of the blockchain data synchronization system is high.

In one embodiment, the data writing component in the blockchain data synchronization system may also be flexibly pluggable as a function of the number of library types of the accessed database. Comprising the following steps: (3) when the database type of the database accessed by the block chain data synchronization system is newly increased, deploying a data writing component which is matched with the newly increased database type in the block chain data synchronization system; (4) when the library type of a database to which the blockchain data synchronization system has access decreases, a data writing component adapted to the reduced library type is disabled in the blockchain data synchronization system.

Specifically, when the blockchain data synchronization system accesses a database of a newly added library type, then to support data writing to the newly added database, the computer device may deploy a data writing component in the blockchain data synchronization system that is adapted to the newly added library type. When the database type of the accessed database in the blockchain data synchronization system is reduced, the computer equipment can disable the data writing component which is matched with the reduced database type in the blockchain data synchronization system, wherein the disabling can be realized by deleting or closing, and further, if the database of the reduced database type is newly added later, the database can be re-started under the condition that the original data writing component is closed, and the re-deployment is not needed. Under the condition that the original data writing component is deleted, the device can be redeployed, and has high flexibility. Based on this, the number of data writing parts provided in the blockchain data synchronization system may vary with the number of library types of the accessed databases, for example, based on the blockchain data synchronization system shown in fig. 8, the blockchain data synchronization system shown in fig. 9b is provided, as shown in (1) in fig. 9b, the library types of the accessed databases in the blockchain data synchronization system are newly increased to 4 (i.e., newly added Oracle databases), then the number of data writing parts provided may also be newly increased to 4 (i.e., newly added data writing parts: oracle-writing parts), as shown in (2) in fig. 9b, the library types of the accessed databases in the blockchain data synchronization system are reduced to 2 (i.e., HDFS databases are reduced), then the data writing parts provided (i.e., HDFS-writing parts) may be disabled (identified by disabling the identifier 920), for example: HDFS-write unit, the number of data write units is reduced to 2. The respective data writing means are adapted to the database so that one data writing means can write the read data into a database of one library type, as shown in fig. 9 b. Therefore, no matter how many databases of library types are newly added, the data can be written into the newly added databases by deploying corresponding data writing components, and the system has extremely high expansibility and can cope with various complex data synchronization processes by flexibly adding and deleting the data writing components.

Therefore, the blockchain data synchronization system in the embodiment of the application not only can realize stable and efficient data synchronization from blockchains of different types to various databases, but also has strong expandability and low complexity, and can uniformly synchronize data to the databases through the blockchain data synchronization system for the blockchains of different types, when the types of the blockchains are increased, the diversified blockchains can be dealt with through an expanded data reading component (SPV-plug in), and when the types of libraries of the databases are increased, different types of centralized storage can be docked through an expanded data writing component (Writer-plug in). Through the tool, the data to be synchronized can be redistributed to the corresponding database after centralized processing, so that the cost is low, the maintenance is easy, and the expandability is high.

The data processing apparatus provided in the embodiments of the present application will be explained in the following.

Referring to fig. 10, fig. 10 is a schematic structural diagram of a data processing apparatus according to an exemplary embodiment of the present application. The data processing means may be a computer program (comprising program code) running in a computer device, for example the data processing means is an application software; the data processing device may be used to perform the corresponding steps in the methods provided by the embodiments of the present application. The method provided by the embodiment of the application is applied to a block chain data synchronization system, N data reading components are arranged in the block chain data synchronization system, and N is a positive integer; each data reading component is for reading data from a blockchain of one chain type. As shown in fig. 10, the data processing apparatus may include: a determination unit 1001 and a processing unit 1002.

A determining unit 1001 configured to determine a target blockchain to be synchronized;

a determining unit 1001, configured to determine, from the N data reading units, a target data reading unit adapted to the target blockchain according to a chain type of the target blockchain;

the processing unit 1002 is configured to invoke a target data reading unit, and read data to be synchronized from a target blockchain; the method comprises the steps of,

the processing unit 1002 is further configured to write the read data to a database.

In one embodiment, the blockchain data synchronization system is further provided with M data writing components, wherein M is a positive integer; each data writing component is used for writing data read from the blockchain into a database of one library type; the processing unit 1002 is specifically configured to, when writing the read data into the database:

determining a target database to which the read data needs to be written;

determining a target data writing component which is matched with the target database from M data writing components according to the library type of the target database;

and calling a target data writing component to write the read data into a target database.

In one embodiment, the determining unit 1001 is specifically configured to, when determining that the read data needs to be written to the target database:

Acquiring synchronous configuration information aiming at a target block chain, and determining a target database to which read data needs to be written according to a library identifier preset in the synchronous configuration information; or,

determining a database matched with the chain type of the target blockchain as a target database to which read data needs to be written; or,

and determining the database which is matched with the data type of the read data as a target database to which the read data needs to be written.

In one embodiment, the processing unit 1002, when invoking the target data reading component, reads the data to be synchronized from the target blockchain, is specifically configured to:

acquiring historical synchronous data of a target block chain, wherein the historical synchronous data corresponds to a historical maximum block height;

monitoring the current maximum block height corresponding to the target block chain, and comparing the current maximum block height with the historical maximum block height;

if the current maximum block height is greater than the historical maximum block height, determining the data in the blocks with the block heights in the target block chain greater than the historical maximum block height as the data to be synchronized; the method comprises the steps of,

and calling a target data reading component to read the data to be synchronized.

In another embodiment, a data processing component is further provided in the blockchain data synchronization system; the processing unit 1002 is specifically configured to, when writing the read data into the database:

Invoking a data processing component to perform transmission processing on the read data;

and writing the data after the transmission processing into a database.

In one embodiment, when the processing unit 1002 invokes the data processing unit to perform transmission processing on the read data, the processing unit is specifically configured to:

invoking a data processing component to judge whether the read data meets a cache condition;

if the buffer condition is met, calling a data processing component to buffer the read data into a buffer space; the data after transmission processing refers to cache data in a cache space, and the cache data in the cache space supports batch writing into a database.

In one embodiment, the processing unit 1002 is specifically configured to, when calling the data processing unit to determine whether the read data satisfies the cache condition:

invoking a data processing part to detect the data quantity of the data read by the target data reading part in unit time;

if the detected data amount of the data read by the target data reading component in the unit time is larger than the preset data amount, determining that the read data meets the caching condition.

In another embodiment, the read data is written into the database by the target data writing means; the processing unit 1002 is specifically configured to, when calling the data processing unit to determine whether the read data satisfies the buffering condition:

Invoking a data processing part, and detecting the data reading rate of a target data reading part and the data writing rate of a target data writing part;

if the data reading rate is greater than the data writing rate, determining that the read data meets the caching condition.

In another embodiment, the processing unit 1002 is specifically configured to, when invoking the data processing unit to perform transmission processing on the read data:

acquiring data filtering conditions;

invoking a data processing component, and screening the read data according to the data filtering condition to obtain target data meeting the data filtering condition; wherein the data after the transmission processing refers to target data.

calling a data processing component, and performing format conversion processing on read data according to the data format of the database to obtain converted data; or,

invoking a data processing component, selecting the data with the target quantity from the read data, and performing format conversion processing on the data with the target quantity to obtain converted data;

wherein, the data after transmission processing refers to the data after conversion.

In another embodiment, a data processing component is further provided in the blockchain data synchronization system; the processing unit 1002 is further configured to:

calling a data processing component to perform flow control processing on the read data; wherein, the flow control treatment includes: one or both of the data read rate and the data write rate are controlled.

In one embodiment, the processing unit 1002 is further configured to:

when the chain type of the block chain to be synchronized accessed by the block chain data synchronization system is newly increased, deploying a data reading component matched with the newly increased chain type in the block chain data synchronization system;

when the chain type of the blockchain to be synchronized that has been accessed by the blockchain data synchronization system is reduced, a data reading component adapted to the reduced chain type is disabled in the blockchain data synchronization system.

In one embodiment, the processing unit 1002 is further configured to:

when the database type of the database accessed by the block chain data synchronization system is newly increased, deploying a data writing component which is matched with the newly increased database type in the block chain data synchronization system;

when the library type of a database to which the blockchain data synchronization system has access decreases, a data writing component adapted to the reduced library type is disabled in the blockchain data synchronization system.

A related description of the computer device provided in the embodiments of the present application follows.

Referring to fig. 11, fig. 11 is a schematic structural diagram of a computer device (i.e. the first common node) according to an embodiment of the present application. The computer device may comprise a stand-alone device (e.g., one or more of a server, node, terminal, etc.), or may comprise components (e.g., a chip, software module, hardware module, etc.) internal to the stand-alone device. The computer device may include at least one processor 1101 and a communication interface 1102, and further optionally, may include at least one memory 1103 and a bus 1104. Wherein the processor 1101, the communication interface 1102 and the memory 1103 are connected by a bus 1104.

The processor 1101 is a module for performing arithmetic operation and/or logic operation, and may specifically be one or more of a central processing unit (central processing unit, CPU), a picture processor (graphics processing unit, GPU), a microprocessor (microprocessor unit, MPU), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA), a complex programmable logic device (Complex programmable logic device, CPLD), a coprocessor (assisting the central processing unit to perform corresponding processing and application), a micro control unit (Microcontroller Unit, MCU), and other processing modules.

The communication interface 1102 may be used to provide information input or output to at least one processor. And/or the communication interface 1102 may be used to receive externally transmitted data and/or transmit externally, may be a wired link interface including, for example, an ethernet cable, or may be a wireless link (Wi-Fi, bluetooth, universal wireless transmission, vehicle-mounted short-range communication technology, other short-range wireless communication technology, etc.) interface. The communication interface 1102 may act as a network interface.

The memory 1103 is used to provide storage space in which data such as an operating system and computer programs can be stored. The memory 1103 may be one or more of a random access memory (random access memory, RAM), a read-only memory (ROM), an erasable programmable read-only memory (erasable programmable read only memory, EPROM), or a portable read-only memory (compact disc read-only memory, CD-ROM), etc.

In one possible implementation manner, the data processing method provided by the application is applied to a block chain data synchronization system, wherein N data reading components are arranged in the block chain data synchronization system, and N is a positive integer; each data reading component is for reading data from a blockchain of one chain type. The processor 1101 in the computer device is configured to invoke a computer program stored in the at least one memory 1103 for performing the following operations: determining a target block chain to be synchronized; determining a target data reading component which is matched with the target blockchain from N data reading components according to the chain type of the target blockchain; invoking a target data reading component to read data to be synchronized from a target block chain; and writing the read data into a database.

In one embodiment, the blockchain data synchronization system is further provided with M data writing components, wherein M is a positive integer; each data writing component is used for writing data read from the blockchain into a database of one library type; the processor 1101 is specifically configured to, when writing the read data into the database:

determining a target database to which the read data needs to be written;

In one embodiment, the processor 1101 is specifically configured to, when determining a target database to which read data needs to be written:

In one embodiment, the processor 1101 is specifically configured to, when invoking the target data reading component to read data to be synchronized from the target blockchain:

In another embodiment, a data processing component is further provided in the blockchain data synchronization system; the processor 1101 is specifically configured to, when writing the read data into the database:

and writing the data after the transmission processing into a database.

In one embodiment, the processor 1101 is specifically configured to, when invoking the data processing unit to perform transmission processing on the read data:

In one embodiment, the processor 1101 is specifically configured to, when calling the data processing unit to determine whether the read data satisfies the cache condition:

In another embodiment, the read data is written into the database by the target data writing means; the processor 1101 is specifically configured to, when calling the data processing unit to determine whether the read data satisfies the buffering condition:

In another embodiment, the processor 1101 is specifically configured to, when invoking the data processing unit to perform transmission processing on the read data:

acquiring data filtering conditions;

In another embodiment, a data processing component is further provided in the blockchain data synchronization system; the processor 1101 is further configured to:

In one embodiment, the processor 1101 is further configured to:

In addition, it should be noted that the present application also provides a computer-readable storage medium in which a computer program of the foregoing data processing method is stored, which when executed by a processor, performs the description of the data processing method in the present application embodiment. That is, when one or more processors loads and executes the computer program, the description of the data processing method in the embodiment may be implemented, which is not repeated herein, and the description of the beneficial effects of the same method is not repeated herein.

The computer readable storage medium may be the data processing apparatus provided in any one of the foregoing embodiments or an internal storage unit of the computer device, for example, a hard disk or a memory of the computer device. The computer readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card) or the like, which are provided on the computer device. Further, the computer-readable storage medium may also include both internal storage units and external storage devices of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

In one aspect of the present application, a computer program product or computer program is provided, the computer program product comprising a computer program stored in a computer readable storage medium. A processor of a computer device reads the computer program from a computer-readable storage medium, and the processor executes the computer program to cause the computer device to perform a method of data processing provided in an aspect of an embodiment of the present application.

In one aspect of the present application, there is provided another computer program product or computer program comprising a computer program which, when executed by a processor, implements the steps of the data processing method provided by the embodiments of the present application.

The steps in the method of the embodiment of the application can be sequentially adjusted, combined and deleted according to actual needs.

The units in the device of the embodiment of the application can be combined, divided and deleted according to actual needs.

The foregoing disclosure is only a partial examples of the present application and is not intended to limit the scope of the claims hereof, as such, equivalent variations as may be contemplated by the present application are within the scope of the present application.

Claims

1. The data processing method is characterized by being applied to a block chain data synchronization system, wherein N data reading components are arranged in the block chain data synchronization system, and N is a positive integer; each data reading component is for reading data from a blockchain of one chain type; the method comprises the following steps:

determining a target block chain to be synchronized;

determining a target data reading component which is matched with the target blockchain from the N data reading components according to the chain type of the target blockchain;

invoking the target data reading component to read data to be synchronized from the target blockchain; the method comprises the steps of,

and writing the read data into a database.

2. The method of claim 1, wherein M data writing units are further provided in the blockchain data synchronization system, M being a positive integer; each data writing component is used for writing data read from the blockchain into a database of one library type;

the writing of the read data into the database comprises the following steps:

determining a target database to which the read data needs to be written;

determining a target data writing component which is matched with the target database from the M data writing components according to the library type of the target database;

And calling the target data writing component to write the read data into the target database.

3. The method of claim 2, wherein the determining the target database to which the read data is to be written comprises:

acquiring synchronous configuration information aiming at the target block chain, and determining a target database to which the read data needs to be written according to a library identifier preset in the synchronous configuration information; or,

determining a database matched with the chain type of the target blockchain as a target database to which the read data needs to be written; or,

and determining a database matched with the data type of the read data as a target database to which the read data needs to be written.

4. The method of claim 1, wherein the invoking the target data reading component to read data to be synchronized from the target blockchain comprises:

acquiring historical synchronous data of the target block chain, wherein the historical synchronous data corresponds to a historical maximum block height;

and calling the target data reading component to read the data to be synchronized.

5. The method of claim 1, wherein the blockchain data synchronization system is further provided with a data processing component; the writing of the read data into the database comprises the following steps:

invoking the data processing component to perform transmission processing on the read data;

and writing the data after the transmission processing into a database.

6. The method of claim 5, wherein said invoking said data processing component to transfer process said read data comprises:

invoking the data processing component to judge whether the read data meets a cache condition;

if the cache condition is met, calling the data processing component to cache the read data into a cache space; the data after transmission processing refers to cache data in the cache space, and the cache data in the cache space supports batch writing into a database.

7. The method of claim 6, wherein said invoking the data processing component to determine whether the read data meets a cache condition comprises:

invoking the data processing component to detect the data quantity of the data read by the target data reading component in unit time;

and if the detected data quantity of the data read by the target data reading component in unit time is larger than the preset data quantity, determining that the read data meets the caching condition.

8. The method of claim 6, wherein the read data is written to a database by a target data writing component; the calling the data processing component to judge whether the read data meets a cache condition comprises the following steps:

invoking the data processing component, and detecting the data reading rate of the target data reading component and the data writing rate of the target data writing component;

and if the data reading rate is larger than the data writing rate, determining that the read data meets a caching condition.

9. The method of claim 5, wherein said invoking said data processing component to transfer process said read data comprises:

Acquiring data filtering conditions;

invoking the data processing component, and screening the read data according to the data filtering condition to obtain target data meeting the data filtering condition; wherein the data after the transmission processing refers to the target data.

10. The method of claim 5, wherein said invoking said data processing component to transfer process said read data comprises:

invoking the data processing component, and performing format conversion processing on the read data according to the data format of the database to obtain converted data; or,

invoking the data processing component, selecting a target number of data from the read data, and performing format conversion processing on the target number of data to obtain converted data;

wherein the transmission processed data refers to the converted data.

11. The method of claim 1, wherein the blockchain data synchronization system is further provided with a data processing component; the method further comprises the steps of:

invoking the data processing component to perform flow control processing on the read data; wherein the flow control process comprises: one or both of the data read rate and the data write rate are controlled.

12. The method of any one of claims 1-10, wherein the method further comprises:

when the chain type of the block chain to be synchronized, which is accessed by the block chain data synchronization system, is newly increased, a data reading component which is matched with the newly increased chain type is deployed in the block chain data synchronization system;

when the chain type of the blockchain to be synchronized which is accessed by the blockchain data synchronization system is reduced, disabling a data reading component adapted to the reduced chain type in the blockchain data synchronization system.

13. The method of any one of claims 2-10, wherein the method further comprises:

when the database type of the database accessed by the block chain data synchronization system is newly increased, a data writing component which is matched with the newly increased database type is deployed in the block chain data synchronization system;

when the database type of the accessed database of the blockchain data synchronization system is reduced, a data writing component matched with the reduced database type is forbidden in the blockchain data synchronization system.

14. A data processing apparatus, comprising:

a determining unit for determining a target blockchain to be synchronized;

The determining unit is further configured to determine, according to the chain type of the target blockchain, a target data reading unit adapted to the target blockchain from the N data reading units;

the processing unit is used for calling the target data reading component and reading data to be synchronized from the target block chain; the method comprises the steps of,

the processing unit is also used for writing the read data into the database.

15. A computer device, comprising:

a processor adapted to execute a computer program;

a computer readable storage medium having stored therein a computer program which, when executed by the processor, performs the data processing method according to any of claims 1-13.

16. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, performs the data processing method according to any of claims 1-13.

17. A computer program product, characterized in that the computer program product comprises a computer program or computer instructions which are executed by a processor to implement the data processing method according to any of claims 1-13.