CN114079660B - High-performance distributed storage block data, time stamp, cross-chain communication and data collaboration method - Google Patents
High-performance distributed storage block data, time stamp, cross-chain communication and data collaboration method Download PDFInfo
- Publication number
- CN114079660B CN114079660B CN202111141592.5A CN202111141592A CN114079660B CN 114079660 B CN114079660 B CN 114079660B CN 202111141592 A CN202111141592 A CN 202111141592A CN 114079660 B CN114079660 B CN 114079660B
- Authority
- CN
- China
- Prior art keywords
- data
- block
- information
- nodes
- chain
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 238000004891 communication Methods 0.000 title claims abstract description 34
- 230000008569 process Effects 0.000 claims abstract description 23
- 239000012634 fragment Substances 0.000 claims description 30
- 238000012795 verification Methods 0.000 claims description 9
- 238000013507 mapping Methods 0.000 claims description 8
- 238000005516 engineering process Methods 0.000 claims description 6
- 238000002360 preparation method Methods 0.000 claims description 6
- 230000001960 triggered effect Effects 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 5
- 230000007246 mechanism Effects 0.000 claims description 5
- 230000015556 catabolic process Effects 0.000 claims description 4
- 238000006731 degradation reaction Methods 0.000 claims description 4
- 230000004069 differentiation Effects 0.000 claims description 4
- 230000010076 replication Effects 0.000 claims description 4
- 238000013461 design Methods 0.000 claims description 3
- 239000006185 dispersion Substances 0.000 claims description 3
- 238000000605 extraction Methods 0.000 claims description 3
- 238000007726 management method Methods 0.000 claims description 3
- 238000012858 packaging process Methods 0.000 claims description 3
- 230000003068 static effect Effects 0.000 claims description 3
- 230000001360 synchronised effect Effects 0.000 claims description 3
- 238000000926 separation method Methods 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/602—Providing cryptographic facilities or services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/64—Protecting data integrity, e.g. using checksums, certificates or signatures
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/04—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks
- H04L63/0428—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/12—Applying verification of the received information
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
- H04L67/1074—Peer-to-peer [P2P] networks for supporting data block transmission mechanisms
- H04L67/1078—Resource delivery mechanisms
- H04L67/108—Resource delivery mechanisms characterised by resources being split in blocks or fragments
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
- H04L9/32—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials
- H04L9/3236—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials using cryptographic hash functions
- H04L9/3239—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials using cryptographic hash functions involving non-keyed hash functions, e.g. modification detection codes [MDCs], MD5, SHA or RIPEMD
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
- H04L9/32—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials
- H04L9/3247—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols including means for verifying the identity or authority of a user of the system or for message authentication, e.g. authorization, entity authentication, data integrity or data verification, non-repudiation, key authentication or verification of credentials involving digital signatures
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L2463/00—Additional details relating to network architectures or network communication protocols for network security covered by H04L63/00
- H04L2463/121—Timestamp
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- General Engineering & Computer Science (AREA)
- Computer Hardware Design (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a high-performance distributed storage block data, time stamp, cross-chain communication and data collaboration method; s1, uploading block data information and effectively storing the block data information; s2, performing chain connection on the block data information; s3, carrying a time stamp when the data file information is uploaded; s4, enabling the data file information to be respectively stored in all nodes through distributed storage; s5, performing cross-link communication among different block chains to finish effective cross-link communication and improve the coordination of data information; the invention realizes the distributed storage of the data information of the block chain, completes the rapid identification process through the consensus algorithm, realizes the sliced storage of the data information, carries the time stamp on the data, realizes the cross-chain communication between different block chains and completes the data collaboration between the data information.
Description
Technical Field
The invention belongs to the technical field of block chains, and particularly relates to a high-performance distributed storage block data, a time stamp, cross-chain communication and data collaboration method.
Background
Blockchain involves many scientific and technical problems of mathematics, cryptography, internet and computer programming. From the application perspective, the blockchain is simply a distributed shared ledger and database, and has the characteristics of decentralization, non-falsification, whole trace, traceability, collective maintenance, disclosure transparency and the like. These features ensure the "honest" and "transparent" of the blockchain, laying a foundation for creating trust for the blockchain. The application scene with rich blockchain is basically based on the blockchain, so that the problem of information asymmetry can be solved, and collaborative trust and consistent actions among a plurality of subjects are realized. Blockchains are novel application modes of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanisms, encryption algorithms, and the like. Blockchains are an important concept of bitcoin, and are essentially a de-centralized database, and serve as a bottom technology of bitcoin, and are a series of data blocks which are generated by correlating by using a cryptography method, wherein each data block contains a batch of information of bitcoin network transaction and is used for verifying the validity (anti-counterfeiting) of the information and generating the next block, however, various problems still exist in various blockchains on the market.
The method and apparatus for distributed storage and data acquisition based on blockchain as disclosed in the grant bulletin CN112637297a, although it is realized that the terminal device configured with D2D protocol can rapidly upload data to the blockchain server, the blockchain server uses the blockchain technology to store in other nodes in a distributed manner. Meanwhile, all nodes can directly communicate through multiplexing cell resources (D2D protocol), data stored in other nodes in a distributed mode can be rapidly and efficiently obtained, but the problems that high-performance distributed storage data cannot be effectively carried out, time stamps are carried on the data, effective cross-link communication and data coordination are carried out and the like are solved, and therefore a high-performance distributed storage block data, time stamps, cross-link communication and data coordination method is provided.
Disclosure of Invention
The present invention is directed to a method for high performance distributed storage block data, time stamping, cross-link communication and data collaboration, so as to solve the problems set forth in the background art.
In order to achieve the above purpose, the present invention provides the following technical solutions: a high-performance distributed storage block data, time stamp, cross-chain communication and data cooperation method comprises the following method steps:
S1, uploading block data information and effectively storing: when the block storage of the data information is carried out, a request is sent to one node in a block chain, the quick determination is completed through a consensus algorithm, then the data file is fragmented, encryption and hash generation are carried out on each data fragment, and then copying, differentiation and recording are carried out on a ledger;
s2, performing chain connection on the block data information: after the slicing is carried out in the S1, all the slicing in each data file are connected, and branched chain type connection is carried out through an address timestamp, so that the organic data connection of all the slicing in the same data file is completed, and the rapidity of the data information during searching is improved;
S3, carrying a time stamp when the data file information is uploaded: when uploading data file information, carrying a time stamp on the block header of the data file information, carrying the same time stamp on the block header of each fragment of the data file information, finally connecting the blocks according to the sequence of block generation time to form a block chain, realizing distributed storage of the data file information according to different time stamps, and establishing a connection between each independent node through a P2P network, so that a decentralized distributed time stamp service system is formed for recording information data;
s4, enabling the data file information to be respectively stored in all nodes through distributed storage: the distributed storage is adopted to enable the data file information to be respectively stored in all nodes, so that the data file information can be rapidly processed when being queried and stored, the data collaboration operation is effectively improved, a server is arranged at the center of the blockchain, all the information of the blockchain nodes is received and forwarded, and the complexity of accessing all the nodes when the nodes query the data is reduced;
S5, performing cross-link communication among different block chains to complete effective cross-link communication and improve the coordination of data information: communication connection can be realized among different blockchains through notary mechanisms, side chains, relays, hash locking and distributed private key control among different blockchains.
Preferably, the consensus algorithm in S1 includes a workload proof, a benefit proof, a delegated benefit proof, and a practical bye-busy fault tolerance algorithm, where a state machine copy replication algorithm, that is, a service is modeled as a state machine, the state machine performs copy replication at different nodes of the distributed system, the copy of each state machine preserves the state of the service, and also implements operation of the service, the set of all copies is represented by capital R, each copy is represented by an integer from 0 to |r| -1, and assuming |r|=3f+1, where f is the maximum number of copies that may fail, although there may be more than 3f+1 copies, additional copies cannot improve reliability except for performance degradation.
Preferably, the data size of the slice in S1 is kept at 128M Bytes, and then the slice is written into the dat file in the form of sequential byte codes, and if the current writing file size plus the block size is detected to be greater than 128M Bytes in the process of serialization, a dat file is regenerated, and the specific serialization process is as follows:
The current dat file size npos is obtained, and the block size is additionally written into the dat file,
Serializing the block data and transaction data in the block, and appending the serialized data to the dat file,
During the writing of the data, the block and transaction related metadata is generated,
If the current writing file size plus the block size is detected to be less than or equal to 128M Bytes, a dat file is directly generated.
Preferably, the slicing in S1 includes the following steps:
Creating a data fragment: the storage system breaks the data into smaller pieces, a process called slicing, which breaks the data down into manageable pieces that can be distributed across multiple nodes;
Encrypt each fragment: after the sharding, the storage system needs to encrypt each data shard on the local system, the content owner has full control of this process, with the goal of ensuring that anyone other than the content owner cannot view/access the data in the shard, no matter where the data is located, whether the data is static or dynamic;
Generating a hash for each tile: the blockchain storage system generates a unique hash, i.e., a fixed length encrypted output string, from the sliced data or encryption key, which will be added to the ledger and the sliced metadata to link the transaction to the stored slice;
Copy each tile: the storage system replicates each fragment, so that enough redundant copies exist, the availability and the performance can be ensured, the performance degradation and the data loss can be prevented, the copy number of each fragment and the position of the fragments are determined by a content owner, and in the process, the content owner establishes a threshold value for the minimum copy number to be maintained so as to ensure that the data cannot be lost;
Distributing duplicate slices: the P2P network distributes replicated segments to geographically dispersed storage nodes, whether regional or global, multiple organizations or individuals owning the storage nodes, can trade off some type of compensation by leasing additional storage space, none of the entities can own all of the storage resources, or control the storage infrastructure, only the content owners can fully access all of their data, wherever they are located;
Record transaction to ledgers: the storage system records all transactions in the blockchain ledger and synchronizes this information between all nodes, the ledger storing detailed information about the transaction, which has transparency, verifiability, traceability, and tamper resistance since the ledger is based on blockchain technology.
Preferably, the timestamp in S3 is located in a block header, the block header and the block body together form a block, all transaction data in the time period in which the block is created is recorded in the block body, the records are organized by merkle trees, the hash value of merkle tree root is used as summary, abstract and fingerprint of all transaction records in the block, not only merkle tree root but also abstract of the previous block header, namely merkle tree root of the previous block, timestamp and height of the block are put into the block header: i.e. the number of blocks from the first block, the information of which block the present block is.
Preferably, the connection between the fragments in the S2 is performed by an accurate algorithm through address information and a hash algorithm, so that the connection between the fragments is maintained, the connection during query can be realized, and all the extraction of the data files during query is completed.
Preferably, the storage in S4 supports storing the structured data of the custom type, thereby increasing expansion; nervos to support the balance of performance and network dispersion by deploying different consensus algorithms through a producer, nervos proposes a new blockchain design, which is composed of five elements of a producer, a validder, a Type and an Identity of an application execution logic Generator capable of storing a plurality of data, wherein the five elements are used for autonomous and verification separation, nervos changes the original data input, verification and packaging processes, when an event X occurs, a client side firstly determines Identity through Identity verification, and simultaneously stores an out-of-chain state Y into the client side through mapping corresponding to consensus, the execution logic Generator transmits an event state Y to a node for accounting through hash, and the node only needs to verify whether the mapping on the chain is consistent with the under-chain event state Y, and if so, the next block is packaged.
Preferably, the stored state map indicates whether the link chain is consistent with the link data under the link chain or not through reverse logic, and the newly added node is a synchronous node, the accounting node carrying the calculation work is an operator or a participant, and the management contract can set the weight of the accounting node, so that better node voting resolution and dynamic addition are realized.
Preferably, the working process of each master node in the practical bayer and busy-tolerant algorithm is called a view, the view number is denoted by v,
The master nodes are selected by the common nodes in turn, the specific calculation process is that the master nodes p=v mod|R| (|R| is the number of the nodes),
The operation method comprises the following steps: during normal operation, receiving a transaction request of a client, after verifying a request identity, setting a number for the request, and broadcasting a pre-preparation message;
When a New master node is selected, according to the View-Change information collected by the master node, sending View-New information to enable other nodes to synchronize data;
The master node maintains heartbeats with all other nodes;
The main node is down, and the reelect is triggered due to the timeout of the heartbeat, so that the system is ensured to run stably;
The master node maliciously sends the message with the wrong number, and then the message is perceived by the duplicate node in the subsequent operation, because both the preparation and commit phases are broadcasted, and once the message is inconsistent, the view-change is triggered;
the master node does not send the received request, and the client retransmits the request to all the duplicate nodes and triggers view-change when the client does not reply after overtime;
the primary node falsifies the message, and because the data and the signature of the client are contained in the Request, the primary cannot falsify the message, other copies can verify the validity of the message a priori, otherwise, the primary node falsifies the message, and the primary node discards the message and triggers view-change.
Compared with the prior art, the invention has the beneficial effects that:
The invention realizes the distributed storage of the data information of the block chain, completes the rapid identification process through the consensus algorithm, quickens the storage and uploading of the data information in the block chain, realizes the slicing storage of the data information, quickens the storage speed and the query speed, is convenient for processing and use, carries the timestamp on the data, can facilitate the storage and the query of the data information, realizes the cross-chain communication of different block chains, completes the data collaboration among the data information, improves the data information transmission and connection among the block chains, and is convenient for use and operation.
Drawings
FIG. 1 is a schematic diagram of the steps of the present invention;
FIG. 2 is a schematic diagram showing a dat file serialization procedure according to the present invention;
FIG. 3 is a schematic diagram of a slicing step according to the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1-3, the present invention provides a technical solution: a high-performance distributed storage block data, time stamp, cross-chain communication and data cooperation method comprises the following method steps:
S1, uploading block data information and effectively storing: when the block storage of the data information is carried out, a request is sent to one node in a block chain, the quick determination is completed through a consensus algorithm, then the data file is fragmented, encryption and hash generation are carried out on each data fragment, and then copying, differentiation and recording are carried out on a ledger;
s2, performing chain connection on the block data information: after the slicing is carried out in the S1, all the slicing in each data file are connected, and branched chain type connection is carried out through an address timestamp, so that the organic data connection of all the slicing in the same data file is completed, and the rapidity of the data information during searching is improved;
S3, carrying a time stamp when the data file information is uploaded: when uploading data file information, carrying a time stamp on the block header of the data file information, carrying the same time stamp on the block header of each fragment of the data file information, finally connecting the blocks according to the sequence of block generation time to form a block chain, realizing distributed storage of the data file information according to different time stamps, and establishing a connection between each independent node through a P2P network, so that a decentralized distributed time stamp service system is formed for recording information data;
s4, enabling the data file information to be respectively stored in all nodes through distributed storage: the distributed storage is adopted to enable the data file information to be respectively stored in all nodes, so that the data file information can be rapidly processed when being queried and stored, the data collaboration operation is effectively improved, a server is arranged at the center of the blockchain, all the information of the blockchain nodes is received and forwarded, and the complexity of accessing all the nodes when the nodes query the data is reduced;
S5, performing cross-link communication among different block chains to complete effective cross-link communication and improve the coordination of data information: communication connection can be realized among different blockchains through notary mechanisms, side chains, relays, hash locking and distributed private key control among different blockchains.
In order to improve the fast identification operation on the data information uploaded by the node, in this embodiment, preferably, the consensus algorithm in S1 includes workload proof, rights proof, delegated rights proof and practical byesting fault-tolerant algorithm, and the practical byesting fault-tolerant algorithm is a state machine copy duplication algorithm, that is, the service is modeled as a state machine, the state machine copies at different nodes of the distributed system, each state machine copy stores the state of the service, and meanwhile, the operation of the service is implemented, the set formed by all copies is represented by a capital letter R, each copy is represented by an integer from 0 to |r| -1, and assuming that |r|=3f+1, where f is the maximum number of copies that may fail, although there may be more than 3f+1 copies, additional copies cannot improve reliability except for reducing performance.
In order to make the data information storage fast and convenient for query, in this embodiment, preferably, the data size of the slice in S1 is kept at 128M Bytes, and then the slice is written into the dat file in the form of sequential byte codes, and if the current written file size plus the block size is detected to be greater than 128M Bytes in the process of serialization, a dat file is regenerated, and the specific serialization process is as follows:
The current dat file size npos is obtained, and the block size is additionally written into the dat file,
Serializing the block data and transaction data in the block, and appending the serialized data to the dat file,
During the writing of the data, the block and transaction related metadata is generated,
If the current writing file size plus the block size is detected to be less than or equal to 128M Bytes, a dat file is directly generated.
In order to implement the slicing operation on the data information, in this embodiment, it is preferable that the slicing in S1 includes the following steps:
Creating a data fragment: the storage system breaks the data into smaller pieces, a process called slicing, which breaks the data down into manageable pieces that can be distributed across multiple nodes;
Encrypt each fragment: after the sharding, the storage system needs to encrypt each data shard on the local system, the content owner has full control of this process, with the goal of ensuring that anyone other than the content owner cannot view/access the data in the shard, no matter where the data is located, whether the data is static or dynamic;
Generating a hash for each tile: the blockchain storage system generates a unique hash, i.e., a fixed length encrypted output string, from the sliced data or encryption key, which will be added to the ledger and the sliced metadata to link the transaction to the stored slice;
Copy each tile: the storage system replicates each fragment, so that enough redundant copies exist, the availability and the performance can be ensured, the performance degradation and the data loss can be prevented, the copy number of each fragment and the position of the fragments are determined by a content owner, and in the process, the content owner establishes a threshold value for the minimum copy number to be maintained so as to ensure that the data cannot be lost;
Distributing duplicate slices: the P2P network distributes replicated segments to geographically dispersed storage nodes, whether regional or global, multiple organizations or individuals owning the storage nodes, can trade off some type of compensation by leasing additional storage space, none of the entities can own all of the storage resources, or control the storage infrastructure, only the content owners can fully access all of their data, wherever they are located;
Record transaction to ledgers: the storage system records all transactions in the blockchain ledger and synchronizes this information between all nodes, the ledger storing detailed information about the transaction, which has transparency, verifiability, traceability, and tamper resistance since the ledger is based on blockchain technology.
In order to implement carrying the timestamp on the data information, in this embodiment, preferably, the timestamp in S3 is located in a block header, where the block header and a block body together form a block, all transaction data in the time period in which the block is created is recorded in the block body, the records are organized by merkle trees, the hash value of the merkle tree root is used as a summary, and a "fingerprint" of all transaction records in the block, and the block header includes not only merkle tree roots but also the summary of the previous block header, that is, merkle tree roots of the previous block, the timestamp and the height of the present block: i.e. the number of blocks from the first block, the information of which block the present block is.
In order to realize the address connection between the fragments, in this embodiment, preferably, the connection between the fragments in S2 is performed by an accurate algorithm through address information and a hash algorithm, so that the connection between the fragments is maintained, the connection during the query can be realized, and all the extraction of the data file during the query is completed.
In order to realize the storage of the stored structured data with the custom type, in this embodiment, preferably, the storage in S4 supports the storage of the structured data with the custom type, thereby increasing expansion; nervos supporting the balance of performance and network dispersion by deploying different consensus algorithms through a producer, nervos proposes a new blockchain design, which is composed of five elements of a producer, a validder, a Type and an Identity of an execution logic Generator which can store a plurality of data, respectively, forming a distributed architecture which can be autonomous and independent of verification, nervos changes the original data input, verification and packaging processes, when event X occurs, a client firstly determines Identity through Identity verification, simultaneously stores the state Y outside the chain into the client through mapping corresponding to consensus, an execution logic Generator transmits the event state Y to a node which is billed through hash, only needs the validder to verify whether the mapping on the chain is consistent with the event state Y under the chain, if so, the chain is packaged into a next block, the stored mapping of the state is consistent with the data under the chain through reverse logic, and the newly added node is a synchronous node, and the billing party is a voting party or a contract management party can be better placed in the node, and the node is charged dynamically.
In order to implement the consensus algorithm and determine the certainty between nodes, in this embodiment, preferably, the working process of each master node in the practical bayer fault-tolerant algorithm is called a view, the view number is denoted by v,
The master nodes are selected by the common nodes in turn, the specific calculation process is that the master nodes p=v mod|R| (|R| is the number of the nodes),
The operation method comprises the following steps: during normal operation, receiving a transaction request of a client, after verifying a request identity, setting a number for the request, and broadcasting a pre-preparation message;
When a New master node is selected, according to the View-Change information collected by the master node, sending View-New information to enable other nodes to synchronize data;
The master node maintains heartbeats with all other nodes;
The main node is down, and the reelect is triggered due to the timeout of the heartbeat, so that the system is ensured to run stably;
The master node maliciously sends the message with the wrong number, and then the message is perceived by the duplicate node in the subsequent operation, because both the preparation and commit phases are broadcasted, and once the message is inconsistent, the view-change is triggered;
the master node does not send the received request, and the client retransmits the request to all the duplicate nodes and triggers view-change when the client does not reply after overtime;
the primary node falsifies the message, and because the data and the signature of the client are contained in the Request, the primary cannot falsify the message, other copies can verify the validity of the message a priori, otherwise, the primary node falsifies the message, and the primary node discards the message and triggers view-change.
The working principle and the using flow of the invention are as follows:
the first step, the block data information is uploaded, and effective storage is performed: when the block storage of the data information is carried out, a request is sent to one node in a block chain, the quick determination is completed through a consensus algorithm, then the data file is fragmented, encryption and hash generation are carried out on each data fragment, and then copying, differentiation and recording are carried out on a ledger;
secondly, the block data information is connected in a chain way: after the slicing is carried out in the S1, all the slicing in each data file are connected, and branched chain type connection is carried out through an address timestamp, so that the organic data connection of all the slicing in the same data file is completed, and the rapidity of the data information during searching is improved;
Step three, carrying a time stamp when the data file information is uploaded: when uploading data file information, carrying a time stamp on the block header of the data file information, carrying the same time stamp on the block header of each fragment of the data file information, finally connecting the blocks according to the sequence of block generation time to form a block chain, realizing distributed storage of the data file information according to different time stamps, and establishing a connection between each independent node through a P2P network, so that a decentralized distributed time stamp service system is formed for recording information data;
Fourth, the data file information is respectively stored in all nodes through distributed storage: the distributed storage is adopted to enable the data file information to be respectively stored in all nodes, so that the data file information can be rapidly processed when being queried and stored, the data collaboration operation is effectively improved, a server is arranged at the center of the blockchain, all the information of the blockchain nodes is received and forwarded, and the complexity of accessing all the nodes when the nodes query the data is reduced;
Fifthly, performing cross-link communication among different block chains to complete effective cross-link communication and improve the coordination of data information: communication connection can be realized among different blockchains through notary mechanisms, side chains, relays, hash locking and distributed private key control among different blockchains.
Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (9)
1. The high-performance distributed storage block data, time stamp, cross-chain communication and data cooperation method is characterized by comprising the following steps:
S1, uploading block data information and effectively storing: when the block storage of the data information is carried out, a request is sent to one node in a block chain, the quick determination is completed through a consensus algorithm, then the data file is fragmented, encryption and hash generation are carried out on each data fragment, and then copying, differentiation and recording are carried out on a ledger;
s2, performing chain connection on the block data information: after the slicing is carried out in the S1, all the slicing in each data file are connected, and branched chain type connection is carried out through an address timestamp, so that the organic data connection of all the slicing in the same data file is completed, and the rapidity of the data information during searching is improved;
S3, carrying a time stamp when the data file information is uploaded: when uploading data file information, carrying a time stamp on the block header of the data file information, carrying the same time stamp on the block header of each fragment of the data file information, finally connecting the blocks according to the sequence of block generation time to form a block chain, realizing distributed storage of the data file information according to different time stamps, and establishing a connection between each independent node through a P2P network, so that a decentralized distributed time stamp service system is formed for recording information data;
s4, enabling the data file information to be respectively stored in all nodes through distributed storage: the distributed storage is adopted to enable the data file information to be respectively stored in all nodes, so that the data file information can be rapidly processed when being queried and stored, the data collaboration operation is effectively improved, a server is arranged at the center of the blockchain, all the information of the blockchain nodes is received and forwarded, and the complexity of accessing all the nodes when the nodes query the data is reduced;
S5, performing cross-link communication among different block chains to complete effective cross-link communication and improve the coordination of data information: communication connection can be realized among different blockchains through notary mechanisms, side chains, relays, hash locking and distributed private key control among different blockchains.
2. The method for high-performance distributed storage block data, time stamping, cross-chain communication and data collaboration according to claim 1, wherein: the consensus algorithm in S1 includes workload proof, rights and interests proof, delegated rights and interests proof and practical bye' S family fault tolerance algorithm, where a state machine copy replication algorithm, i.e. a service is modeled as a state machine, the state machine performs copy replication at different nodes of the distributed system, the state of the service is saved by copies of each state machine, and meanwhile, the operation of the service is implemented, a set of all copies is represented by uppercase R, each copy is represented by an integer from 0 to |r| -1, and assuming |r|=3f+1, where f is the maximum number of copies that may fail, although there may be more than 3f+1 copies, additional copies cannot improve reliability except for performance reduction.
3. The method for high-performance distributed storage block data, time stamping, cross-chain communication and data collaboration according to claim 1, wherein: the data size of the slice in S1 is kept at 128M Bytes, and then the slice is written into the dat file in the form of sequential byte codes, and if the current writing file size plus the block size is detected to be greater than 128M Bytes in the process of serialization, a dat file is regenerated, and the specific serialization process is as follows:
The current dat file size npos is obtained, and the block size is additionally written into the dat file,
Serializing the block data and transaction data in the block, and appending the serialized data to the dat file,
During the writing of the data, the block and transaction related metadata is generated,
If the current writing file size plus the block size is detected to be less than or equal to 128M Bytes, a dat file is directly generated.
4. The method for high-performance distributed storage block data, time stamping, cross-chain communication and data collaboration according to claim 1, wherein: the slicing in the step S1 comprises the following steps:
Creating a data fragment: the storage system breaks the data into smaller pieces, a process called slicing, which breaks the data down into manageable pieces that can be distributed across multiple nodes;
Encrypt each fragment: after the sharding, the storage system needs to encrypt each data shard on the local system, the content owner has full control of this process, with the goal of ensuring that anyone other than the content owner cannot view/access the data in the shard, no matter where the data is located, whether the data is static or dynamic;
Generating a hash for each tile: the blockchain storage system generates a unique hash, i.e., a fixed length encrypted output string, from the sliced data or encryption key, which will be added to the ledger and the sliced metadata to link the transaction to the stored slice;
Copy each tile: the storage system replicates each fragment, so that enough redundant copies exist, the availability and the performance can be ensured, the performance degradation and the data loss can be prevented, the copy number of each fragment and the position of the fragments are determined by a content owner, and in the process, the content owner establishes a threshold value for the minimum copy number to be maintained so as to ensure that the data cannot be lost;
Distributing duplicate slices: the P2P network distributes replicated segments to geographically dispersed storage nodes, whether regional or global, multiple organizations or individuals owning the storage nodes, can trade off some type of compensation by leasing additional storage space, none of the entities can own all of the storage resources, or control the storage infrastructure, only the content owners can fully access all of their data, wherever they are located;
Record transaction to ledgers: the storage system records all transactions in the blockchain ledger and synchronizes this information between all nodes, the ledger storing detailed information about the transaction, which has transparency, verifiability, traceability, and tamper resistance since the ledger is based on blockchain technology.
5. The method for high-performance distributed storage block data, time stamping, cross-chain communication and data collaboration according to claim 1, wherein: the timestamp in S3 is in the block header, the block header and the block body together form a block, all transaction data in the time period in which the block is created are recorded in the block body, the records are organized by merkle trees, the hash value of merkle tree root is taken as the summary, abstract and fingerprint of all transaction records in the block, not only merkle tree root is in the block header, but also the abstract of the previous block header, namely merkle tree root of the previous block, timestamp and height of the block: i.e. the number of blocks from the first block, the information of which block the present block is.
6. The method for high-performance distributed storage block data, time stamping, cross-chain communication and data collaboration according to claim 1, wherein: and the connection between the fragments in the S2 is realized by an accurate algorithm through the address information and the hash algorithm, the connection between the fragments is maintained, the connection during inquiry can be realized, and the whole extraction of the data file during inquiry is completed.
7. The method for high-performance distributed storage block data, time stamping, cross-chain communication and data collaboration according to claim 1, wherein: the storage in the S4 supports the storage of the structured data of the custom type, so that the expansion is increased; nervos to support the balance of performance and network dispersion by deploying different consensus algorithms through a producer, nervos proposes a new blockchain design, which is composed of five elements of a producer, a validder, a Type and an Identity of an application execution logic Generator capable of storing a plurality of data, wherein the five elements are used for autonomous and verification separation, nervos changes the original data input, verification and packaging processes, when an event X occurs, a client side firstly determines Identity through Identity verification, and simultaneously stores an out-of-chain state Y into the client side through mapping corresponding to consensus, the execution logic Generator transmits an event state Y to a node for accounting through hash, and the node only needs to verify whether the mapping on the chain is consistent with the under-chain event state Y, and if so, the next block is packaged.
8. The method for high-performance distributed storage block data, time stamping, cross-chain communication and data collaboration according to claim 7, wherein: the state mapping is stored, the chain can push whether the data on the chain is consistent with the data under the chain or not through reverse logic, and the newly added node is a synchronous node, the accounting node for calculation work is an operator or a participant, and the management contract can set the weight of the accounting node, so that better node voting resolution and dynamic addition are realized.
9. The method for high-performance distributed storage block data, time stamping, cross-chain communication and data collaboration according to claim 2, wherein: the working process of each master node in the practical bayer fault-tolerant algorithm is called a view, the view number is denoted by v,
The main nodes are selected by the common nodes in turn, the specific calculation process is that the main nodes p=v mod|R|, the|R| is the number of the nodes,
The operation method comprises the following steps: during normal operation, receiving a transaction request of a client, after verifying a request identity, setting a number for the request, and broadcasting a pre-preparation message;
When a New master node is selected, according to the View-Change information collected by the master node, sending View-New information to enable other nodes to synchronize data;
The master node maintains heartbeats with all other nodes;
The main node is down, and the reelect is triggered due to the timeout of the heartbeat, so that the system is ensured to run stably;
The master node maliciously sends the message with the wrong number, and then the message is perceived by the duplicate node in the subsequent operation, because both the preparation and commit phases are broadcasted, and once the message is inconsistent, the view-change is triggered;
the master node does not send the received request, and the client retransmits the request to all the duplicate nodes and triggers view-change when the client does not reply after overtime;
the primary node falsifies the message, and because the data and the signature of the client are contained in the Request, the primary cannot falsify the message, other copies can verify the validity of the message a priori, otherwise, the primary node falsifies the message, and the primary node discards the message and triggers view-change.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111141592.5A CN114079660B (en) | 2021-09-28 | 2021-09-28 | High-performance distributed storage block data, time stamp, cross-chain communication and data collaboration method |
| PCT/CN2021/134240 WO2023050555A1 (en) | 2021-09-28 | 2021-11-30 | Method for high-performance distributed storage of block data and timestamp, cross-chain communication and data collaboration |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111141592.5A CN114079660B (en) | 2021-09-28 | 2021-09-28 | High-performance distributed storage block data, time stamp, cross-chain communication and data collaboration method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114079660A CN114079660A (en) | 2022-02-22 |
| CN114079660B true CN114079660B (en) | 2024-06-11 |
Family
ID=80283515
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111141592.5A Active CN114079660B (en) | 2021-09-28 | 2021-09-28 | High-performance distributed storage block data, time stamp, cross-chain communication and data collaboration method |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN114079660B (en) |
| WO (1) | WO2023050555A1 (en) |
Families Citing this family (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114185997B (en) * | 2022-02-17 | 2022-05-13 | 天津眧合数字科技有限公司 | Pet information credible storage system based on block chain |
| CN114238867B (en) * | 2022-02-28 | 2022-05-17 | 南开大学 | Automatic switching access method for distributed multi-backup copyright content |
| CN114629918B (en) * | 2022-03-02 | 2023-10-27 | 北京国联视讯信息技术股份有限公司 | Big data information processing method based on block chain |
| CN114510539B (en) * | 2022-04-18 | 2022-06-24 | 北京易鲸捷信息技术有限公司 | Method for generating and applying consistency check point of distributed database |
| CN115357932B (en) * | 2022-08-18 | 2023-11-28 | 山东浪潮智慧供应链科技有限公司 | Cross-chain secure storage method and device based on blockchain technology |
| CN115860744B (en) * | 2023-02-20 | 2023-05-09 | 中国信息通信研究院 | Processing method and device of cross-blockchain transaction, blockchain system and equipment |
| CN116308667A (en) * | 2023-03-25 | 2023-06-23 | 北京农夫铺子技术研究院 | Commodity big data intelligent storage system for meta-universe shopping platform |
| CN116567631B (en) * | 2023-07-06 | 2023-09-15 | 北京航空航天大学 | Mobile terminal security authentication method based on segmented block chain |
| CN116561182B (en) * | 2023-07-07 | 2023-09-19 | 华北电力大学 | Power data storage method and system based on block chain |
| CN116842029B (en) * | 2023-09-01 | 2023-11-17 | 天津理工大学 | Asset management method, device, computer equipment and storage medium |
| CN116915404B (en) * | 2023-09-13 | 2023-11-17 | 华中科技大学 | Engineering data cross-chain collaboration method and system based on block chain technology |
| CN116910126B (en) * | 2023-09-14 | 2023-11-24 | 国网山东省电力公司营销服务中心(计量中心) | System and method for conveniently storing, classifying and inquiring massive daily clear electric quantity data |
| CN117200977B (en) * | 2023-11-07 | 2024-01-19 | 天津市城市规划设计研究总院有限公司 | Method and system for hierarchical storage of blockchain data in smart city field |
| CN117319422B (en) * | 2023-11-28 | 2024-02-02 | 天津市城市规划设计研究总院有限公司 | Block chain blocking method and system for Internet of things data in urban planning field |
| CN117478304B (en) * | 2023-12-28 | 2024-03-01 | 湖南天河国云科技有限公司 | Block chain management method, system and computer equipment |
| CN117494219A (en) * | 2023-12-29 | 2024-02-02 | 智慧(东营)大数据有限公司 | Bill management system and method based on blockchain technology |
| CN117574447B (en) * | 2024-01-16 | 2024-04-16 | 中国标准化研究院 | Data storage evidence and traceability system based on blockchain technology |
| CN117763052B (en) * | 2024-02-22 | 2024-05-10 | 浩鲸云计算科技股份有限公司 | Data synchronization method and system for charging multi-center memory database |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106295401A (en) * | 2016-08-13 | 2017-01-04 | 深圳市樊溪电子有限公司 | A kind of read-only secure file storage system and method for block chain |
| CN111683058A (en) * | 2020-05-18 | 2020-09-18 | 国科微城市智能科技(南京)有限责任公司 | Data storage system based on block chain and storage method thereof |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11194837B2 (en) * | 2018-05-01 | 2021-12-07 | International Business Machines Corporation | Blockchain implementing cross-chain transactions |
| US10805094B2 (en) * | 2018-10-08 | 2020-10-13 | International Business Machines Corporation | Blockchain timestamp agreement |
| CN109587253B (en) * | 2018-12-11 | 2021-06-29 | 浙江大学 | Method for realizing distributed storage of files based on block chain |
| CN111507730A (en) * | 2020-05-12 | 2020-08-07 | 北京俩撇科技有限公司 | Block chain cross-chain method and device |
| CN112256663A (en) * | 2020-10-30 | 2021-01-22 | 深圳壹账通智能科技有限公司 | Block chain based distributed file storage method and system |
| CN112988764B (en) * | 2021-05-14 | 2022-05-10 | 北京百度网讯科技有限公司 | Data storage method, device, equipment and storage medium |
-
2021
- 2021-09-28 CN CN202111141592.5A patent/CN114079660B/en active Active
- 2021-11-30 WO PCT/CN2021/134240 patent/WO2023050555A1/en unknown
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106295401A (en) * | 2016-08-13 | 2017-01-04 | 深圳市樊溪电子有限公司 | A kind of read-only secure file storage system and method for block chain |
| WO2018032377A1 (en) * | 2016-08-13 | 2018-02-22 | 深圳市樊溪电子有限公司 | Read-only security file storage system for block chain, and method thereof |
| CN111683058A (en) * | 2020-05-18 | 2020-09-18 | 国科微城市智能科技(南京)有限责任公司 | Data storage system based on block chain and storage method thereof |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114079660A (en) | 2022-02-22 |
| WO2023050555A1 (en) | 2023-04-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114079660B (en) | High-performance distributed storage block data, time stamp, cross-chain communication and data collaboration method | |
| Tseng et al. | Blockchain for managing heterogeneous internet of things: A perspective architecture | |
| CN108846010B (en) | Method, system, computer system and storage medium for tracing product source in network | |
| US20220231869A1 (en) | Cross-blockchain mutual data storage | |
| CN109002725B (en) | Data processing system based on block chain | |
| CN113114759B (en) | Chain-crossing method and system for realizing multi-chain intercommunication | |
| US20230092484A1 (en) | Block chain-based block consensus method and related device | |
| Maniatis et al. | Secure history preservation through timeline entanglement | |
| Qi et al. | A reliable storage partition for permissioned blockchain | |
| CN111444277B (en) | Anti-terrorism information cooperative sharing platform and method based on blockchain technology | |
| US20230316273A1 (en) | Data processing method and apparatus, computer device, and storage medium | |
| KR20070110367A (en) | Method and apparatus for data management | |
| CN112307501B (en) | Big data system based on block chain technology, storage method and using method | |
| CN112615847B (en) | Data sharing and privacy protection method based on block chain | |
| CN110730081B (en) | Block chain network-based certificate revocation method, related equipment and medium | |
| CN114338040B (en) | Block chain node grouping multi-chain three-time consensus method | |
| CN114329528A (en) | File data management method and system based on block chain | |
| JP2024515022A (en) | Blockchain-based data processing method, device, equipment, and computer program | |
| CN115695449A (en) | Block chain digital collection source file cross-platform synchronization and transfer method | |
| CN110601857A (en) | Data interaction method based on local-global block chain cooperation | |
| Qi et al. | Blockchain based consensus checking in cloud storage | |
| Hwang et al. | Efficient real-time auditing and proof of violation for cloud storage systems | |
| CN114374502A (en) | Energy data uploading and consensus system and method based on-chain and off-chain cooperation | |
| CN113221135A (en) | Personnel archive data safe storage method based on block chain technology | |
| CN115796863A (en) | Consensus method of block chain system and related equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |