CN114168685A - Novel database architecture based on block chain system and operation method - Google Patents
Novel database architecture based on block chain system and operation method Download PDFInfo
- Publication number
- CN114168685A CN114168685A CN202111531166.2A CN202111531166A CN114168685A CN 114168685 A CN114168685 A CN 114168685A CN 202111531166 A CN202111531166 A CN 202111531166A CN 114168685 A CN114168685 A CN 114168685A
- Authority
- CN
- China
- Prior art keywords
- database
- metadata
- blockchain system
- data
- blockchain
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a novel database architecture based on a block chain system, which comprises the following components: a built-in blockchain system and a traditional database system to support an external blockchain system. An external blockchain system typically has multiple nodes, each node having its own independent database system, and each database system having an internal blockchain system. The external blockchain system at least comprises 16 built-in blockchain sub-nodes in a general situation, and each built-in blockchain sub-system can independently operate to maintain the functions and the performances of the database system; all metadata is stored by the built-in blockchain system. Therefore, the data of the external blockchain system is stored in the traditional database system, but the metadata exists in the internal blockchain system, so that the high security that the metadata is not damaged and cannot be tampered and deleted is ensured.
Description
Technical Field
The invention belongs to the field of block chain database technology, a block chain system and database system related technology, and particularly relates to a novel database architecture and an operation method based on the block chain system.
Background
The blockchain is considered as a special database, but the data structure of the blockchain is far from the traditional database, and in addition, the performance of the blockchain is always problematic because the blockchain has a redundancy mechanism and a layer-by-layer encryption mechanism. The traditional database system can store a large amount of data, can be fragmented, can have expandability and fault tolerance, but the data stored on the database can be changed. The block chain system has the advantages that the expansibility is difficult, a special structure is needed for the expansibility, and the data is difficult to change due to the fact that the block chain system has an encryption mechanism. Both micro-soft and IBM have proposed their own blockchain databases, but each has its drawbacks. Microsoft's BlockchainDB blockchain database adopts partial consensus (consensus is performed only by nodes participating in transactions), and reduces security while improving efficiency. The original IBM blockchain database was actually a centralized design, placing the sequencing services on a centralized server to sequence the transaction sequence, and the centralized and blockchain number characteristics were violated.
The method has the advantages that the method is greatly improved for the blockchain system, shows great progress in system architecture and application flow, and presents the gradual merging situation of the blockchain system and the database system. The system is characterized in that a novel block chain system and a framework are continuously generated, the framework difference between the framework and a super account book is very large, and a transaction flow based on the novel block chain is different from a transaction mode of a traditional block chain system. However, in the current blockchain database system or cloud platform, metadata (metadata) is the most important subsystem. If the metadata system has a problem, the system needs to stop working and can continue to process the metadata after the metadata is processed.
Therefore, there is a need for a high performance, highly fault tolerant database system, particularly one that is capable of efficiently protecting metadata and is scalable.
Disclosure of Invention
The invention provides a high-performance and high-fault-tolerance database system, in order to solve one or more technical problems in the prior art, in particular to a database system which can effectively protect metadata and can be expanded, namely a novel database architecture based on a block chain system. The method has the characteristics that the characteristics of MySQL relationship lines are kept, the unstructured storage structure of MongoDB is used for reference, the performance of the method is excellent like Redis, metadata is stored by a block chain, and the high safety that the data is not damaged and is not tampered and deleted is guaranteed.
The invention aims to provide a novel database architecture based on a block chain system, which can support the operation of the block chain system and comprises the following steps:
the system comprises a built-in blockchain system, a database system and an external blockchain system, wherein the blockchain system is supported by the database system; wherein within the external blockchain system there are a plurality of nodes, each of the nodes comprising the database system comprising the internal blockchain system;
the external blockchain system is provided with a plurality of nodes, each node is provided with an independent sub-database, each independent database is provided with an independent internal blockchain sub-system, so that the external blockchain system comprises at least 16 internal blockchain link points, and each internal blockchain sub-system can independently operate to maintain the functions and the performances of the database system;
all metadata in the novel database architecture based on the blockchain system are managed by the built-in blockchain system and the external blockchain system, the metadata are stored in the built-in blockchain system and the external blockchain system, data are stored in the database system, a Connection SDK is provided externally, the Connection SDK is provided with an SDK Api, and a user uses the SDK Api to perform CRUD operation of a table on the database system based on an application program, wherein the CRUD operation represents creation, reading, updating and deletion;
the metadata includes:
DB: data information describing the entire database system;
name: describing database names or names of a certain table, wherein the same table cannot exist in the same database;
BaseDir: a path of the database on disk;
hash and pre-Hash: marking the current metadata version, if the current metadata version is updated, assigning a hash to a hash, updating the hash to a latest version hash, and if the hash is not equal to the hash, indicating that the metadata is modified and a serialization task needs to be carried on a block chain;
writer: marking the user right of owning writing;
reader: marking the user right of owning the reading;
TableList: describing all table information under the current database;
BlockList: describing all block information under the current table;
TxList: recording all service data information on the current block, wherein the data information is serialized on a magnetic disk;
ActiviTime: recording the last modification time of the current block, if the database is larger than the pool of the cache, it means that not all blocks can be cached in the pool, and at this time, the block with the farthest activiTime distance needs to be serialized on a magnetic disk for reservation;
table: describing basic information of a table below a database system;
block (Block): information describing the sub-blocks of the table;
index: describing all index data on the table;
tx: the most original transaction data of the user is stored, and is a character string type of JSON.
Preferably, among the plurality of nodes of the external blockchain system, part of the nodes use a traditional database system, the internal blockchain system is not used for storing the metadata, and part or all of the nodes use the novel database architecture based on the blockchain system for storing the metadata. Since each node operates independently, it is the preferred embodiment that can be selected as long as the function and performance can be met.
Preferably, each of the database systems uses a different kind of database, the database comprising a relational database.
Preferably, the database system forms a cluster, a plurality of Master processes, a backup process and a Connection control are arranged in the cluster, the Master processes are database processes for storing data, the backup process is a process for backing up a Master, when the Master processes fail, the backup process replaces the Master processes to complete work, process information of the Master processes and the backup process is integrated and stored on the block chain system for management, and the Connection space obtains global process information through the block chain system and establishes Connection, and then distributes a command of CRUD operation to each cluster for execution; the Master process is provided with a plurality of Master sub-processes which respectively and independently run, and the number of the Master sub-processes can be increased or decreased according to the requirements of different database systems; the backup process is provided with a plurality of backup sub-processes which respectively and independently run, and the number of the backup sub-processes can be increased or reduced according to the requirements of different database systems; the processes have different relations, wherein some processes can be masters, and some processes are Worker, and the Worker listens to the masters; the number of processes increases or decreases depending on the application environment, and on large machines there may be multiple processes to process and run, and on small systems the number of processes may decrease accordingly.
Preferably, in an external blockchain system, the database system in each node can use different databases, and can store data including structured data and unstructured data in different or mixed modes even if the same database is used.
Preferably, the stored metadata and data information includes 4 layers:
layer 1 data information: storing the basic principle of the database system so as to maintain the basic information of the database system, including the name of the database, the path of the database on a disk and list information, as a part of metadata;
and (3) layer 2 data information: the list of layer 1 data information points to data information of a Table structure of layer 2 data information as a part of metadata;
and information of layer 3 data: the table structure of the layer 2 data information points to the Block and Index data structure information as part of the metadata;
and 4-layer data information: actual transaction information, these are data.
Preferably, the novel database architecture based on the blockchain system uses simple and easy-to-understand object-oriented Query statements oql (object Query language) or uses a traditional database Query language (such as SQL) to perform the CRUD operation of data; the novel database architecture based on the blockchain system has a database expansibility mechanism and a metadata fault tolerance mechanism.
The invention also aims to provide an operation method of a novel database architecture based on a block chain system, which is realized by adopting a Request-Response mode and comprises the following steps:
step 1, a client initiates a request through connection, and the request is delivered to an authority manager to verify the authority;
step 2, after passing the verification, submitting the request to other verification tasks for verification;
step 3, after the verification of the other auditing tasks, submitting the information to a Table Table, and analyzing whether the metadata of the Index needs to be modified or not by the Table Table;
step 4, calculating a Block Id of the metadata to be operated, carrying out CRUD operation on the metadata, and sending an operation result to a corresponding Block manager according to the Block Id;
step 5, sending the information to Pool, and acquiring a Block entity according to the Block Id, wherein the acquisition is realized through a cache or a disk;
step 6, executing corresponding CRUD operation according to the obtained Block entity and the request;
and 7, writing the operated result into a response object and returning the result to the connection.
Preferably, the step 4 adopts different BlockManager algorithms, including: the Table Table handles multiple tables at once; the Blockmanager creates a plurality of blocks each time; the unused Block can be stored first and used when needed later.
Preferably, the method further comprises: metadata updates, including: all metadata are completely stored in a memory except a block chain system to guarantee performance, when a process is started each time, whether the Hash of the metadata is consistent with the Hash of the metadata is firstly inquired, and if the Hash of the metadata is consistent with the Hash of the metadata, the process is executed; if not, performing "PreHash = Hash" after reading back the updated metadata from the blockchain system; and when the database needs to update the metadata, the metadata is changed, the Hash is updated, and the updated metadata is stored in the built-in block chain system.
The invention has the beneficial effects that:
a high-performance and high-fault-tolerance database system is provided, particularly a database system which can effectively protect metadata and can be expanded, namely a novel database architecture based on a block chain system. The method has the characteristics that the characteristics of MySQL relationship lines are kept, the unstructured storage structure of MongoDB is used for reference, the performance of the method is excellent like Redis, metadata is stored by a block chain, and the high safety that the data is not damaged and is not tampered and deleted is guaranteed.
The above and other objects, advantages and features of the present invention will become more apparent to those skilled in the art from the following detailed description of specific embodiments thereof, taken in conjunction with the accompanying drawings.
Drawings
Some specific embodiments of the invention will be described in detail hereinafter, by way of illustration and not limitation, with reference to the accompanying drawings. The same reference numbers in the drawings identify the same or similar elements or components. Those skilled in the art will appreciate that the drawings are not necessarily drawn to scale. The objects and features of the present invention will become more apparent in view of the following description taken in conjunction with the accompanying drawings, in which:
FIG. 1 is a conceptual illustration of a library-in-library principle according to a preferred embodiment of the present invention;
FIG. 2 is a diagram of a novel database architecture based on a blockchain system according to a preferred embodiment of the present invention.
Fig. 3 is a data structure diagram of metadata according to a preferred embodiment of the present invention.
Fig. 4 is a database process flow according to a preferred embodiment of the present invention.
Fig. 5 is a process flow diagram according to a preferred embodiment of the present invention.
Detailed Description
In order to make the present invention more comprehensible with respect to its gist, the present invention will be further described with reference to the accompanying drawings and examples. In the following description, numerous specific details and specific examples are set forth in order to provide a more thorough understanding of the present invention and to provide a thorough understanding of the present invention. While this invention is susceptible of embodiment in many different forms than that described herein, there will be many equivalents to those skilled in the art which incorporate such variations and modifications without departing from the spirit and scope of the invention as defined by the appended claims and their equivalents.
In the following description, numerous specific details and specific examples are set forth in order to provide a more thorough understanding of the present invention and to provide a thorough understanding of the present invention. While this invention is susceptible of embodiment in many different forms than that described herein, there will be many equivalents to those skilled in the art which incorporate such variations and modifications without departing from the spirit and scope of the invention as defined by the appended claims and their equivalents.
As shown in fig. 1, the principle of the architecture of the present embodiment: since the blockchain data is still stored in the database system, the blockchain system is used in the database system to form a library-in-library architecture: an external blockchain system contains one database system per node, and this database system uses another (internal) blockchain system to support database operations. Because the block chain system is in the database system, the block chain system can bear the operation work corresponding to a plurality of redundancy and fault-tolerant mechanisms, thereby effectively preventing the possibility of cluster damage caused by power failure and not worrying about cluster breakdown caused by misoperation.
The present embodiment provides a novel database architecture based on a blockchain system, which includes:
a built-in blockchain system, a database system to support operation of an external blockchain system, wherein said database system is included in said external blockchain system, said external blockchain system including a plurality of nodes, each node including one of said database systems, each of said database systems including one of said built-in blockchain systems. Since an external blockchain system has multiple nodes, each node has a separate database, and each separate database is provided with its own separate internal blockchain system, an external blockchain system comprises at least 16 internal blockchain nodes, and each internal blockchain system can independently operate to maintain the functions and performance of the database system.
Preferably, some blockchains may have more than 4 nodes, or less than 4 nodes, including an external blockchain system or an internal blockchain system. The more nodes, the poorer the performance, but the better the reliability; the fewer nodes, the better the performance, and if there are 4 nodes less, the blockchain system may have difficulty maintaining the byzantine general agreement.
A plurality of nodes are arranged in an external blockchain system, and the more nodes are arranged, the more built-in blockchain systems are arranged, and the system performance is better.
In a preferred embodiment, among the plurality of nodes of the external blockchain system, a part of the nodes may use a conventional database system, and the metadata is stored without using the internal blockchain system, and a part or all of the nodes use the novel database architecture based on the blockchain system to store the metadata. Since each node operates independently, it is the preferred embodiment that can be selected as long as the function and performance can be met.
In a preferred embodiment, each of the database systems uses a different kind of database, including a relational database or other database system.
Because the external blockchain system can be the internal blockchain system of another database system, a hierarchical complex blockchain network can be established
As shown in fig. 2, all metadata in the novel database architecture based on the blockchain system is handed over to the built-in blockchain system, the metadata is stored in the built-in blockchain system and the external blockchain system, data is stored in the database system, a Connection SDK is externally provided, the Connection SDK has an SDK Api, and a user performs a CRUD operation on the database system based on an application program by using the SDK Api, wherein the CRUD operation represents creation (Create), Read (Read), Update (Update), and Delete (Delete).
As a preferred embodiment, the database system forms a cluster, a plurality of Master processes, a backup process and a Connection control are arranged in the cluster, the Master processes are database processes for storing data, the backup process is a process for backing up a Master, when the Master processes fail, the backup process replaces the Master processes to complete work, the process information of the Master processes and the backup process is centralized together and stored on the block chain system for management, the Connection space obtains global process information through the block chain system and establishes Connection, and then the command of the CRUD operation is distributed to each cluster for execution, the more the processes are, the stronger the concurrency capability of the cluster is, and the stronger the disaster tolerance capability is.
As a preferred embodiment, the Master process has a plurality of Master sub-processes (in this embodiment, there are 3 Master sub-processes) that run independently, and the number of the Master sub-processes can be increased or decreased according to the requirements of different database systems.
As a preferred embodiment, the backup process has a plurality of backup sub-processes (in this embodiment, there are 3 backup sub-processes) that operate independently, and the number of the backup sub-processes may be increased or decreased according to the requirements of different database systems.
In a preferred embodiment, the database system in each node in an external blockchain system can use different databases, and can store data including structured data and unstructured data in different or mixed modes even if the same database is used.
As a preferred embodiment, the processes may have different relationships, for example, some processes may be Master and some processes may be Worker, which listens to Master.
The number of these processes may be increased or decreased as a preferred embodiment. On large machines, there may be multiple processes to process, while on small systems, there may be fewer processes.
In the present embodiment, as shown in fig. 2, the blockchain system includes 4 nodes (node 1, node 2, node 3, node 4). In a practical system, the number of nodes may increase; when the system fails, the system can continue to operate by ensuring that 3 nodes still operate normally.
Fig. 3 shows a diagram of a metadata and data structure.
In a preferred embodiment, the database system stores data using a hybrid mode, the data including structured data as well as unstructured data. The stored data information comprises 4 layers:
layer 1 data information: storing basic principles of a database system so as to maintain basic information of the database system, wherein the basic information comprises database names, paths of the database on a disk, lists (Tablelist) and the like, and the information is used as a part of metadata;
and (3) layer 2 data information: the list of layer 1 data information points to (Table) data information of a Table structure of layer 2 data information as part of the metadata;
and information of layer 3 data: the table structure of layer 2 data information points to Block (Block) and Index (Index) data structure information as part of the metadata;
and 4-layer data information: actual transaction information as data.
Thus, the data structure can start from the list, go to the list and then go to the block, and the block chain system can be inquired at high speed; and yet use index data to assist in high speed queries.
In a preferred embodiment, the layer 4 data information may be partially adjusted according to different database requirements, for example, a change in storage structure may cause some updates to the data structure, but the basic principle is consistent: from the list to the table, from the table to the block and index, and finally to the transaction information.
See fig. 3, where:
DB: data information describing the entire database system;
name: describing database names or names of a certain table, wherein the same table cannot exist in the same database;
BaseDir: a path of the database on disk;
hash and pre-Hash: marking the current metadata version, if the current metadata version is updated, assigning a hash to a hash, updating the hash to a latest version hash, and if the hash is not equal to the hash, indicating that the metadata is modified and a serialization task needs to be carried on a block chain;
writer: marking the user right of owning writing;
reader: marking the user right of owning the reading;
TableList: describing all table information under the current database;
BlockList: describing all block information under the current table;
TxList: recording all service data information on the current block, wherein the data information is serialized on a magnetic disk;
ActiviTime: recording the last modification time of the current block, if the database is larger than the pool of the cache, it means that not all blocks can be cached in the pool, and at this time, the block with the farthest activiTime distance needs to be serialized on a magnetic disk for reservation;
table: describing basic information of a table below a database system;
block (Block): information describing the sub-blocks of the table;
index: describing all index data on the table;
tx: the most original transaction data of the user is stored, and is a character string type of JSON.
As a preferred embodiment, the novel database architecture based on the blockchain system uses simple and easy-to-understand object-oriented Query statements oql (object Query language) or uses a conventional database Query language (e.g., SQL) for the CRUD operation of data. If SQL is used, the workload of the database is large, and the used object-oriented query language is easier; the novel database architecture based on the block chain system has a database expansibility mechanism, and a traditional database can use different expansibility mechanisms, such as fragmentation and other mechanisms; in the embodiment, the database is divided into two different data types, namely data and metadata, wherein a part of the data can use a traditional database, and the metadata occupies small storage, so that the database has expansibility; the novel database architecture based on the blockchain system has a metadata fault-tolerant mechanism, the traditional database has a fault-tolerant mechanism, and the built-in blockchain system also has a fault-tolerant mechanism.
As shown in fig. 4, the preferred embodiment further provides an operation method of a novel database architecture based on a blockchain system, which is implemented by using a Request-Response mode, and includes:
step 1, a client initiates a request through connection, and the request is delivered to an authority manager to verify the authority;
step 2, after passing the verification, submitting the request to other verification tasks for verification;
step 3, after the verification of the other auditing tasks, submitting the information to a Table Table, and analyzing whether the metadata of the Index needs to be modified or not by the Table Table;
step 4, calculating a Block Id of the metadata to be operated, carrying out CRUD operation on the metadata, and sending an operation result to a corresponding Block manager according to the Block Id;
step 5, sending the information to Pool, and acquiring a Block entity according to the Block Id, wherein the acquisition is realized through a cache or a disk;
step 6, executing corresponding CRUD operation according to the obtained Block entity and the request;
and 7, writing the operated result into a response object and returning the result to the connection.
As a preferred embodiment, the above method may have some adjustments or optimizations, including: for example, the optimization algorithm is implemented in step 3; while step 4 employs a different BlockManager algorithm. For example, a Table may process multiple tables at once; the Block manager can create a plurality of blocks each time, and the blocks which are not used can be stored firstly and then used when needed. But none of the optimizations changes the principle of the whole flow, namely the management from the table to the Block.
As a preferred embodiment, the method further comprises: metadata updates, including: all metadata are completely stored in a memory except a block chain system to guarantee performance, when a process is started each time, whether the Hash of the metadata is consistent with the Hash of the metadata is firstly inquired, and if the Hash of the metadata is consistent with the Hash of the metadata, the process is executed; if not, performing "PreHash = Hash" after reading back the updated metadata from the blockchain system; when the database needs to update the metadata, the metadata is changed, the Hash is updated, and the updated metadata is stored in the built-in block chain system; since the blockchain system uses Merkle trees, it is very easy to find the metadata locations that need to be updated.
The preferred embodiment shown in fig. 5:
the preferred embodiment provides a distributed database based on a blockchain, wherein all metadata are managed by a blockchain system, a Connection SDK is provided externally, and an App uses the SDK Api to perform table CRUD operation on the database.
And (3) testing and performing:
corresponding novel database system software has been developed, and is implemented in a manner that a CPU i 78 kernel 16 thread, a memory 16G ddr 42200 Mhz, a disk: 400Gssd 400M/s, network card: compared with the industry named databases Redis and MongoDB under the environment of 1000M/s and the size of a system cache pool of 2GB, the performance of the MongoDB is almost the same as that of the Redis, but is much better than that of the MongoDB. The larger the data volume is, the better the performance of the patent is than Redis. However, the performance of MongoDB is very poor. For example, in a 1 million transaction environment, the system of the present invention can run 8.6 million transactions per second, Redis can run 8.85 million transactions, and MongoDB can only run 1 million transactions. Under the environment of 1 million transactions, the invention can reach 7.5 million transactions, Redis can reach 6.98 million transactions, and MongoDB can only reach 1.2 million transactions.
This embodiment uses the Tiande blockchain system to store metadata, whereas Redis and MongoDB do not, the system performance of the present invention greatly surpasses both famous database systems in problematic environments. This also means that the blockchain system in our country has matured to be used in high throughput database applications.
The embodiment provides a high-performance and high-fault-tolerance database system, in particular to a database system which can effectively protect metadata and can be expanded, namely a novel database architecture based on a block chain system. The method has the characteristics that the characteristics of MySQL relationship lines are kept, the unstructured storage structure of MongoDB is used for reference, the performance of the method is excellent like Redis, metadata is stored by a block chain, and the high safety that the data is not damaged and is not tampered and deleted is guaranteed.
While the present invention has been described with reference to the particular illustrative embodiments, it is not to be restricted by the embodiments but only by the appended claims. It will be understood by those skilled in the art that variations and modifications of the embodiments of the present invention can be made without departing from the scope and spirit of the invention.
Claims (10)
1. A novel database architecture based on blockchain system, the novel database architecture can support a blockchain system to operate, comprising:
the system comprises an internal blockchain system and a database system, wherein the blockchain system supported by the internal blockchain system is an external blockchain system; wherein said one external blockchain system has a plurality of nodes therein, each of said nodes comprising said database system, said data system comprising said internal blockchain system;
the external blockchain system is provided with a plurality of nodes, each node is provided with an independent sub-database, each independent database is provided with an independent internal blockchain sub-system, so that the external blockchain system comprises at least 16 internal blockchain link points, and each internal blockchain sub-system can independently operate to maintain the functions and the performances of the database system;
all metadata in the novel database architecture based on the blockchain system are managed by the built-in blockchain system, data are stored in a traditional database system, a Connection SDK is provided externally, the Connection SDK is provided with an SDK Api, and a user uses the SDK Api to perform CRUD operation of a table on the database system based on an application program, wherein the CRUD operation represents creation, reading, updating and deletion;
the metadata includes:
DB: data information describing the entire database system;
name: describing database names or names of a certain table, wherein the same table cannot exist in the same database;
BaseDir: a path of the database on disk;
hash and pre-Hash: marking the current metadata version, if the current metadata version is updated, assigning a hash to a hash, updating the hash to a latest version hash, and if the hash is not equal to the hash, indicating that the metadata is modified and a serialization task needs to be carried on a block chain;
writer: marking the user right of owning writing;
reader: marking the user right of owning the reading;
TableList: describing all table information under the current database;
BlockList: describing all block information under the current table;
TxList: recording all service data information on the current block, wherein the data information is serialized on a magnetic disk;
ActiviTime: recording the last modification time of the current block, if the database is larger than the pool of the cache, it means that not all blocks can be cached in the pool, and at this time, the block with the farthest activiTime distance needs to be serialized on a magnetic disk for reservation;
table: describing basic information of a table below a database system;
block (Block): information describing the sub-blocks of the table;
index: describing all index data on the table;
tx: the most original transaction data of the user is stored, and is a character string type of JSON.
2. The novel database architecture based on blockchain system of claim 1, wherein the external blockchain system has a plurality of nodes, some of which can use a traditional database system and do not use an internal blockchain system to store metadata, and some or all of which use the novel database architecture based on blockchain system to store metadata;
since each node operates independently, it is the preferred embodiment that can be selected as long as the function and performance can be met.
3. The novel database architecture of claim 1, wherein each database system uses different kinds of databases, and the databases comprise relational databases.
4. The novel database architecture based on the blockchain system as claimed in claim 3, wherein the database system forms a cluster, a plurality of Master processes, a backup process and a Connection control are arranged in the cluster, the Master processes are database processes for storing data, the backup process is a process responsible for backing up a Master, when the Master processes fail, the backup process replaces the Master processes to complete work, the process information of the Master processes and the backup process is centralized together and stored on the blockchain system for management, and the Connection space obtains global process information through the blockchain system and establishes Connection, and then distributes a command of CRUD operation to each cluster for execution; the Master process is provided with a plurality of Master sub-processes which respectively and independently run, and the number of the Master sub-processes can be increased or decreased according to the requirements of different database systems; the backup process is provided with a plurality of backup sub-processes which respectively and independently run, and the number of the backup sub-processes can be increased or reduced according to the requirements of different database systems; the processes have different relations, wherein some processes can be masters, and some processes are Worker, and the Worker listens to the masters; the number of processes increases or decreases depending on the application environment, and on large machines there may be multiple processes to process and run, and on small systems the number of processes may decrease accordingly.
5. The novel database architecture based on blockchain system as claimed in claim 4, wherein the database system in each node in an external blockchain system can use different databases, even if the same database is used, the database system can use different or mixed mode to store data, wherein the data includes structured data and unstructured data.
6. The novel database architecture of claim 5, wherein the stored metadata and data information comprises 4 layers:
layer 1 data information: storing the basic principle of the database system so as to maintain the basic information of the database system, including the name of the database, the path of the database on a disk and list information, as a part of metadata;
and (3) layer 2 data information: the list of layer 1 data information points to data information of a Table structure of layer 2 data information as a part of metadata;
and information of layer 3 data: the table structure of the layer 2 data information points to the Block and Index data structure information as part of the metadata;
and 4-layer data information: actual transaction information as data.
7. The blockchain system based novel database architecture of claim 1, wherein the blockchain system based novel database architecture uses simple and easy to understand object Oriented Query Language (OQL) or uses traditional database query language for the CRUD operation of data; the novel database architecture based on the blockchain system has a database expansibility mechanism and a metadata fault tolerance mechanism.
8. A method for operating a novel database architecture based on a blockchain system according to any one of claims 1 to 7, wherein the novel database architecture is implemented by using a Request-Response mode, and the method comprises the following steps:
step 1, a client initiates a request through connection, and the request is delivered to an authority manager to verify the authority;
step 2, after passing the verification, submitting the request to other verification tasks for verification;
step 3, after the verification of the other auditing tasks, submitting the information to a Table Table, and analyzing whether the metadata of the Index needs to be modified or not by the Table Table;
step 4, calculating a Block Id of the metadata to be operated, carrying out CRUD operation on the metadata, and sending an operation result to a corresponding Block manager according to the Block Id;
step 5, sending the information to Pool, and acquiring a Block entity according to the Block Id, wherein the acquisition is realized through a cache or a disk;
step 6, executing corresponding CRUD operation according to the obtained Block entity and the request;
and 7, writing the operated result into a response object and returning the result to the connection.
9. The method according to claim 8, wherein the step 4 adopts different BlockManager algorithms, and includes: the Table Table handles multiple tables at once; the Blockmanager creates a plurality of blocks each time; the unused Block can be stored first and used when needed later.
10. The method of claim 8, further comprising: metadata updates, including: all metadata are completely stored in a memory except a block chain system to guarantee performance, when a process is started each time, whether the Hash of the metadata is consistent with the Hash of the metadata is firstly inquired, and if the Hash of the metadata is consistent with the Hash of the metadata, the process is executed; if not, performing "PreHash = Hash" after reading back the updated metadata from the blockchain system; and when the database needs to update the metadata, the metadata is changed, the Hash is updated, and the updated metadata is stored in the built-in block chain system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111531166.2A CN114168685B (en) | 2021-12-15 | 2021-12-15 | Novel database architecture based on blockchain system and operation method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111531166.2A CN114168685B (en) | 2021-12-15 | 2021-12-15 | Novel database architecture based on blockchain system and operation method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114168685A true CN114168685A (en) | 2022-03-11 |
| CN114168685B CN114168685B (en) | 2023-07-18 |
Family
ID=80486588
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111531166.2A Active CN114168685B (en) | 2021-12-15 | 2021-12-15 | Novel database architecture based on blockchain system and operation method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114168685B (en) |
Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106897878A (en) * | 2017-02-27 | 2017-06-27 | 钱德君 | A kind of intelligent contract type implementation of quantum catenary system |
| CN107766453A (en) * | 2017-09-26 | 2018-03-06 | 上海策赢网络科技有限公司 | Data base management method, device and storage medium based on block chain |
| CN108023879A (en) * | 2017-11-30 | 2018-05-11 | 济南浪潮高新科技投资发展有限公司 | A kind of NB-IOT device data acquisition system and methods based on block chain |
| CN110019235A (en) * | 2017-12-29 | 2019-07-16 | 百度在线网络技术（北京）有限公司 | Data storage, restoration methods, device, equipment and medium based on block chain |
| CN110232080A (en) * | 2019-05-23 | 2019-09-13 | 智慧谷(厦门)物联科技有限公司 | A kind of method for quickly retrieving based on block chain |
| US20190288850A1 (en) * | 2016-08-12 | 2019-09-19 | ALTR Solutions, Inc. | Decentralized database optimizations |
| CN111241590A (en) * | 2018-11-29 | 2020-06-05 | 华为技术有限公司 | Database system, node and method |
| CN111406252A (en) * | 2019-11-06 | 2020-07-10 | 支付宝(杭州)信息技术有限公司 | Consensus of error correction code based shared blockchain data storage |
| CN111400761A (en) * | 2020-03-17 | 2020-07-10 | 吉林亿联银行股份有限公司 | Data sharing method and device, storage medium and electronic equipment |
| CN112035466A (en) * | 2020-07-29 | 2020-12-04 | 北京智融云河科技有限公司 | Block chain query external index development framework |
| CN112115116A (en) * | 2020-08-05 | 2020-12-22 | 西安电子科技大学 | Block chain-based high-performance tamper-resistant database construction method |
| CN112134834A (en) * | 2020-05-19 | 2020-12-25 | 北京天德科技有限公司 | Data lake system architecture based on block chain |
| WO2021034274A1 (en) * | 2019-08-20 | 2021-02-25 | Singapore University Of Technology And Design | Blockchain for operational data security in industrial control systems |
| CN112564985A (en) * | 2020-12-24 | 2021-03-26 | 南京联成科技发展股份有限公司 | Safe operation and maintenance management method based on block chain |
| CN112602076A (en) * | 2018-08-24 | 2021-04-02 | 甲骨文国际公司 | DAG-based transaction processing method and system in distributed ledger |
| CN112714172A (en) * | 2020-12-23 | 2021-04-27 | 平安普惠企业管理有限公司 | Data synchronization method and device based on block chain technology and computer equipment |
| US20210182423A1 (en) * | 2019-01-31 | 2021-06-17 | Salesforce.Com, Inc. | Systems, methods, and apparatuses for storing pii information via a metadata driven blockchain using distributed and decentralized storage for sensitive user information |
-
2021
- 2021-12-15 CN CN202111531166.2A patent/CN114168685B/en active Active
Patent Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190288850A1 (en) * | 2016-08-12 | 2019-09-19 | ALTR Solutions, Inc. | Decentralized database optimizations |
| CN106897878A (en) * | 2017-02-27 | 2017-06-27 | 钱德君 | A kind of intelligent contract type implementation of quantum catenary system |
| CN107766453A (en) * | 2017-09-26 | 2018-03-06 | 上海策赢网络科技有限公司 | Data base management method, device and storage medium based on block chain |
| CN108023879A (en) * | 2017-11-30 | 2018-05-11 | 济南浪潮高新科技投资发展有限公司 | A kind of NB-IOT device data acquisition system and methods based on block chain |
| CN110019235A (en) * | 2017-12-29 | 2019-07-16 | 百度在线网络技术（北京）有限公司 | Data storage, restoration methods, device, equipment and medium based on block chain |
| CN112602076A (en) * | 2018-08-24 | 2021-04-02 | 甲骨文国际公司 | DAG-based transaction processing method and system in distributed ledger |
| CN111241590A (en) * | 2018-11-29 | 2020-06-05 | 华为技术有限公司 | Database system, node and method |
| US20210182423A1 (en) * | 2019-01-31 | 2021-06-17 | Salesforce.Com, Inc. | Systems, methods, and apparatuses for storing pii information via a metadata driven blockchain using distributed and decentralized storage for sensitive user information |
| CN110232080A (en) * | 2019-05-23 | 2019-09-13 | 智慧谷(厦门)物联科技有限公司 | A kind of method for quickly retrieving based on block chain |
| WO2021034274A1 (en) * | 2019-08-20 | 2021-02-25 | Singapore University Of Technology And Design | Blockchain for operational data security in industrial control systems |
| CN111406252A (en) * | 2019-11-06 | 2020-07-10 | 支付宝(杭州)信息技术有限公司 | Consensus of error correction code based shared blockchain data storage |
| CN111400761A (en) * | 2020-03-17 | 2020-07-10 | 吉林亿联银行股份有限公司 | Data sharing method and device, storage medium and electronic equipment |
| CN112134834A (en) * | 2020-05-19 | 2020-12-25 | 北京天德科技有限公司 | Data lake system architecture based on block chain |
| CN112035466A (en) * | 2020-07-29 | 2020-12-04 | 北京智融云河科技有限公司 | Block chain query external index development framework |
| CN112115116A (en) * | 2020-08-05 | 2020-12-22 | 西安电子科技大学 | Block chain-based high-performance tamper-resistant database construction method |
| CN112714172A (en) * | 2020-12-23 | 2021-04-27 | 平安普惠企业管理有限公司 | Data synchronization method and device based on block chain technology and computer equipment |
| CN112564985A (en) * | 2020-12-24 | 2021-03-26 | 南京联成科技发展股份有限公司 | Safe operation and maintenance management method based on block chain |
Non-Patent Citations (4)
| Title |
|---|
| KRITIKA RANI 等: ""Tampering Detection of Distributed Databases using Blockchain Technology"", 《2019 TWELFTH INTERNATIONAL CONFERENCE ON CONTEMPORARY COMPUTING》 * |
| KRITIKA RANI 等: ""Tampering Detection of Distributed Databases using Blockchain Technology"", 《2019 TWELFTH INTERNATIONAL CONFERENCE ON CONTEMPORARY COMPUTING》, 19 September 2019 (2019-09-19), pages 1 - 4 * |
| 蔡维德 等: ""库中库：新型区块链数据库架构"", 《信息安全研究 》 * |
| 蔡维德 等: ""库中库：新型区块链数据库架构"", 《信息安全研究 》, 5 May 2020 (2020-05-05), pages 429 - 436 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114168685B (en) | 2023-07-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108804112B (en) | Block chain settlement processing method and system | |
| CN111338766B (en) | Transaction processing method and device, computer equipment and storage medium | |
| JP4586019B2 (en) | Parallel recovery with non-failing nodes | |
| CA2913036C (en) | Index update pipeline | |
| Baker et al. | Megastore: Providing scalable, highly available storage for interactive services. | |
| US9146934B2 (en) | Reduced disk space standby | |
| EP2378421B1 (en) | Ownership reassignment in a shared-nothing database system | |
| US11132350B2 (en) | Replicable differential store data structure | |
| US20140032496A1 (en) | Information storage system and data replication method thereof | |
| US9886443B1 (en) | Distributed NFS metadata server | |
| US20090012932A1 (en) | Method and System For Data Storage And Management | |
| US9652346B2 (en) | Data consistency control method and software for a distributed replicated database system | |
| US10366075B2 (en) | Database management system and method | |
| US8832022B2 (en) | Transaction processing device, transaction processing method and transaction processing program | |
| US11494271B2 (en) | Dynamically updating database archive log dependency and backup copy recoverability | |
| US20150319265A1 (en) | Unique identifier for a transaction | |
| JP2012238061A (en) | Transaction processing device, transaction processing method, and transaction processing program | |
| US11714794B2 (en) | Method and apparatus for reading data maintained in a tree data structure | |
| US11461201B2 (en) | Cloud architecture for replicated data services | |
| US11966297B2 (en) | Identifying database archive log dependency and backup copy recoverability | |
| US10162841B1 (en) | Data management platform | |
| CN105956041A (en) | Data model processing method based on Spring Data for MongoDB cluster | |
| Yang | From Google file system to omega: a decade of advancement in big data management at Google | |
| CN108351801A (en) | Multi version removes manager | |
| CN114168685B (en) | Novel database architecture based on blockchain system and operation method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |