EP2976714B1

EP2976714B1 - Method and system for byzantine fault tolerant data replication

Info

Publication number: EP2976714B1
Application number: EP14714616.1A
Authority: EP
Inventors: Dan Dobre; Ghassan KARAME; Marko Vukolic
Original assignee: NEC Corp
Current assignee: NEC Corp
Priority date: 2013-03-20
Filing date: 2014-03-18
Publication date: 2017-05-03
Anticipated expiration: 2034-03-18
Also published as: JP6097880B2; JP2016517605A; WO2014147085A3; EP2976714A2; US9753792B2; WO2014147085A2; US20160283123A1

Description

The present invention relates to a method for a byzantine fault tolerant data replication with a plurality of 2f+1 data servers and one or more clients.
The present invention further relates to a system for byzantine fault tolerant data replication with a plurality of 2f+1 data servers and one or more clients Conventional solutions for Byzantine fault tolerant data replication are known from the non-patent literature of Liskov B et al: "Tolerating Byzantine Faulty Clients in a Quorum System", Distributed Computing Systems, 2006, ICDCS 2006, 26the IEEE International Conference on Distributed Computing Systems, Lisboa, Portugal 04-07 July 2006, Piscataway, NJ, USA, IEEE, NJ, USA 4 July 2006 (2006-07-04-) XP010927339 and from the non-patent literature of Eduardo A P Alchieri et al: "Proactive Byzantine Quorum Systems", 1 November 2009 (2009-11-01), On the Move to Meaningful Internet Systems: OTM 2009, Springer Berlin Heidelberg, Berlin, Heidelberg, pages 708-725, XP019133426.
Data replication provides in general availability and data durability in the presence of failures. For example a replicated data store features strong consistency if it displays a behavior of a non-replicated data store in which operations are executed sequentially. However, performance and scalability are major problems. With the increasing complexity of data-serving systems and an increased criticality of the stored data byzantine-fault tolerance was established as an alternative to crash-fault tolerance since a large spectrum of issues including simple outages, software-bugs, misconfigurations and even intrusions and malware can be masked and grouped together under the term "arbitrary failure" respectively "byzantine failure".
To implement a robust replicated data store, i.e. guaranteeing correctness under arbitrary failures, in the presence of asynchrony, concurrency and failures, quorum based replication is used. In the non-patent literature of Jean-Philippe Martin, Lorenzo Alvisi, Michael Dahlin: Minimal Byzantine Storage, DISC 2002: 311-325, it is shown that 3f+1 servers have to be used for byzantine-fault tolerance of f arbitrary failures. To read a correct value, a quorum Qr, i.e. a set of servers Qr, queried by a read operation needs to intersect a quorum Qw updated by a write operation in f+1 servers. That could guarantee that there is at least one correct server in the intersection, since at most f may be fail Byzantine. This translates to the following requirement:
1) |Qr| + |Qw| - n >= f+1, wherein n is the total number of servers to be used.
Furthermore, to avoid indefinitely waiting for crashed servers during a read or a write operation the quorums of servers, Qr and Qw can comprise of at most n-f servers. Since at most f servers may be faulty, n-f servers are guaranteed to eventually reply. This translates to the following requirement:
2) |Qr| = |Qw| <= n-f
By combining the two requirements 1) and 2) this leads to: $2 n - 2 f - n > = f + 1 \Rightarrow n > = 3 f + 1.$
Therefore conventional systems use 3f+1 servers on byzantine fault-tolerant storage.
For example, in the non-patent literature of Alysson Neves Bessani, Miguel P. Correia, Bruno Quaresma, Fernando André, Paulo Sousa: DepSky: dependable and secure storage in a cloud-of-clouds. EuroSys 2011: 31-46, 3f+1 servers or clouds are used to tolerate the failure up to f servers using byzantine quorum-based data replication.
In the further non-patent literature of Miguel Castro, Barbara Liskov: Practical byzantine fault tolerance and proactive recovery. ACM Trans. Comput. Syst. 20(4): 398-461 (2002) and Dahlia Malkhi, Michael K. Reiter: Byzantine Quorum Systems. Distributed Computing 11(4): 203-213 (1998) other conventional byzantine fault tolerance systems are shown.
Since tolerating Byzantine faults requires f servers more than needed to tolerate only crash failures, one of the problems of byzantine quorum-based data replication are the additional costs compared to crash tolerant systems. For example as shown in the non-patent literature of Rui Fan, Nancy A. Lynch: Efficient Replication of Large Data Objects. DISC 2003: 75-91, 2f+1 servers are used to tolerate f crashes, yet no byzantine faults can be tolerated.
It is therefore an objective of the present invention to provide a method and a system for a byzantine fault tolerant data replication which are more efficient, in particular in terms of storage server acquisition costs and maintenance costs as well as in terms of storage space.
Specifically, it is an objective to provide a method for byzantine fault-tolerant data replication with less than 3f+1 data servers.
It is a further objective of the present invention to provide a method and a system for byzantine fault tolerant data replication which have improved read performance and have a linear read scalability in terms of the number of storage servers.
It is an even further objective of the present invention to provide a method and a system for byzantine fault tolerant data replication, enabling transactional data access.
The aforementioned objectives are accomplished by a method in accordance with the appended independent claim 1 and a system in accordance with the appended independent claim 12. In claim 1 a method for byzantine fault tolerant data replication with a plurality of 2f+1 data servers and one or more clients is defined.
According to claim 1 the method is characterized in that data is stored in at least f+1 different data servers out of the 2f+1 data servers and that metadata of the stored data is stored under a metadata-identifier in a fault-tolerant, metadata-service, wherein the metadata to be stored includes a unique identifier of the stored data, a fingerprint of the stored data and a list of data servers which have stored the data.
In claim 12 a system for byzantine fault tolerant data replication with a plurality of 2f+1 data servers and one or more clients is defined
According to claim 12 the system is characterized by at least f+1 different data servers out of the 2f+1 data servers each being operable to store data and by a fault-tolerant, metadata-service for storing metadata of the stored data being operable to store the metadata under a metadata-identifier, wherein the metadata to be stored includes a unique identifier of the stored data, a fingerprint of the stored data and a list of data servers which have stored the data.
The term "fingerprint" is to be understood in its broadest sense meaning for example a characteristic well defined string for identification including in particular a cryptographic hash.
According to the invention it has been recognized that a metadata service is leveraged together with a set of 2f+1 data servers of which f may occur arbitrary failures. While the data servers store the actual data, the metadata service keeps track of associated metadata including but not limited to for example the location of the data copies, the fingerprints, for example cryptographic hashes of the data for integrity verification, or the like.
According to the invention it has been further recognized that the efficiency is significantly enhanced and the replication costs are reduced in particular when relying on the assumption that data objects are large relative to metadata and thus the cost of data operations represents the dominant costs in terms of bandwidth and latency. As a consequence metadata accesses are significantly cheaper than data accesses provided the metadata services designed correspondingly.
According to the invention it has been recognized that scalability is provided.
According to the invention it has been recognized that correctness, wait-freedom and linearizability of the replicated data respectively the metadata service is enabled.
Further features, advantages and preferred embodiments are described in the following sub-claims.
According to a preferred embodiment the metadata is only stored upon successful storing of the data on the f+1 data servers. This enhances the reliability since only upon successfully storing on all of the f+1 data servers the metadata is stored. The list of f+1 data servers can then be provided in an easy way for the metadata service.
According to a further preferred embodiment for reading data by a client

a) a data server of the list of data servers is queried with the unique identifier of the data, based on metadata provided by the metadata service upon querying with a metadata identifier,
b) the stored data according to the unique identifier is provided and
c) upon matching of the fingerprints of the stored data and of the stored metadata the stored data is provided to the client.

For reading data the metadata service is therefore queried returning the queried value together for example with a tag for the data, a cryptographic digest of the stored data and a list of servers. The querying client then may retrieve the corresponding data by the tag from one of the servers of the list. If the returned value matches the digest, then the read operation returns the correct value and the read operation is completed. Therefore a fast and easy as well as reliable read operation is provided.
According to a further preferred embodiment in case of a mismatch of the fingerprints steps a)-c) are repeated with one or more other data servers of the list of data servers until at least one matching is achieved. This ensures that in case of a Byzantine or crash failure of f data servers of the list, the client can nevertheless obtain a correct value of the stored data from one correct data server. Therefore the reliability of reading of data is further enhanced.
According to a further preferred embodiment repeating of the steps a)-c) is performed sequentially with one data server of the list of data servers only per repetition. This enables that only one server is queried for providing the data and the other servers out of the list are not blocked for other write operations for example. The other servers are only queried in case the current queried server does not provide the correct fingerprint and subsequently the correct data.
According to a further preferred embodiment version information of the data is stored in the metadata service and provided to the client upon querying. If version information is stored then transactions are enabled. For example operations can be supported beyond basic overwrites including for example conditional updates or conditional multi-updates when the metadata service offers corresponding operations.
According to a further preferred embodiment the metadata service is provided as a cloud service. This enables in an easy way to lower the costs for providing a metadata service, i.e. maintenance costs, or the like. Further cloud data is usually provided in a fault tolerant way, therefore reliability is further enhanced.
According to a further preferred embodiment at least two of the data servers provide different file storage services for storing data, preferably at least two of the f+1 data servers. By using different file storage services, for example different databases, security and reliability are enhanced, since for example data can be read even in case that one of the storage services is offline or being attacked or hacked.
According to a further preferred embodiment for storing a plurality of data simultaneously, each data is stored on each of the f+1 data servers. This enables in an easy way to provide advanced operations like multi-update etc. further enhancing the flexibility.
According to a further preferred embodiment the metadata service uses a quorum-based protocol for fault-tolerance. This enables robustness against failures and enables an easy implementation.
According to a further preferred embodiment the metadata service linearizes operations on the metadata. This enables for example read and/or write operations to be executed to an execution order of (low-level) read and write operations on the metadata. Therefore the real-time order of operations on metadata can be respected.
There are several ways how to design and further develop the teaching of the present invention in an advantageous way. To this end it is to be referred to the patent claims subordinate to patent claim 1 on the one hand and to the following explanation of preferred embodiments of the invention by way of example, illustrated by the figure on the other hand. In connection with the explanation of the preferred embodiments of the invention by the aid of the figure, generally preferred embodiments and further developments of the teaching will be explained.
In the drawings

Fig. 1: shows schematically steps of a method according to a first embodiment of the present invention;
Fig. 2: shows schematically steps of a method according to a second embodiment of the present invention; and
Fig. 3: shows schematically steps of a method according to a third embodiment of the present invention.

For the Fig. 1-3 a distributed system comprising a set of data servers S={s₁,s₂, ..., ss) and a set of clients C={c₁, ..., cc) is assumed. Up to a minority of the data servers byzantine failures may incur (|S|≧ 2f+1, where f is the failure threshold). Further any number of clients may fail by crashing. Furthermore a non-faulty metadata service offering support for reading and updating metadata is assumed. Further H indicates a collision-resistant cryptographic hash function and operations on metadata are linearized by the metadata service MDS.
Fig. 1 shows schematically steps of a method according to a first embodiment of the present invention.
In Fig. 1 the case is showing for f=1, i.e. the number of servers for which a byzantine failure may occur and |S| =3, i.e. the number of data servers specifically in Fig. 1a write operation for updating a data object k having a value V is shown. A client C initiates in a first step 1a storing of the value V on the servers s₁ and s₂ together with a corresponding tag. Upon successfully storing the data each of the f+1 servers S, i.e. s₁ and s₂ send in a second step 2 an acknowledgement message to the client C.
The client C then subsequently computes a digest of the stored value V using a cryptographic hash function H(V) and invokes then in a third step 3 the metadata service MDS supplying as metadata md the key k and a value comprising the tag, the digest and the list of data servers, i.e. s₁ and s₂. Further, client C may also supply a version information for the value V, for example in case of a conditional update, cf. figure 3.
Writing a copy of the value into the f+1 servers, in Fig. 1 s₁ and s₂, ensures that a subsequent read can retrieve the value V even if f servers, in Fig. 1 either server s₁ or s₂, fail after being updated by the write operation. The additional f data servers in Fig. 1 the server s₃, are used to prevent the write operation from blocking while trying to update f+1 servers s₁, s₂ in the present of f faulty servers. In the optimal case f respectively f+1 representing lower bounds for any data replication protocol including crash-tolerant ones.
In a fourth step 4 the metadata service MDS provides a corresponding acknowledgement message to the client C indicating that also the metadata was stored successfully in the metadata service MDS.
Fig. 2 shows schematically steps of a method according to a second embodiment of the present invention.
In Fig. 2 a read operation for reading a value for a key k is shown. The client C contacts in a first step 5 the metadata service MDS by querying it with the key k. The metadata service MDS then returns the metadata md and - if transactions are supported - version information ver to the client C in a second step 6. The client C then retrieves the corresponding data by tag from one of the servers s₁, s₂ in the list included in the metadata md in a seventh step 7. The contacted server, i.e. in Fig. 2 the server s₁, then provides the value V' corresponding to the tag and stored under this tag on the server s₁ in a further step 8 back to the client C. The client C then compares if the returned value V' matches the digest, i.e. the client C computes the fingerprint H(V') of the value V' and compares it with the obtained metadata md which includes the stored fingerprint H(V) of the value V. Therefore if the returned value V' matches the digest, i.e. H(V) = H(V') then the read operation returns the value V' as correct value V and the operation is completed.
Otherwise the client C iterates through the list of servers, in Fig. 2 the servers s₁ and s₂, until some of the servers s₁, s₂ reply with a matching value. For example if the value V' does match the digest then the client C queries the second server s₂ out of the list of the f+1 servers s₁, s₂ with the tag and the server s₂ then replies with the value V"; the client C then compares again the digest included in the metadata md and the fingerprint H(V) of the returned value V"; upon matching, i.e. H(V) = H(V") the read operation is completed.
Fig. 3 shows schematically steps of a method according to a third embodiment of the present invention.
In Fig. 3 steps of a multi-update operation is shown. The multi-update operation enables to perform update operations to different keys atomically in an all or nothing fashion.
In Fig. 3 the client C stores a plurality of values V₁, ..., V_n with the corresponding tags tag₁, ..., tag_n on each of the f+1 servers s₁, s₂. Upon receiving an acknowledgement message from the servers s₁, s₂ storing the values V₁, V_n, the client C computes the metadata md; for each of the values: md_i=(tag_i, H(V_i), {s₁, s₂}) and contacts in a further step 11 the metadata service MDS for storing an operation identifier op_i, and the corresponding key k_i, metadata md_i and version data ver_i for all indices 1..., n of the values V₁, ... V_n. Upon successfully storing these information, the metadata service MDS provides in a further step 12 the corresponding acknowledgement message back to the client C.
For example the corresponding operation op_i may include a conditional write or a conditional delete operation or the like. For a conditional write, corresponding version information ver_i are included in the metadata md_i or stored in the metadata service MDS linked with the corresponding metadata md_i for the stored data. Then with regard to Fig. 1 the values are stored in a write operation atomically. If some version ver_i does not match the respective current version of the data, no modification occurs to any of the data object, i.e. of the values V_i and for example an error information may be returned.
In case of a conditional update, a value V is stored under the key k only if the supplied version matches the current version of the data and otherwise no modification occurs and error information may be returned to the client C. A conditional delete operation deletes a key k only if the supplied version matches the current version of the value of the data. Otherwise no modification occurs and an error information may be returned. A delete operation may be implemented by write operation the writing value {null} as value for deletion.
The method and system of byzantine fault-tolerant data replication is correct: To prove the correctness in the following it is described that the operations according to the invention are wait-freedom and linearziable. Wait-freedom means, that operations by correct clients are always complete regardless of the behavior of other clients. According to the invention wait-freedom is insured provided that the metadata service exports wait-free operations and that no more than f data servers are faulty. Further linearizability also known as atomicity means that operations appear to take effect instantaneously at a single instant in time between their invocation and response steps. According to the invention linearizablity is achieved provided that the metadata operations are linearizable wherein linearizability is independent of number of servers f being faulty.
When assuming that low-level operations invoked on the metadata service MDS are wait-free it has to be shown that no operation blocks when accessing one of the data servers. Since at most f data servers are faulty, each update operation eventually receives a reply from f+1 correct servers. Therefore no update operation blocks while awaiting replies from the data servers. When performing an update operation the metadata is only written after storing the corresponding data on the data servers and by the time the reader obtains the metadata the corresponding update operation has stored a matching value in every of the f+1 servers under the corresponding tag. Since among these servers at most f are byzantine faulty, there is at least one correct server of these f+1 servers that stores a matching value V under the corresponding tag and eventually replies to the reader, i.e. the client. After checking that H(V) is matching the digest, the read operation is completed. While in asynchronies runs with failures which is the worst case, a read may need to query f+1 data servers before receiving a matching reply in failure-free and synchronous runs which is the common case, just one data server is queried by a read.
The metadata service MDS as mentioned above linearizes all operations on the metadata. Specifically the read and write operation are linearized according to the execution order of the corresponding low-level write and read operations on the metadata. With the linearizability of the metadata operations the history of low-level operations satisfies a sequential specification of the metadata service MDS and respects the real-time order of metadata operations. Since the sequential specification of the metadata service is equivalent to that of the overall system and the metadata service maintains the real-time precedence relation between high level operations can be concluded that high level operations are linearziable.
In summary the present invention enables a decoupling of the metadata from data in the byzantine context in order to reduce the number of data servers from 3f+1 to 2f+1. The present invention further leverages a metadata service together with a set of 2f+1 data servers to implement strongly consistent data replication. The present invention further enables querying just one data-server in the common case when the metadata service is consulted before querying the data servers. Even further the present invention enables export of data operations provided that an equivalent metadata operation is exposed by the metadata service.
The present invention has inter alia the following advantages: The present invention provides savings and terms of storage server acquisition costs and maintenance costs as well as in terms of storage space. The present invention further improves read performance and read scalability linear in the number of storage servers and enables transactional data access. Even further the present invention enables a substantial reduction of replication costs from 3f+1 servers to just 2f+1 data servers and read scalability as mentioned above: As only a single data server is typically accessed during a read operation, the load incurred by each server is 1/|S|, provided that the data objects to be stored are evenly spread across servers, resulting in a capacity of |S|, i.e. the number of data servers.
Many modifications and other embodiments of the invention set forth herein will come to mind the one skilled in the art to which the invention pertains having the benefit of the teachings presented in the foregoing description and the associated drawings. Therefore, it is to be understood that the invention is not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

Claims

A method for byzantine fault tolerant data replication with a plurality of 2f+1 data servers (S) and one or more clients (C), characterized in that data (V) is stored (1) in at least f+1 different data servers (S) out of the 2f+1 data servers (S) and that
metadata (md) of the stored data is stored under a metadata-identifier (k) in a fault-tolerant, metadata-service (MDS), wherein the metadata (md) to be stored includes a unique identifier (tag) of the stored data (V), a fingerprint (H(V)) of the stored data (V) and a list of data servers ({s₁, s₂, ...}) which have stored the data (V).
The method according to claim 1, characterized in that the metadata (md) is only stored upon successful storing of the data (V) on the f+1 data servers (S).
The method according to one of the claims 1-2, characterized in that for reading data (V) by a client (C)
a) a data server of the list of data servers is queried with the unique identifier (tag) of the data (V), based on metadata provided by the metadata service (MDS) upon querying with a metadata identifier (k),

b) the stored data (V) according to the unique identifier (tag) is provided and

c) upon matching of the fingerprints (H(V), H(V')) of the stored data (V') and of the stored metadata the stored data (V) is provided to the client (C).
The method according to claim 3, characterized in that in case of a mismatch of the fingerprints (H(V), H(V')) steps a)-c) are repeated with one or more other data servers ({s₂, ...}) of the list of data servers ({s₁, s₂ ...}) until at least one matching is achieved.
The method according to claim 4, characterized in that repeating of the steps a)-c) is performed sequentially with one data server (S) of the list of data servers ({s₁, s₂ ...}) only per repetition.
The method according to one of the claims 1-5, characterized in that version information (ver) of the data (V) is stored in the metadata (MDS) service and provided to the client (C) upon quering.
The method according to one of the claims 1-6, characterized in that the metadata-service (MDS) is provided as a cloud service.
The method according to one of the claims 1-7, characterized in that at least two of the data servers (S) provide different file storage services for storing data (V).
The method according to one of the claims 1-8, characterized in that for storing a plurality of data (V) simultaneously, each data (V) is stored on each of the f+1 data servers ({s₁, s₂...}).
The method according to one of the claims 1-9, characterized in that the metadata service (MDS) uses a quorum-based protocol for fault-tolerance.
The method according to one of the claims 1-10, characterized in that the metadata service (MDS) linearizes operations on the metadata (md).
A system for byzantine fault tolerant data replication with a plurality of 2f+1 data servers (S) and one or more clients (C),
characterized by
at least f+1 different data servers ({s₁,...}) out of the 2f+1 data servers (S) each being operable to store data (V) and by
a fault-tolerant metadata-service (MDS) for storing metadata (md) of the stored data (V) being operable to store the metadata (md) under a metadata-identifier (k), wherein the metadata (md) to be stored includes a unique identifier (tag) of the stored data (V), a fingerprint (H(V)) of the stored data (V) and a list of data servers ({s₁, s₂...}) which have stored the data (V).