CN110069431B - Elastic Key-Value Key Value pair data storage method based on RDMA and HTM - Google Patents
Elastic Key-Value Key Value pair data storage method based on RDMA and HTM Download PDFInfo
- Publication number
- CN110069431B CN110069431B CN201810070442.1A CN201810070442A CN110069431B CN 110069431 B CN110069431 B CN 110069431B CN 201810070442 A CN201810070442 A CN 201810070442A CN 110069431 B CN110069431 B CN 110069431B
- Authority
- CN
- China
- Prior art keywords
- data
- key
- hash
- rdma
- value pair
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000000034 method Methods 0.000 title claims abstract description 58
- 238000013500 data storage Methods 0.000 title claims abstract description 37
- 238000013523 data management Methods 0.000 claims abstract description 77
- 238000004891 communication Methods 0.000 claims abstract description 41
- 238000005516 engineering process Methods 0.000 claims abstract description 32
- 238000003780 insertion Methods 0.000 claims abstract description 24
- 230000037431 insertion Effects 0.000 claims abstract description 24
- 230000008569 process Effects 0.000 claims abstract description 15
- 230000005540 biological transmission Effects 0.000 claims abstract description 13
- 230000004044 response Effects 0.000 claims abstract description 6
- 238000012795 verification Methods 0.000 claims abstract description 4
- 230000006870 function Effects 0.000 claims description 58
- 241000544061 Cuculus canorus Species 0.000 claims description 44
- 238000007726 management method Methods 0.000 claims description 16
- 238000013507 mapping Methods 0.000 claims description 14
- 238000013479 data entry Methods 0.000 claims description 11
- 238000010586 diagram Methods 0.000 claims description 11
- 238000001152 differential interference contrast microscopy Methods 0.000 claims description 9
- 230000003993 interaction Effects 0.000 claims description 8
- 230000015556 catabolic process Effects 0.000 claims description 5
- 238000006731 degradation reaction Methods 0.000 claims description 5
- 238000007405 data analysis Methods 0.000 claims description 4
- 238000005295 random walk Methods 0.000 claims description 4
- 241000714203 Rabbit hemorrhagic disease virus Species 0.000 claims description 3
- 230000000977 initiatory effect Effects 0.000 claims description 3
- 238000012544 monitoring process Methods 0.000 claims description 3
- 230000003068 static effect Effects 0.000 claims description 3
- 230000008901 benefit Effects 0.000 description 6
- 238000012546 transfer Methods 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000011156 evaluation Methods 0.000 description 3
- 238000002474 experimental method Methods 0.000 description 3
- 244000144980 herd Species 0.000 description 3
- 230000006872 improvement Effects 0.000 description 3
- 230000007547 defect Effects 0.000 description 2
- 239000010445 mica Substances 0.000 description 2
- 229910052618 mica group Inorganic materials 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000003321 amplification Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 238000013467 fragmentation Methods 0.000 description 1
- 238000006062 fragmentation reaction Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000009897 systematic effect Effects 0.000 description 1
- 230000005641 tunneling Effects 0.000 description 1
- 238000009827 uniform distribution Methods 0.000 description 1
- 239000011800 void material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
- G06F13/14—Handling requests for interconnection or transfer
- G06F13/20—Handling requests for interconnection or transfer for access to input/output bus
- G06F13/28—Handling requests for interconnection or transfer for access to input/output bus using burst mode transfer, e.g. direct memory access DMA, cycle steal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2255—Hash tables
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides an elastic Key-Value Key Value pair data storage method based on RDMA and HTM, which is designed to follow a server-client architecture mode and comprises the following steps: at a server side, an improved G-Cuckoo hash data management mode is provided by combining a hash map, so that the problem of kisk-out infinite loop between hash tables caused by searching for an idle position in a data insertion process is avoided; analyzing key value to storage performance bottleneck problem caused by round response needed by traditional network transmission information between a client and a server, designing an RDMA network communication engine by using Infiniband remote direct memory access RDMA technology, receiving a data access request and sending back a data request result; the method comprises the steps that a hardware transactional memory HTM technology is utilized to realize two-section lock protocol lock operation and guarantee the atomic characteristics of data operation; and the key value is used for ensuring the data consistency by using the data self-verification check code. The invention can greatly improve the basic operation speed of key value pair data.
Description
Technical Field
The invention belongs to the field of computer systems and data storage systems, and particularly relates to an elastic Key-Value Key Value pair data storage method based on RDMA and HTM, which can provide rapid and large-scale data service.
Background
At this stage, data-intensive computing is increasingly in demand, and data provided to data-intensive computing is typically stored in a storage system. In addition to storage systems such as relational databases, more and more key-value pair storage systems are continuously present, and rapid access to data can also be met. For key-value pair storage systems, how to obtain low latency and high throughput of data access has become an issue worthy of further research and discussion.
In a distributed cluster environment, it is often necessary to conduct remote data access to key-value pair storage systems over traditional networks. In this case, the server responsible for data management and the client accessing data need to be involved together, and there is a problem of data access delay. For example, a classic Ethernet network has a performance bottleneck problem, a network round defect causes a certain disadvantage in fast data transmission and access, and key value is not well utilized for high-performance data access of a storage system. Infiniband Remote Direct Memory Access (RDMA) technology has emerged, which gives the possibility of optimizing data in storage systems using traditional network Remote Access keys. The more common mode is to adopt Infiniband RDMA technology to reduce the problem of long delay of data access request.
Well known intel corporation and mellonox corporation are the major InfiniBand adapter manufacturers. Mellanox40Gbps ConnectX-3 RNIC costs far less communication than 10Gbps Ethernet adapter. In effect, the Infiniband RDMA READS technique conveniently accesses remote memory, as does local memory, bypassing the CPU initialization control transfer. For example, the Pilaf key-value pair storage system performs GET operations on key-value pair data directly using RDMA Reads. More, the reasons for the throughput and speed increase of the FaRM and HERD key-value pair storage systems mainly come from the high performance network stack and the CPU core bypass technology. The RDMA READs technology is directly utilized to perform data read-write operations, wherein it is very difficult to perform complex pointer operations, for example, it is very complicated to operate a dereferencing pointer and a following pointer of a remote memory.
Therefore, when key-value pair storage management is designed, the memory pointer operation is tried to be carried out by not directly adopting RDMA READs, but key-value pair data is managed for a hash data management mode, and a high-performance RDMA message communication library is designed to carry out data transmission communication, so that the operation of a complex pointer in a remote memory can be effectively avoided. An effective communication library engine is constructed by using a high-performance RDMA communication protocol to carry out data communication, and data in a hash data management mode are directly operated, so that the same effect of directly accessing the memory to acquire the data by the RDMA READs is achieved.
When data is operated in the key value pair storage system, the atomicity of data operation needs to be guaranteed, and data concurrent request conflict is avoided. While key-value pair data storage system data operations are simpler, in the hash data management mode, the underlying data structure operations are more complex. Any concurrent conflicting requests for data on a remote machine will result in an interrupted exit of the overall job. Currently, Hardware Transactional Memory Hardware Transactional Memory (HTM) appears mainly in the form of Restricted Transactional Memory (RTM) and Hardware lock elision Hardware LockElision (HLE). Researchers and engineers have utilized HTM techniques to provide request contention management to effectively avoid deadlock.
In fact, it is important to design a key-value pair storage system based on a hash data management mode in combination with HTM and RDMA technologies. The "RDMA + HTM" key-value-based storage management needs to contain several major features: (i) high efficiency and high throughput characteristics of data access; (ii) RDMA-based communication with lower CPU usage; (iii) efficient hash data management mode design; (iii) no data request collision, etc.
The Key-Value Key Value pair database is a simplification of the traditional relational database, so that the performance advantage is the maximum, and the coupling degree between data is the minimum. In general, there are mainly 3 main implementations: (1) implemented by a B + tree. The B + tree reduces the height of the whole tree as much as possible by increasing the number of child nodes of each computing node, and reduces the read-write times of the external memory device. The method has the advantages of high query efficiency; the disadvantage is that a large number of random writes easily cause node fragmentation, resulting in performance degradation. (2) The method is realized based on the Hash table, such as Redis, and the Hash table has the advantages of high query efficiency and incapability of supporting range query, and the read-write efficiency is reduced when a large number of Hash values collide. The ShenZhaoyan et al propose a Key-Value Key Value pair storage system based on a Hash table, and solve the problem of redundant mapping existing in an address mapping table of the Hash table and an FTL. (3) The implementation based on a Log-Structured-Merge tree (LSM tree), such as levelDB, has the advantages of high write performance, and the disadvantages of poor read performance and easy occurrence of severe write amplification.
The prior art provides a series of elastic Key-Value Key Value pair data storage management principles and implementation and the like. Key-value pair storage systems have gained explosive growth. For example, Redis a RAM-based Key-Value pair storage system, which is an open-source, log-type Key-Value pair database written in ANSI C language, supporting network, and can be based on memory and also be persistent, and provides API functions in multiple languages. However, it does not directly utilize the RDMA protocol to carry out data transfers. Because of the high bandwidth and low latency characteristics of RDMA technology and the unilateral zero load CPU consumption capability, efficient data transfer using RDMA protocol has been a great concern, and the advantage and disadvantage of systematic analysis of RDMA primitives by the HERD key-value pair storage system. HERD does not guarantee reliable data transfer. In addition, the RAMCloud memory system is a memory system that completely uses DRAM, and all data is stored in memory. In addition, the Pilaf key-value pair storage system utilizes RDMA READs to improve the performance of data GET operations. Key-value storage system MICA key-value storage system deployed in a multi-core system may operate 65.6 to 76.9 million key-value-pair operations per second. The MICA directly maps the client request to a specific CPU (central processing unit) core NIC (network interface card) layer of the server, and uses a lightweight network stack technology to bypass a core to carry out data operation. Wu et al propose a key-value-pair data cache zExpander with high performance and lower accuracy, and they also propose a novel key-value-pair cache storage system NVMcached based on NVM.
In general, on one hand, the elastic Key-Value Key Value pair data storage method based on RDMA and HTM lacks powerful software and program support for hardware transaction memory management such as a high-performance data transmission network protocol and guarantee data operation atomic characteristics; on the other hand, the efficiency of the data management mode is not high, the data which has been accessed recently is not designed to be stored in the data cache, the data consistency mechanism is poor, and the like, and the parallelism degree of the data operation is also to be further improved. Therefore, the research on the elastic Key-Value Key Value pair data storage management based on RDMA and HTM can greatly improve the capacity and the parallelism of intensive data processing and effectively relieve the data transmission delay and the like.
Disclosure of Invention
Aiming at the defects in the prior art, the invention aims to provide an elastic Key-Value Key Value pair data storage method based on RDMA and HTM. From the perspective of communication speed and I/O throughput, attention is paid to the improvement of high throughput and low delay indicators. The development of the existing technologies such as a high-performance communication network, a high-memory technology, a hardware transactional memory and the like also provides powerful technical support for the elastic Key-Value Key Value to manage data storage. The method fully utilizes the advanced technology, improves the data read-write performance of the key value pair storage system, and lays a foundation for large-scale data intensive calculation performance improvement in a cluster system.
The invention is realized according to the following technical scheme:
an elastic Key-Value Key Value pair data storage method based on RDMA and HTM is characterized by comprising the following steps:
step 1: following a server-client architecture mode, constructing a large-scale data intensive computing elastic Key-Value Key Value pair data storage management system by utilizing a hardware transaction memory HTM and remote direct memory access RDMA;
step 2: at a server side, a key value pair data storage management system constructs a Hash diagram by using a bucket-point bucket-vertex mapping mode, and provides an improved G-Cuckoo Hash data management mode, so that the problem that data items are kicked out by infinite loop kick-out between Hash tables due to the fact that an idle position needs to be searched in the data insertion process is solved;
and step 3: acquiring a data request of a client through a user-defined data access interface;
and 4, step 4: between a client and a server, analyzing a key value to storage performance bottleneck problem caused by round response to and fro of a traditional network, designing an RDMA network communication engine by using an Infiniband remote direct memory access RDMA SEND/RECVverbs technology, and receiving a data access request and returning a data request result to the client;
and 5: the method comprises the steps that a hardware transactional memory HTM is utilized to realize locking operation following a locking and releasing protocol, a data operation HTM area is formed, and the atomic characteristics of basic data operation are guaranteed;
step 6: self-verifying Checksum check codes for key-value pair data are used to ensure data consistency.
In the above technical solution, in the step 1, an elastic Key-Value Key Value pair data storage management system is constructed, specifically as follows:
the key-value pair storage system is integrally deployed in an InfiniBand network cluster environment, plays roles of data access and data I/O request service provision for upper-layer data analysis type application, a client locates a key-value pair data storage position according to an improved Cuckoo Hash data management mode G-Cuckoo, the key-value pair storage system manages Hash table key-value pair data, and message communication is carried out by utilizing a two-sided RDMA SEND/RECV verbs technology;
all data are stored in a DRAM (dynamic random access memory) of a server side, and effective data service is provided for a client side through an RDMA (remote direct memory access) network; using the computing power of the server, adopting HTM-aware, RDMA-friendly hash tables, each Key-Value Key-Value pair having a corresponding storage location and backup location in both hash tables, connecting the two locations with an edge, the start of the edge corresponding to the actual storage location of the Key-Value pair and the end of the edge corresponding to the actual backup storage location of the Key-Value pair, the hash data management mode adopting two static hash functions h1(x) and h2 (x); when a data request is sent to the server, the related control message needs to be sent to the server in advance; and after receiving the data request, the server side provides data read-write service, encapsulates the data into payload and returns the payload to the client side.
In the above technical solution, in the step 2, the G-Cuckoo hash data management mode is specifically as follows:
selecting a classic Cuckoo hash data management mode, improving correspondingly, and using the classic Cuckoo hash data management mode as a key value pair storage system data management strategy, wherein the classic Cuckoo hash data management mode comprises the following steps: if there are d (d >0) hash tables, let K be the key set of key-value pairs, and hash table T1 and hash table T2 possess n data items respectively, given two independent functions h1, h2: K → { 0.,. n-1}, the key K ∈ K can be inserted into position h1(K) of hash table T1 or position h2(K) of hash table T2, the Cuckoo hash data management mode possesses basic data operation mode; when the key-value pair PUT operation is carried out, the server side needs to calculate candidate positions for inserting the key values, the key has one or two positions in the hash table, once the candidate position needing to be inserted is occupied by other key-value pair data, the kick-out operation is executed, namely the key values need to be kicked from one hash table to the idle position of the other hash table;
when a key-value pair data entry is inserted, the traditional Cuckoo Hash data management mode adopts a random walk method to search for an idle bucket position, which causes the problems of infinite circulation and performance degradation of the inserted data; in a classic Cuckoo hash data management mode, in order to find an idle data entry insertion position, a constant kick-out operation problem can be interrupted, and alternative re-hash operation is carried out, so that a time delay problem can occur;
a classic Cuckoo hash data management mode is improved, an infinite loop prediction function is fused in a kick-out operation, when d is 2, the Cuckoo hash data management mode has two tables, each actual position inserted into the hash data management mode has a corresponding backup position, storage positions after the kick-out operation are provided, multiple edges exist among the tables, each position of the hash table is mapped into a Hash diagram in a bucket-vertex mapping mode, an improved Cuckoo hash data management mode G-Cuckoo is provided, and the problem that the infinite kick-out is kicked back and forth is avoided in prediction.
In the above technical solution, in step 3, the data access interface is defined to obtain the data request of the client, specifically as follows:
interfaces used by a user are provided, and the interfaces comprise a numerical setting function rhkv _ set, a numerical obtaining function rhkv _ get and a numerical updating function rhkv _ update.
In the above technical solution, in the step 4, an Infiniband RDMA network communication engine is designed, specifically as follows:
the data message communication operation Infiniband RDMA network engine adopts two-position tunnel semantics, a user space program accesses RDMA NICs by using an RDMA verbs function, wherein the RDMA NICs provide reliable transmission based on a hardware lost packet retransmission mode, the RDMANICs comprise the two-position tunnel semantics and one-position memory semantics bypassing a target CPU, and a message queue is maintained in the RDMA NICs and exists in two forms: a sending queue send queue and a receiving queue receive queue, which are called as aque pair; when the RHVV receives a data operation request, data interaction is carried out between the RHDV and the improved G-Cuckoo hash data management mode through the RDMA network engine, and finally the corresponding key value pair data position is found for data insertion or other data operation.
In the above technical solution, in step 5, a data operation HTM area is formed following a lock operation of a lock adding and release protocol, which specifically includes:
during the Start phase, HTM transactions lock and prefetch remote data records, initiating HTM transactions by running XACQUIRE; in the local transaction LocalTX phase, the HTM transaction provides transaction-level reads and writes to all local records; in the Commit phase, the data record is updated to release the lock by committing the HTM transaction of the data operation using XRELESE.
In the above technical solution, in the step 6, the self-verification Checksum process of the key-value pair data is as follows:
(1) firstly, calculating an actual hash value of a key value;
(2) inquiring a value to be obtained from the hash table;
(3) calculating a Checksum value according to the content of the key-value pair;
(4) the comparison is made as to whether the Checksum value of the packet stored in the hash table is the same.
In the above technical solution, the request for data needs to perform message interaction with the hash data management mode, and specifically includes:
(1) the Server side Server registers and calls back an operation handle, binds and monitors newly arrived client data operation, and after the data request operation is executed, the connection is interrupted and the unbinding operation of the client connection monitoring is carried out;
(2) client sets host information, RDMA communication component gives constructor and destructor respectively used for initializing and releasing Infiniband connection, function connection is defined for adding connection array, in the Infiniband RDMA communication component, the Infiniband connection array is operated by using functions of init, insert and free, content sent by the Client is stored in a charvector data structure and packaged into communication message.
Compared with the prior art, the invention has the following beneficial effects:
the invention utilizes the high-performance characteristic of data transmission of RDMA verbs message, and the client transmits the request to the G-Cuckoo hash data management mode of the server. Designing an improved Hash data management mode-G-Cuckoo, and constructing a Cuckoo graph in a bucket-point mapping bucket-vertex mapping mode; in the key value pair insertion process, infinite kick-out circulation of data items among hash tables is effectively avoided through a prediction mode, the problem of infinite circulation of the data items during insertion is relieved, and each data request is guaranteed to be completed under the conditions of low delay and low cost as far as possible. And after receiving the data request, the server side hash data management mode completes the data operation and returns a response result to the client side. And an RDMA communication model is built, an RDMA messaging verbs technology is used for building an RDMA network engine, and data access and result transmission are accelerated. And ensuring the atomic characteristics of data operation by combining an HTM technology and a two-segment lock protocol (2PL) technology. Substantial latency reduction can be achieved with basic data operations, resulting in significantly higher throughput. Following the key-value-pair data management method provided by the invention, the developed key-value-pair storage prototype system can be continuously perfected and matured and become a reliable storage system or become a storage component of a data-intensive computing system independently.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:
FIG. 1 is a flow chart of a method for storing data of an elastic Key-Value pair based on RDMA and HTM in the present invention.
FIG. 2 is a diagram of the overall architecture of an RDMA and HTM-based elastic Key-Value pair data storage prototype system according to the present invention.
FIG. 3 is a schematic diagram of a maximum directed forest and a non-maximum directed forest in the present invention.
FIG. 4 is a diagram of the RDMA communication model of the present invention.
Detailed Description
The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that it would be obvious to those skilled in the art that various changes and modifications can be made without departing from the spirit of the invention. All falling within the scope of the present invention.
Referring to fig. 1-2, a method for storing elastic Key-Value Key Value pair data based on RDMA and HTM according to the present invention is characterized by comprising the following steps:
step 1: following a server-client architecture mode, constructing a large-scale data intensive computing elastic Key-Value Key Value pair data storage management system by utilizing a hardware transaction memory HTM and remote direct memory access RDMA;
step 2: at a server side, a key value pair data storage management system uses a hash map constructed in a bucket-point bucket-vertex mapping mode, an improved G-Cuckoo hash data management mode is given, and the problem that data items are kicked out in infinite loop kick-out between hash tables due to the fact that an idle position needs to be searched in the data insertion process is solved;
and step 3: acquiring a data request of a client through a user-defined data access interface;
and 4, step 4: between a client and a server, analyzing the key value-to-storage performance bottleneck problem caused by round response of the traditional network, designing an RDMA network communication engine by using an Infiniband remote direct memory access RDMA SEND/RECV verbs technology, and receiving a data access request and returning a data request result to the client;
and 5: the method comprises the steps that a hardware transactional memory HTM is utilized to realize locking operation following a locking and releasing protocol, a data operation HTM area is formed, and the atomic characteristics of basic data operation are guaranteed;
step 6: self-verifying Checksum check codes for key-value pair data are used to ensure data consistency.
In step 1, constructing an elastic Key-Value Key Value pair data storage management system, which specifically comprises the following steps:
the key-value pair storage system is integrally deployed in an InfiniBand network cluster environment, plays roles of data access and data I/O request service provision for upper-layer data analysis type application, a client locates a key-value pair data storage position according to an improved Cuckoo Hash data management mode G-Cuckoo, the key-value pair storage system manages Hash table key-value pair data, and message communication is carried out by utilizing a two-sided RDMA SEND/RECV verbs technology;
all data are stored in a DRAM (dynamic random access memory) of a server side, and effective data service is provided for a client side through an RDMA (remote direct memory access) network; using the computing power of the server, adopting HTM-aware, RDMA-friendly hash tables, each Key-Value Key-Value pair having a corresponding storage location and backup location in both hash tables, connecting the two locations with an edge, the start of the edge corresponding to the actual storage location of the Key-Value pair and the end of the edge corresponding to the actual backup storage location of the Key-Value pair, the hash data management mode adopting two static hash functions h1(x) and h2 (x); when a data request is sent to the server, the related control message needs to be sent to the server in advance; and after receiving the data request, the server side provides data read-write service, encapsulates the data into payload and returns the payload to the client side.
In step 2, the G-Cuckoo hash data management mode is specifically as follows:
and selecting a classic Cuckoo hash data management mode, and performing corresponding improvement to serve as a key value pair storage system data management strategy. The reason is that the Cuckoo hash data management mode is an open address data management mode, and each non-empty cell in the hash table contains one key or key value pair. A hash function may be used to determine the actual location of each key, and the free locations are found by scanning the hash table through the locations. The lookup process requires checking two locations of the hash table, which takes some time. Compared with other hash table algorithms, the method is more efficient in carrying out data searching operation. For the Cuckoo hash data management model, the data deletion operation also appears to be simpler.
Classic Cuckoo hash data management schema: if there are d (d >0) hash tables, let K be the key set of key-value pairs, and hash table T1 and hash table T2 possess n data items respectively, given two independent functions h1, h2: K → { 0.,. n-1}, the key K ∈ K can be inserted into position h1(K) of hash table T1 or position h2(K) of hash table T2, the Cuckoo hash data management mode possesses basic data operation mode; when the key-value pair PUT operation is carried out, the server side needs to calculate candidate positions for inserting the key values, the key has one or two positions in the hash table, once the candidate position needing to be inserted is occupied by other key-value pair data, the kick-out operation is executed, namely the key values need to be kicked from one hash table to the idle position of the other hash table;
when a key-value pair data entry is inserted, the traditional Cuckoo Hash data management mode adopts a random walk method to search for an idle bucket position, which causes the problems of infinite circulation and performance degradation of the inserted data; in a classic Cuckoo hash data management mode, in order to find an idle data entry insertion position, a constant kick-out operation problem can be interrupted, and alternative re-hash operation is carried out, so that a time delay problem can occur; in fact, interrupting the kick-out operation and moving to an alternative re-hashing operation does not solve the infinite loop problem well.
A classic Cuckoo hash data management mode is improved, an infinite loop prediction function is integrated in the kick-out operation, when d is 2, the Cuckoo hash data management mode has two tables, each actual position inserted into the hash data management mode has a corresponding backup position, storage positions after the kick-out operation are provided, multiple edges exist among the tables, a bucket-vertex mapping mode is adopted, all positions of the hash tables are mapped into a Hash diagram, an improved Cuckoo hash data management mode G-Cuckoo is provided, and the problem that infinite kick-out kick back and forth is avoided in prediction.
In step 3, defining a data access interface to obtain a data request of the client, specifically as follows:
providing a data base operation interface that can be used by a user includes: a numerical setting function rhkv _ set, a numerical acquisition function rhkv _ get, and a numerical update function rhkv _ update.
In step 4, designing an Infiniband RDMA network communication engine, which is as follows:
the data message communication operation Infiniband RDMA network engine adopts two-position tunnel semantics, a user space program accesses RDMA NICs by using an RDMA verbs function, wherein the RDMA NICs provide reliable transmission based on a hardware lost packet retransmission mode, the RDMANICs comprise the two-position tunnel semantics and one-position memory semantics bypassing a target CPU, and a message queue is maintained in the RDMA NICs and exists in two forms: a sending queue send queue and a receiving queue receive queue, which are called as aque pair; when the RHVV receives a data operation request, data interaction is carried out between the RHDV and the improved G-Cuckoo hash data management mode through the RDMA network engine, and finally the corresponding key value pair data position is found for data insertion or other data operation. While InfiniBand RoCE (RDMA over converted Ethernet) can provide low latency data manipulation functionality.
The request of data needs to perform message interaction with a hash data management mode, and comprises basic Infiniband Server and Infiniband Client. When the data reading operation is executed, the Infiniband Client searches a key from the number sequence of the hash table, and the Client obtains the key entry corresponding to the hash table. The specific data interaction operation of the server and the client specifically comprises the following steps:
(1) the Server side Server registers and calls back an operation handle, binds and monitors a newly arrived client and an operation, and after the data request operation is executed, the connection is interrupted and the unbinding operation of the client connection monitoring is carried out;
(2) client sets host information, RDMA communication component gives constructor and destructor respectively used for initializing and releasing Infiniband connection, function connection is defined for adding connection array, in the Infiniband RDMA communication component, the Infiniband connection array is operated by using functions of init, insert and free, content sent by the Client is stored in a charvector data structure and packaged into communication message.
In step 5, a data operation HTM area is formed following the lock operation of the lock and release protocol, which specifically includes:
during the Start phase, HTM transactions lock and prefetch remote data records, initiating HTM transactions by running XACQUIRE; in the local transaction LocalTX phase, the HTM transaction provides transaction-level reads and writes to all local records; in the Commit phase, the data record is updated to release the lock by committing the HTM transaction of the data operation using XRELESE.
The atomic characteristics of basic data operation are guaranteed by adopting an HTM technology, which specifically comprises the following steps:
the data intensive computing application program can access the hash table of the server side, and the problems of lock competition and request conflict exist when the data are requested concurrently. In order to ensure atomicity of data operations, each operation on the hash table needs to use a lock to ensure access to the atomic property of the Key-Value Key Value pair data record without being interfered by other data requests. Once the data is accessed by the Inifiniband client, it is likely to be modified to another new value, requiring a lock to be added on this data operation before returning the result to the client. This can lead to data inconsistency problems for concurrent data requests if the atomic nature of the data operation cannot be guaranteed.
Data operations are mainly based on the complex operations of the Cuckoo graph, but are more complex for the operations of the bottom layer. For a data operation, it needs to be completed within a certain time interval, at which point additional data operation requests may have come. This results in the new operation request also operating on this data key-value pair entry. By using the strong atomicity guarantee sensed by the HTM, the corresponding lock is obtained before the HTM enters the HTM protection area, and each basic data request cannot be interfered by other data requests in the execution process of data operation, so that the problem of data inconsistency is prevented.
And ensuring the strong atomic property of data operation of the computing node, and detecting access request conflict. HTM is an efficient way to guarantee the atomic nature of data operations, including HLE (Hardware Lock Elision) technology. Intel's RTM (managed transaction Memory) is a concrete representation of HLE technology, and RTM provides a series of interfaces such as XBegIN, XEND, and XABORT, which represent start, end, and interrupt transactions, respectively. HLE technology is easily integrated into existing x86 software, which is well compatible with x86 microprocessors. These microprocessors have the transaction Synchronization Extension TSX Interactive Synchronization Extension (TSX) function of Intel Haswell, and execute load and store instructions in an automated manner. The method uses an Intel RTM technology to guarantee the atomic characteristics of data operation. HLE has two basic instructions, XACQUIRE and XRELEASE. Wherein the XACQUIRE instruction is used primarily to obtain a lock, representing the beginning of a lock region. Correspondingly, the XRELESE instruction is used primarily to release the lock, representing the end of the lock area. To guarantee the atomicity of data operations, LOCK and unlock operations are defined, containing a function ACQUIRE _ LOCK () and a function RELEASE _ LOCK (). These functions contain the basic instructions XACQUIRE and XRELEASE.
In step 6, the consistency and correctness of the data before and after the data transmission between the client and the server are ensured by using the data consistency principle. The method comprises the following specific steps:
when a data operation is performed on the hash table, it may cause the data operation stored by the key-value pair to be in an inconsistent state, and the scenario causing the inconsistent state includes: (1) initializing a hash table; (2) triggering a repeated hash operation; (3) inserting or updating key-value pair entries; (4) InfiniBand RDMA communication; and (5) deleting an existing key, and so on. The change of the value of the key value to the client can be notified through the callback function. Specifically, within a given time, if the Infiniband server finds that a value has changed to a new value, it returns a success signal to the Infiniband client; otherwise, it returns an error signal to the Infiniband client. For data recovery, the data storage policy may use an apend operation to guarantee a multi-version numerical property of the key, one of which is marked as active.
In order to ensure the accuracy of message communication between the message server and the client, the specific key-value pair data self-verification Checksum process is as follows:
(1) firstly, calculating an actual hash value of a key value;
(2) inquiring a value to be obtained from the hash table;
(3) calculating a Checksum value according to the content of the key-value pair;
(4) the comparison is made as to whether the Checksum value of the packet stored in the hash table is the same.
The detailed design of the G-Cuckoo hash data management mode is combined with the Cuckoo graph to construct. The method comprises the following specific steps:
define (Cuckoo graph) the location between two hash tables constitutes a directed graph G ═ V, E, where V and E are the set of vertices and edges. Each vertex in the graph has a unique identification number, and the vertex set can be divided into a plurality of subgraph sets.
A subgraph of definition (Cuckoo subgraph). G can be represented as Pi＝(Vi,Ei) Wherein, in the step (A),
Ei＝{(vi,vj)∈E|vi∈Vi}. It is to be noted that EiIncluding all vertex sets ViIt is possible to cross between subgraphs. All subgraphs are distributed between two hash tables; at the same time, each vertex should belong to a subgraph, and the invention represents the map as φ p, which records to which subgraph the set of vertices belong.
FIG. 3 is a schematic diagram of a maximum directed forest and a non-maximum directed forest in the present invention.
Define (maximum directed forest and non-maximum directed forest) a maximum directed forest is a directed graph, each vertex has a unique degree of departure corresponding to it. The non-maximum directed forest may be converted to a maximum directed forest by connecting the vertex with out-degree 0 in the graph with any other vertex in the graph with an edge.
Bucket-to-node mapping, elicitation from several of the above definitions, a Cuckoo graph between two hash tables is constructed using Bucket-to-node mapping. And representing the Cuckoo as a directed forest, wherein each vertex in the forest corresponds to a bucket (position) in the hash table, and each edge is positioned between the positions of each key value pair. This graph contains several directed edges, the kick-out operation is performed in the graph, finding free locations and inserting key-value pairs.
The largest subgraph cannot add a new edge because the infinite loop problem easily occurs. The new edge can be added to a non-maximum subgraph, which does not have the infinite loop problem. The vertex with zero out represents the position (bucket) of the directed forest void. It is very easy to update or delete key-value pairs in the hash table. However, when inserting a key value pair, the two candidate locations are typically occupied by two data items, no empty location exists in the sub-graph, and key value pair data cannot be inserted directly.
Successful key-value pair insertion operations typically depend on whether a free location is found for data storage. The vertex corresponding to the candidate position of a key-value pair should belong to a sub-graph, and the sub-graph needs to contain a vertex with zero out degree, which corresponds to a free position in the hash table. In order to improve the I/O throughput of the server side, for the insertion operation of the logarithm value of the key value, an empty position is searched, and infinite loop kick-out operation is predicted.
Searching a free position and inserting key value pair data, wherein the specific process is as follows:
when the key-value pair storage is inserted into the G-Cuckoo hash data management mode, a free position needs to be found. At least one idle candidate position is positioned in the non-maximum subgraph, and in the directed forest, a vertex with zero out degree represents an idle bucket position which is positioned on the maximum directed path of the non-maximum subgraph. When a data entry is inserted into a non-maximal subgraph, it needs to be stored in an idle bucket position, which corresponds to the last vertex of the directional Cuckoo path.
Based on the already constructed Cuckoo graph, the insertion of vertex data items will result in an increasing change in the number of vertices, which can be expressed as count (v) + i, (i ═ 2,1,0), as follows:
case A count (v) + 2. Assuming that neither of the two candidate positions for a key-value pair corresponds to the constructed Cuckoo graph (directed forest), the insertion of the key-value pair data will form a new subgraph, which is a non-maximal directed subgraph. It is easy to know that a new data entry will be inserted successfully, the number of vertices of the subgraph is increased by 2, and the number of edges is increased by 1.
Case B count (v) + 1. One candidate location corresponds to an existing vertex in the Cuckoo subgraph, and the other candidate location does not correspond to a vertex in the Cuckoo subgraph. In this case, the key-value pair will be successfully inserted, increasing the number of vertices by 1 and the number of edges by 1 as well. In the directed forest subgraph, this new edge connects the newly added vertex with the already existing vertex.
Case C count (v) + 0. When inserting a key pair, the number of Cuckoo graph vertices is not increased. After inserting key-value pairs, the situation is more complex, and three different situations can occur, as follows:
(1) when two candidate positions are located in the same non-maximum directed subgraph or two different non-maximum directed subgraphs, the bucket in the hash table is likely to be a candidate position for inserting a key value pair, and the kick-out operation will be carried out. The original non-maximum subgraph may be converted to the maximum subgraph.
(2) When one of the candidate positions is located in the non-maximum directed subgraph and the other one is located in the maximum directed subgraph, the data entry is inserted into the non-maximum subgraph. Two subgraphs may be merged. If inserted into the largest subgraph, this would lead to an infinite kick-out operation problem.
(3) When all candidate positions are located in the same maximum subgraph or separately located in two maximum directed subgraphs, the insertion result will fail, and an infinite kick-out loop problem will occur.
The method has a process for predicting whether an infinite kick-out loop exists, and comprises the following specific steps:
in the G-Cuckoo data hash management mode, it is known in advance which Case result of claim 9 will be produced after data manipulation. When the data is inserted, the number of vertices encountered is incremented to result in count (v) +0, and the insertion of data will not cause the problem of infinite loops. The increment of the edge number of the subgraph is 1, and the number of the nodes cannot be increased. The data insertion operation is successful when at least one sub-graph is non-maximal.
During key-value pair insertion, the present invention needs to determine if numerical value insertion would cause an infinite kick-out loop problem. To know the result after data insertion, it needs to periodically determine the state condition of each sub-graph, whether it is the maximum sub-graph or the non-maximum sub-graph.
Specifically, the technical scheme of the invention is characterized in that:
(1) method for constructing elastic Key-Value pair data storage following server-client architecture mode
The system can be integrally deployed in an InfiniBand high-performance network cluster environment, plays roles of data access and providing I/O request service for upper-layer data analysis type application, and comprises a client and a server. Specifically, the client locates the position of key-value pair data storage according to the improved Cuckoo Hash data management mode-G-Cuckoo (combined with Cuckoo graph) designed by the invention. The system effectively manages key-value pair data in the hash table and utilizes two-sided RDMA SEND/RECV verbs technology to carry out message communication. The key-value pair storage prototyping system developed may be continually refined to become a reliable storage system, or individually to become a storage component of a data-intensive computing system.
(2) Improved G-Cuckoo hash data management mode
When a key-value pair data entry is inserted, the conventional Cuckoo hash data management mode adopts a random walk method to find an idle bucket position, which causes the problems of infinite loop and performance degradation when data is inserted into a hash table. It should be noted that in the classic Cuckoo data management mode, in order to find a free data entry insertion position, the continuous kisk-out operation problem may be interrupted and the alternative re-hashing operation may be carried out, but a problem of a large time delay may occur in the process of carrying out a large number of kisk-out operations. Interrupting the kick-out operation and moving to an alternative re-hashing operation does not solve the infinite loop problem well. To obtain a better Cuckoo hash data management schema, the infinite loop problem is detected. The method improves the classic Cuckoo hash data management mode and integrates infinite loop prediction in the kick-out operation. Mapping each position in the hash table into a Hash diagram in a bucket-vertex mapping mode, providing an improved Cuckoo hash data management mode, namely G-Cuckoo, and predicting to avoid the problem of infinite kick-out.
(3) RDMA (remote direct memory Access) -based construction of RDMA (remote direct memory Access) network communication engine
It is very difficult to directly use RDMA READs to perform data reading and writing and to perform complex pointer operations. Between a client and a server, analyzing the key value-to-storage performance bottleneck problem caused by round response to and fro in the traditional network, designing a network communication engine by using Infiniband Remote Direct Memory Access (RDMA) SEND/RECV verbs technology, and receiving a data Access request and sending back a data request result. InfiniBand RoCE (RDMA over converted Ethernet) is capable of data manipulation with low latency. RDMANICs provide reliable transmission based on hardware lost packet retransmission. In effect, the RDMA-capable NICs contain two-sided tunnel semantics and one-sided memory semantics that are bypassed by the target CPU. In the proposed key-value pair data management method, the Infiniband RDMA network engine for data message communication employs the two-sided tunneling semantics.
(4) Atomic properties of data operations based on HTM
Since the key-value pair data storage method is based on the improved G-Cuckoo hash data management mode, the data operation is related to the data operation of the internal bottom layer. For a simple basic data operation, the change of the internal data structure is relatively complex, and if there is no atomic property guarantee of the data operation, under the condition of concurrent data operation, the data read and write will cause deadlock or a certain degree of confusion, thereby causing the interruption of the whole data access program and the inaccuracy of the data. The data intensive computing application can access the hash table of the server side, and under the condition of concurrent data requests, the problems of lock competition and request conflict exist. In order to ensure atomicity of data operation, the operation of each hash table needs to use a lock, so as to ensure that the atomic property of the Key-Value Key Value pair data record is accessed, and the data operation is not interfered by other requests.
(5) Data consistency assurance technology
The key value pair storage method provided by the invention guarantees the consistency of data by using a Checksum data consistency guarantee technology, and ensures that no error occurs in the transmission of the data between the server side and the client side to a certain extent. Meanwhile, the data consistency guarantee technology is light, and is simpler compared with other data consistency guarantee technologies with heavier burdens, and too many hardware resources are not needed. The consistency and correctness of the data are ensured under the condition of ensuring good data access performance.
One embodiment of the present invention is as follows:
the RDMA and HTM based elastic Key-Value Key Value pair storage system Key implementation and software interface comprises:
(1) software interfaces are abstracted, and the system provides basic operation interfaces for users to use, wherein the interfaces comprise a numerical setting function rhkv _ set, a numerical obtaining function rhkv _ get, a numerical updating function rhkv _ update and the like.
The underlying details provided by the software interface are described below: for the function rhkv _ get, the key parameter m is passed to the Hash function Hash over the Infiniband network. The abstract function calculates two candidate positions through mask operation, finds the actual value condition of the hash table and returns the final result. The actual process may be more complex for the function rhkv _ set. Specifically, the function first finds a key value entry in the cache stash. If the entry exists, updating the key-value pair entry; otherwise, it will find the actual location in the Hash table, and use the Hash function Hash to calculate the actual Hash location of the Hash table. In addition, the constructed Cuckoo graph will be updated synchronously. For the function rhkv _ update, as with the set operation, the actual location in the hash table or cache stamp is obtained, and the old value is updated to the new value.
(2) Different hierarchies are defined in the Infiniband configuration component, including a network layer, a transport layer, an encryption layer, an application layer, and corresponding setter and getter methods. The functions setHostName, setPort, getHostname and getPort are defined in the communication module and are a function for setting a host name, a function for setting a port number, a function for acquiring the host name and a function for acquiring the port number respectively.
More, the function server _ on _ complete modifies the state of receive-state to MSG _ RECEIVED. When the server receives the RDMA-read request, the function uses the dynamic buffer and passes the parameters to the SERVERuser, and the user smoothly receives the data information. When the message is processed, the function resets ib _ connection. In addition, in the function client _ on _ complete, if the send-state is set to the MSG _ send state, the receive-status is set to the INIT state, which means that the client receives the request from the server segment, and the server-side information contains RDMA details. When the client READs the data, the state of send-state changes to REDMA _ READ _ DONE. FIG. 4 is a diagram of the RDMA communication model of the present invention.
(3) In the hash data management module, a Node data structure is defined and comprises a variable appVar, a variable occuVar and a variable subVar. Wherein, the variable appVar is used for judging whether the subgraph is a non-maximum subgraph or a maximum subgraph; the variable occuVar indicates whether the vertex in the graph is occupied; the variable subVar represents the identification number of the sub-graph where the vertex is located.
In order to effectively manage the hash table and detect whether infinite kick-out exists in the key value insertion process, some main functions are defined, and the data operation on the hash table is realized by using a function search and a function update. In the implementation process of the function insert, a function judggenom needs to be called to judge whether the subgraph is a non-maximum subgraph or a maximum subgraph. In addition, a data structure UnionSet is designed to carry out subgraph merging operation. The cache data structure is designed to store a portion of the data that has been accessed. The whole data operation phase is divided into the following parts: (1) checking, loading and updating a cache data structure; (2) the server side responds; (3) the client side waits; (4) and returning the result.
And carrying out effective experiment to evaluate the performance of the prototype system, wherein the experiment evaluation can carry out the experiment performance evaluation by using YCSB load which obeys Zipfian and Uniform distribution. Experimental evaluation was performed in the InfiniBand (FDRinterconnects) environment and compared to the actual performance of key-value pair storage systems.
The application scenarios of the present invention are as follows:
distributed parallel computing systems such as S4, Spark, etc. continue to emerge, and the I/O interaction of disk-based storage systems (e.g., HDFS or cloud-based NoSQL storage systems) associated therewith remains a major performance bottleneck. The system prefetches data from the HDFS to the actual program calculation, provides corresponding data service for the upper application I/O request, and sets up an I/O gap of a MapReduce stack. More, for practical commercial value, as the number of internet users is increasing, internet advertising services are rapidly developing, and real-time advertisement bidding systems are also continuously presented. In order to better satisfy the service quality, the user can use the method and the system of the invention to store the advertisement data and improve the actual rate of the access of the advertisement data.
The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes or modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention. The embodiments and features of the embodiments of the present application may be combined with each other arbitrarily without conflict.
Claims (7)
1. An elastic Key-Value Key Value pair data storage method based on RDMA and HTM is characterized by comprising the following steps:
step 1: following a server-client architecture mode, constructing a large-scale data intensive computing elastic Key-Value Key Value pair data storage management system by utilizing a hardware transaction memory HTM and remote direct memory access RDMA;
step 2: at a server side, a key value pair data storage management system constructs a Hash diagram by using a bucket-point bucket-vertex mapping mode, and provides an improved G-Cuckoo Hash data management mode, so that the problem that data items are kicked out by infinite loop kick-out between Hash tables due to the fact that an idle position needs to be searched in the data insertion process is solved;
and step 3: acquiring a data request of a client through a user-defined data access interface;
and 4, step 4: between a client and a server, analyzing the key value-to-storage performance bottleneck problem caused by round response of the traditional network, designing an RDMA network communication engine by using an Infiniband remote direct memory access RDMA SEND/RECV verbs technology, and receiving a data access request and returning a data request result to the client;
and 5: the method comprises the steps that a hardware transactional memory HTM is utilized to realize locking operation following a locking and releasing protocol, a data operation HTM area is formed, and the atomic characteristics of basic data operation are guaranteed;
step 6: self-verifying Checksum check codes of key-value pair data are used to ensure data consistency;
in the step 2, the G-Cuckoo hash data management mode is specifically as follows:
selecting a classic Cuckoo hash data management mode, improving correspondingly, and using the classic Cuckoo hash data management mode as a key value pair storage system data management strategy, wherein the classic Cuckoo hash data management mode comprises the following steps: if there are d (d >0) hash tables, let K be the key set of key-value pairs, and hash table T1 and hash table T2 possess n data items respectively, given two independent functions h1, h2: K → { 0.,. n-1}, the key K ∈ K can be inserted into position h1(K) of hash table T1 or position h2(K) of hash table T2, the Cuckoo hash data management mode possesses basic data operation mode; when the key-value pair PUT operation is carried out, the server side needs to calculate candidate positions for inserting the key values, the key has one or two positions in the hash table, once the candidate position needing to be inserted is occupied by other key-value pair data, the kick-out operation is executed, namely the key values need to be kicked from one hash table to the idle position of the other hash table;
when a key-value pair data entry is inserted, the traditional Cuckoo Hash data management mode adopts a random walk method to search for an idle bucket position, which causes the problems of infinite circulation and performance degradation of the inserted data; in a classic Cuckoo hash data management mode, in order to find an idle data entry insertion position, a constant kick-out operation problem can be interrupted, and alternative re-hash operation is carried out, so that a time delay problem can occur;
a classic Cuckoo hash data management mode is improved, an infinite loop prediction function is fused in a kick-out operation, when d is 2, the Cuckoo hash data management mode has two tables, each actual position inserted into the hash data management mode has a corresponding backup position, storage positions after the kick-out operation are provided, multiple edges exist among the tables, each position of the hash table is mapped into a Hash diagram in a bucket-vertex mapping mode, an improved Cuckoo hash data management mode G-Cuckoo is provided, and the problem that the infinite kick-out is kicked back and forth is avoided in prediction.
2. The RDMA and HTM-based elastic Key-Value pair data storage method according to claim 1, wherein in step 1, an elastic Key-Value pair data storage management system is constructed, specifically as follows:
the key-value pair storage system is integrally deployed in an InfiniBand network cluster environment, plays roles of data access and data I/O request service provision for upper-layer data analysis type application, a client locates a key-value pair data storage position according to an improved Cuckoo Hash data management mode G-Cuckoo, the key-value pair storage system manages Hash table key-value pair data, and message communication is carried out by utilizing a two-sided RDMA SEND/RECV verbs technology;
all data are stored in a DRAM (dynamic random access memory) of a server side, and effective data service is provided for a client side through an RDMA (remote direct memory access) network; using the computing power of the server, adopting HTM-aware, RDMA-friendly hash tables, each Key-Value Key-Value pair having a corresponding storage location and backup location in both hash tables, connecting the two locations with an edge, the start of the edge corresponding to the actual storage location of the Key-Value pair and the end of the edge corresponding to the actual backup storage location of the Key-Value pair, the hash data management mode adopting two static hash functions h1(x) and h2 (x); when a data request is sent to the server, the related control message needs to be sent to the server in advance; and after receiving the data request, the server side provides data read-write service, encapsulates the data into payload and returns the payload to the client side.
3. The RDMA and HTM-based elastic Key-Value pair data storage method according to claim 1, wherein in step 3, a data access interface is defined to obtain a data request of a client, specifically as follows:
basic operation interfaces used by a user are provided, and the interfaces comprise a numerical setting function rhkv _ set, a numerical obtaining function rhkv _ get and a numerical updating function rhkv _ update.
4. The RDMA and HTM based elastic Key-Value pair data storage method according to claim 1, wherein in step 4, an Infiniband RDMA network communication engine is designed, specifically as follows:
the data message communication operation Infiniband RDMA network engine adopts two-position tunnel semantics, a user space program accesses RDMA NICs by using an RDMA verbs function, wherein the RDMA NICs provide reliable transmission based on a hardware lost packet retransmission mode, the RDMANICs comprise the two-position tunnel semantics and one-position memory semantics bypassing a target CPU, and a message queue is maintained in the RDMANICs and exists in two forms: a sending queue send queue and a receiving queue receive queue, which are called as aque pair; when the RHVV receives a data operation request, data interaction is carried out between the RHDV and the improved G-Cuckoo hash data management mode through the RDMA network engine, and finally the corresponding key value pair data position is found for data insertion or other data operation.
5. The RDMA and HTM-based elastic Key-Value pair data storage method according to claim 1, wherein in step 5, a data operation HTM region is formed following a lock operation of a lock and release protocol, specifically comprising:
during the Start phase, HTM transactions lock and prefetch remote data records, initiating HTM transactions by running XACQUIRE; in the local transaction LocalTX phase, the HTM transaction provides transaction-level reads and writes to all local records; in the Commit phase, the data record is updated to release the lock by committing the HTM transaction of the data operation using XRELESE.
6. The RDMA and HTM-based elastic Key-Value pair data storage method according to claim 1, wherein in said step 6, the Key-Value pair data self-verification Checksum procedure is as follows:
(1) firstly, calculating an actual hash value of a key value;
(2) inquiring a value to be obtained from the hash table;
(3) calculating a Checksum value according to the content of the key-value pair;
(4) the comparison is made as to whether the Checksum value of the packet stored in the hash table is the same.
7. The RDMA and HTM based elastic Key-Value Key Value pair data storage method of claim 4, wherein the request for data requires message interaction with a Hash data management mode, specifically comprising:
(1) the Server side Server registers and calls back an operation handle, binds and monitors a newly arrived client side operation request, and after the data request operation is executed, the connection is interrupted and the unbinding operation of the client side connection monitoring is carried out;
(2) client sets host information, RDMA communication components give constructor and destructor respectively used for initializing and releasing Infiniband connection, and function connection is defined for adding connection array; in an Infiniband RDMA communication component, an Infiniband connection array is operated by using functions init, insert and free, and contents sent by a client are stored in a charvector data structure and encapsulated into a communication message.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810070442.1A CN110069431B (en) | 2018-01-24 | 2018-01-24 | Elastic Key-Value Key Value pair data storage method based on RDMA and HTM |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810070442.1A CN110069431B (en) | 2018-01-24 | 2018-01-24 | Elastic Key-Value Key Value pair data storage method based on RDMA and HTM |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110069431A CN110069431A (en) | 2019-07-30 |
| CN110069431B true CN110069431B (en) | 2020-11-24 |
Family
ID=67365734
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810070442.1A Expired - Fee Related CN110069431B (en) | 2018-01-24 | 2018-01-24 | Elastic Key-Value Key Value pair data storage method based on RDMA and HTM |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110069431B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110727612B (en) * | 2019-09-09 | 2021-01-15 | 无锡江南计算技术研究所 | Calculation buffer memory device based on accurate prefetching |
| CN111400306B (en) * | 2020-02-20 | 2023-03-28 | 上海交通大学 | RDMA (remote direct memory Access) -and non-volatile memory-based radix tree access system |
| CN111404931B (en) * | 2020-03-13 | 2021-03-30 | 清华大学 | Remote data transmission method based on persistent memory |
| CN112612803B (en) * | 2020-12-22 | 2022-07-12 | 浙江大学 | Key value pair storage system based on persistent memory and data concurrent insertion method |
| CN112631925B (en) * | 2020-12-29 | 2024-06-25 | 北京轩宇信息技术有限公司 | Method for detecting single-variable atom violation defect |
| CN113326033B (en) * | 2021-06-09 | 2023-08-11 | 北京八分量信息科技有限公司 | Key-value storage system with multi-language API |
| CN117573602B (en) * | 2024-01-16 | 2024-05-14 | 珠海星云智联科技有限公司 | Method and computer device for remote direct memory access message transmission |
| CN117812039B (en) * | 2024-02-24 | 2024-05-14 | 深圳赋乐科技集团有限公司 | Network address translation log recording method, system, equipment and medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105404546A (en) * | 2015-11-10 | 2016-03-16 | 上海交通大学 | RDMA and HTM based distributed concurrency control method |
| CN105938446A (en) * | 2016-01-12 | 2016-09-14 | 上海交通大学 | Data copying and fault-tolerance method based on support of RDMA and HTM |
-
2018
- 2018-01-24 CN CN201810070442.1A patent/CN110069431B/en not_active Expired - Fee Related
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105404546A (en) * | 2015-11-10 | 2016-03-16 | 上海交通大学 | RDMA and HTM based distributed concurrency control method |
| CN105938446A (en) * | 2016-01-12 | 2016-09-14 | 上海交通大学 | Data copying and fault-tolerance method based on support of RDMA and HTM |
Non-Patent Citations (2)
| Title |
|---|
| "Fast and General Distributed Transactions using RDMA and HTM";Yanzhe Chen etc.;《EuroSys’16》;20160421;第1-7页 * |
| "Key-Value Memory Networks for Directly Reading Documents";Alexander H. Miller etc.;《Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing》;20161105;第1400-1409页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110069431A (en) | 2019-07-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110069431B (en) | Elastic Key-Value Key Value pair data storage method based on RDMA and HTM | |
| US10176057B2 (en) | Multi-lock caches | |
| US9792322B2 (en) | Aggregation framework system architecture and method | |
| US10803066B2 (en) | Methods and systems for hardware acceleration of database operations and queries for a versioned database based on multiple hardware accelerators | |
| US8165988B2 (en) | Fast batch loading and incremental loading of data into a database | |
| CN108431774B (en) | Infinite memory fabric flow and API | |
| US9652287B2 (en) | Using databases for both transactions and analysis | |
| US20160246861A1 (en) | Aggregation framework system architecture and method | |
| US20160179865A1 (en) | Method and system for concurrency control in log-structured merge data stores | |
| CN111597015B (en) | Transaction processing method and device, computer equipment and storage medium | |
| CN113282236A (en) | Object memory structure and hardware-based processing node and method thereof | |
| US9229869B1 (en) | Multi-lock caches | |
| CN111159252A (en) | Transaction execution method and device, computer equipment and storage medium | |
| CN108885607A (en) | Use the memory construction operation of fault tolerant object and consistency | |
| US11048669B2 (en) | Replicated state management using journal-based registers | |
| Xia et al. | Graph analytics and storage | |
| Bhatotia | Incremental parallel and distributed systems | |
| US20230367815A1 (en) | Energy-efficient hardware-software collaborative method and apparatus for graph processing | |
| US10642750B2 (en) | System and method of a shared memory hash table with notifications and reduced memory utilization | |
| US20220342888A1 (en) | Object tagging | |
| US20240143594A1 (en) | Offloading graph components to persistent storage for reducing resident memory in distributed graph processing | |
| Wu et al. | RHKV: An RDMA and HTM friendly key–value store for data-intensive computing | |
| CN114756509B (en) | File system operation method, system, device and storage medium | |
| Qian et al. | Xfast: Extreme File Attribute Stat Acceleration for Lustre | |
| Kunkel | Performance Analysis of the PVFS2 Persistency Layer |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20201124 |