CN106789175A - It is a kind of that redis methods extending transversely are carried out based on uniformity hash algorithm - Google Patents
It is a kind of that redis methods extending transversely are carried out based on uniformity hash algorithm Download PDFInfo
- Publication number
- CN106789175A CN106789175A CN201611079971.5A CN201611079971A CN106789175A CN 106789175 A CN106789175 A CN 106789175A CN 201611079971 A CN201611079971 A CN 201611079971A CN 106789175 A CN106789175 A CN 106789175A
- Authority
- CN
- China
- Prior art keywords
- server
- node
- redis
- data storage
- annulus
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0654—Management of faults, events, alarms or notifications using network fault recovery
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/14—Session management
- H04L67/148—Migration or transfer of sessions
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer And Data Communications (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention is provided carries out redis methods extending transversely based on uniformity hash algorithm, and step 1, based on each server node information of redis, using hash function, obtains the cryptographic Hash of each node of redis servers；The cryptographic Hash of each node of redis servers is distributed in into one has on the 2^32 annulus of uniform arrangement point；Step 2, based on the key information for needing data storage, using hash function, obtains the cryptographic Hash for needing each key of data storage；The cryptographic Hash for needing each key of data storage is mapped on the annulus；Step 3, searches the server node being distributed on the annulus, and need data storage to be stored on first server node for finding by described clockwise since the position for needing each key of data storage to be mapped on the annulus respectively.The data of storage will be needed uniformly to be distributed on each server, while realizing the technique effect of failed server node data fast transfer.
Description
Technical field
The present invention relates to field of computer technology, redis is carried out based on uniformity hash algorithm more particularly, to one kind
Method extending transversely.
Background technology
At present, current web application systems are to solve the performance issue that high concurrent is brought, all can web application systems with
Cache layer is taken between database, the most widely used is exactly that (Redis is substantially an internal memory for key-value type to Redis
Database).High-performance and shirtsleeve operation order of the Redis due to it, completely can be as between software and traditional database
Cushioning effect, in the case where ensure that data validity, while also ensure that high-performance.
Redis does not provide distributed support in itself, so when many Redis servers are disposed, it is necessary to which how is solution
Data are distributed to the problem of each server, and when number of servers changes, can accomplish farthest not make data
Redistribution.
Usually used distributed method is the hash values and number of servers N according to the key of wanted data storage, is pressed
The algorithm of hash%N modulus is by data distribution to each server.The advantage of the algorithm is simple enough, and data distribution
Uniformly.But once number of servers N is when change, cache hit rate can fall into the lowest point moment, because most
Data need redistribution.Now have huge pressure and pour into back-end services, may result in performance fault and service event
Barrier, or even machine of delaying.
Accordingly, it is desired to provide a kind of method for more effectively realizing that Redis is extending transversely.
The content of the invention
The present invention provides one kind and overcomes above mentioned problem or at least in part solution to the problems described above.
According to an aspect of the invention, there is provided a kind of method redis extending transversely, comprises the following steps：Step
1, based on each server node information of redis, using hash function, obtain the cryptographic Hash of each node of redis servers；Will
The cryptographic Hash of each node of redis servers is distributed in one to be had on the 2^32 annulus of uniform arrangement point；
Step 2, based on the key information for needing data storage, using hash function, acquisition is described to need each key of data storage
Cryptographic Hash；The cryptographic Hash for needing each key of data storage is mapped on the annulus；
Step 3, searches distribution clockwise since the position for needing each key of data storage to be mapped on the annulus respectively
Server node on the annulus, and need data storage to be stored in first server node for finding by described
On.
Present applicant proposes a kind of redis methods extending transversely, using uniformity hash function, it would be desirable to the number of storage
According to being uniformly distributed on each server, while realizing the technique effect of failed server node data fast transfer.
Brief description of the drawings
Fig. 1 is the overall procedure schematic diagram according to method a kind of redis of the invention extending transversely；
Fig. 2 is the schematic diagram according to Hash annulus in method a kind of redis of the invention extending transversely.
Specific embodiment
With reference to the accompanying drawings and examples, specific embodiment of the invention is described in further detail.Hereinafter implement
Example is not limited to the scope of the present invention for illustrating the present invention.
Such as Fig. 1, in a specific embodiment of the invention, show that a kind of redis method overall flows extending transversely are shown
It is intended to.On the whole, including：Step 1, based on each server node information of redis, using hash function, obtains the redis clothes
The cryptographic Hash of business each node of device；The cryptographic Hash of each node of redis servers is distributed in into one has 2^32 uniform row
On the annulus layouted；Step 2, based on the key information for needing data storage, using hash function, acquisition is described to need data storage
The cryptographic Hash of each key；The cryptographic Hash for needing each key of data storage is mapped on the annulus；Step 3, respectively from needs
The position that each key of data storage is mapped on the annulus starts to search clockwise the server node being distributed on the annulus,
And need data storage to be stored on first server node for finding by described.
In another specific embodiment of the invention, a kind of method redis extending transversely, due to Hash (hash) algorithm knot
Fruit is generally unsigned int types, therefore should be evenly distributed on [0,2^32-1] interval for the result of hash functions, such as
Really we uniformly cut an annulus with 2^32 point, are first according to hash (key) function and calculate server node
Cryptographic Hash, and be distributed on the annulus of 0~2^32.
The cryptographic Hash of the key for needing data storage is obtained with same hash (key) function, and is mapped on annulus.Then
Searched clockwise since the position that data are mapped to, stored data on first server (node) found.Such as Fig. 2
Shown, key1, key2, key3 and server1, server2 can find the position of oneself by hash on this annulus, and
And key is navigated to by server by clockwise mode.For upper figure, server1 is arrived in key1 and key2 storages, and
Server2 is arrived in key3 storages.If increasing a server newly, after hash between key1 and key2, then key1 only can be influenceed
(key1 will be stored on newly-increased server), other are then constant.
In another specific embodiment of the invention, a kind of method redis extending transversely, the step 1 also includes：It is based on
Each virtual server node of node sets at least one of redis servers, using hash function, obtains each virtual server
The cryptographic Hash of node；The cryptographic Hash of each virtual server node is distributed in described with the 2^32 circle of uniform arrangement point
On ring.
In another specific embodiment of the invention, a kind of method redis extending transversely, in fig. 2, it is easy to find out one
Individual problem, sees along clockwise direction, and the interval span between server2 to server1 is big, and the area of server1 to server2
Between span it is small, this may result in a problem：Data distribution is uneven.Most of data are all assigned to server1, only small
Partial data is distributed in server2.When server data is little, data are uneven to be showed clearly.Solve
The method of this problem is to use dummy node, and a real server correspondence multiple dummy node, all dummy nodes are pressed
Hash Distribution values are on uniformity Hash annulus.
In another specific embodiment of the invention, a kind of method redis extending transversely also includes after the step 3：When
When certain or multiple server nodes break down, the data in the failed server are stored in other normal works
On server.
In another specific embodiment of the invention, a kind of method redis extending transversely is based on redis in the step 1
Each virtual server node of node sets at least one of server also includes：For true redis servers set virtual server
Number of nodes, then according to each redis real server IP and port numbers along with an incremental index number calculating hash value.
In another specific embodiment of the invention, a kind of method redis extending transversely, the virtual server nodes
Amount can be adjusted by the quantity of real server, and real server at most copy amount can set a little less, real server
Copy amount needs to set more at least.
In another specific embodiment of the invention, a kind of method redis extending transversely, in virtual server number of nodes
When very big, the corresponding dummy node of each key of data storage is needed using binary search.No matter the algorithm of an optimization is copy
When quantity is set to 10,100 or 10000, on the time required for lookup be substantially do not have it is influential.
In another specific embodiment of the invention, a kind of method redis extending transversely, the step when certain or it is many
When individual server node breaks down, the data in the failed server are stored on the server of other normal works also
Including：Detection failure real server is simultaneously recorded, and is along one step of shifting, and judgement along clockwise direction on disposable Hash annulus
It is not the failure real server being recorded；If it is, next dummy node is directly searched, it is available until finding
Server gets on data storage.
In another specific embodiment of the invention, a kind of method redis extending transversely, according to key find it is virtual
When real server failure corresponding to node, found along a step is moved along clockwise direction on disposable Hash annulus
The corresponding real server of subsequent point dummy node, the data that will be stored storage is got on.Very possible next virtual section
The corresponding real server of point is identical with previous dummy node, or that failure server, and attempt connection every time
The redis services of failure are a very big performance costs.So being accomplished by when first time failed server is detected
Record, then along move on to next dummy node when first judge whether before that failure server,
If that should not just be reattempted being attached, next dummy node is directly searched, until finding available server by number
Get on according to storage.
Finally, the present processes are only preferably embodiment, are not intended to limit the scope of the present invention.It is all
Within the spirit and principles in the present invention, any modification, equivalent substitution and improvements made etc. should be included in protection of the invention
Within the scope of.
Claims (7)
1. a kind of method redis extending transversely, it is characterised in that comprise the following steps：
Step 1, based on each server node information of redis, using hash function, obtains each node of redis servers
Cryptographic Hash；The cryptographic Hash of each node of redis servers is distributed in into one has on the 2^32 annulus of uniform arrangement point；
Step 2, based on the key information for needing data storage, using hash function, obtains the Kazakhstan for needing each key of data storage
Uncommon value；The cryptographic Hash for needing each key of data storage is mapped on the annulus；
Step 3, searches be distributed in institute respectively clockwise since the position for needing each key of data storage to be mapped on the annulus
The server node on annulus is stated, and needs data storage to be stored on first server node for finding by described.
2. the method for claim 1, it is characterised in that the step 1 also includes：Based on each node of redis servers
At least one virtual server node is set, using hash function, the cryptographic Hash of each virtual server node is obtained；By institute
The cryptographic Hash for stating each virtual server node is distributed on the annulus of the point of uniformly being arranged with 2^32.
3. method as claimed in claim 2, it is characterised in that also include after the step 3：When certain or multiple server
When node breaks down, the data in the failed server are stored on the server of other normal works.
4. method as claimed in claim 2, it is characterised in that in the step 1 based on each node sets of redis servers extremely
A few virtual server node also includes：For true redis servers set virtual server number of nodes, then according to each
Redis real server IP and port numbers are along with an incremental index number calculating hash value.
5. method as claimed in claim 3, it is characterised in that the virtual server number of nodes can be by real server
Quantity regulation, real server at most copy amount can set a little less, and copy amount needs to set real server at least
Put more.
6. method as claimed in claim 3, it is characterised in that when virtual server number of nodes is very big, using dichotomy
Lookup needs the corresponding dummy node of each key of data storage.
7. method as claimed in claim 4, it is characterised in that the step occur when certain or multiple server nodes therefore
During barrier, the data in the failed server are stored on the server of other normal works also to be included：Detection failure is true
Server is simultaneously recorded, and along clockwise direction along one step of shifting on disposable Hash annulus, and judges to be described being recorded
Failure real server；If it is, next dummy node is directly searched, until finding available server by data storage
Up.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611079971.5A CN106789175A (en) | 2016-11-30 | 2016-11-30 | It is a kind of that redis methods extending transversely are carried out based on uniformity hash algorithm |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611079971.5A CN106789175A (en) | 2016-11-30 | 2016-11-30 | It is a kind of that redis methods extending transversely are carried out based on uniformity hash algorithm |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106789175A true CN106789175A (en) | 2017-05-31 |
Family
ID=58901309
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611079971.5A Pending CN106789175A (en) | 2016-11-30 | 2016-11-30 | It is a kind of that redis methods extending transversely are carried out based on uniformity hash algorithm |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106789175A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107562533A (en) * | 2017-07-28 | 2018-01-09 | 阿里巴巴集团控股有限公司 | A kind of data loading processing method and device |
| CN108769111A (en) * | 2018-04-17 | 2018-11-06 | 平安科技（深圳）有限公司 | A kind of server connection method, computer readable storage medium and terminal device |
| CN109101635A (en) * | 2018-08-16 | 2018-12-28 | 广州小鹏汽车科技有限公司 | A kind of data processing method and device based on Redis Hash structure |
| CN109246250A (en) * | 2018-11-08 | 2019-01-18 | 程桂平 | The method for adjusting dummy node quantity according to the change of number of servers |
| CN109271391A (en) * | 2018-09-29 | 2019-01-25 | 武汉极意网络科技有限公司 | Date storage method, server, storage medium and device |
| CN111382200A (en) * | 2018-12-29 | 2020-07-07 | 北京中交兴路信息科技有限公司 | Information loading method and device, server and storage medium |
| CN115297131A (en) * | 2022-08-01 | 2022-11-04 | 东北大学 | Sensitive data distributed storage method based on consistent hash |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102045413A (en) * | 2011-01-24 | 2011-05-04 | 北京邮电大学 | DHT expanded DNS mapping system and method for realizing DNS security |
| CN102457571A (en) * | 2011-09-15 | 2012-05-16 | 中标软件有限公司 | Method for uniformly distributing data in cloud storage |
| CN102682110A (en) * | 2012-05-10 | 2012-09-19 | 北京大学 | High-performance cache design method orienting to massive spatial information |
| CN102881008A (en) * | 2012-08-16 | 2013-01-16 | 广西师范大学 | Circular loop statistic characteristic-based anti-rotation image Hash method |
| CN103116661A (en) * | 2013-03-20 | 2013-05-22 | 广东宜通世纪科技股份有限公司 | Data processing method of database |
| CN104050270A (en) * | 2014-06-23 | 2014-09-17 | 成都康赛信息技术有限公司 | Distributed storage method based on consistent Hash algorithm |
| CN104199957A (en) * | 2014-09-17 | 2014-12-10 | 合一网络技术(北京)有限公司 | Redis universal agent implementation method |
| CN104811493A (en) * | 2015-04-21 | 2015-07-29 | 华中科技大学 | Network-aware virtual machine mirroring storage system and read-write request handling method |
| CN105677736A (en) * | 2015-12-29 | 2016-06-15 | 曙光信息产业(北京)有限公司 | Method and apparatus for increasing and deleting server nodes |
-
2016
- 2016-11-30 CN CN201611079971.5A patent/CN106789175A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102045413A (en) * | 2011-01-24 | 2011-05-04 | 北京邮电大学 | DHT expanded DNS mapping system and method for realizing DNS security |
| CN102457571A (en) * | 2011-09-15 | 2012-05-16 | 中标软件有限公司 | Method for uniformly distributing data in cloud storage |
| CN102682110A (en) * | 2012-05-10 | 2012-09-19 | 北京大学 | High-performance cache design method orienting to massive spatial information |
| CN102881008A (en) * | 2012-08-16 | 2013-01-16 | 广西师范大学 | Circular loop statistic characteristic-based anti-rotation image Hash method |
| CN103116661A (en) * | 2013-03-20 | 2013-05-22 | 广东宜通世纪科技股份有限公司 | Data processing method of database |
| CN104050270A (en) * | 2014-06-23 | 2014-09-17 | 成都康赛信息技术有限公司 | Distributed storage method based on consistent Hash algorithm |
| CN104199957A (en) * | 2014-09-17 | 2014-12-10 | 合一网络技术(北京)有限公司 | Redis universal agent implementation method |
| CN104811493A (en) * | 2015-04-21 | 2015-07-29 | 华中科技大学 | Network-aware virtual machine mirroring storage system and read-write request handling method |
| CN105677736A (en) * | 2015-12-29 | 2016-06-15 | 曙光信息产业(北京)有限公司 | Method and apparatus for increasing and deleting server nodes |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107562533A (en) * | 2017-07-28 | 2018-01-09 | 阿里巴巴集团控股有限公司 | A kind of data loading processing method and device |
| CN108769111A (en) * | 2018-04-17 | 2018-11-06 | 平安科技（深圳）有限公司 | A kind of server connection method, computer readable storage medium and terminal device |
| CN109101635A (en) * | 2018-08-16 | 2018-12-28 | 广州小鹏汽车科技有限公司 | A kind of data processing method and device based on Redis Hash structure |
| CN109101635B (en) * | 2018-08-16 | 2020-09-11 | 广州小鹏汽车科技有限公司 | Data processing method and device based on Redis Hash structure |
| CN109271391A (en) * | 2018-09-29 | 2019-01-25 | 武汉极意网络科技有限公司 | Date storage method, server, storage medium and device |
| CN109271391B (en) * | 2018-09-29 | 2021-05-28 | 武汉极意网络科技有限公司 | Data storage method, server, storage medium and device |
| CN109246250A (en) * | 2018-11-08 | 2019-01-18 | 程桂平 | The method for adjusting dummy node quantity according to the change of number of servers |
| CN111382200A (en) * | 2018-12-29 | 2020-07-07 | 北京中交兴路信息科技有限公司 | Information loading method and device, server and storage medium |
| CN115297131A (en) * | 2022-08-01 | 2022-11-04 | 东北大学 | Sensitive data distributed storage method based on consistent hash |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106789175A (en) | It is a kind of that redis methods extending transversely are carried out based on uniformity hash algorithm | |
| CN107408128B (en) | System and method for providing access to a sharded database using caching and shard topology | |
| US10691366B2 (en) | Policy-based hierarchical data protection in distributed storage | |
| WO2021003935A1 (en) | Data cluster storage method and apparatus, and computer device | |
| US20170262638A1 (en) | Distributed database systems and methods with encrypted storage engines | |
| US9251232B2 (en) | Database controller, method, and system for storing encoded triples | |
| US8943103B2 (en) | Improvements to query execution in a parallel elastic database management system | |
| US8438574B1 (en) | Generating monotone hash preferences | |
| US20120179723A1 (en) | Data replication and failure recovery method for distributed key-value store | |
| US20200065306A1 (en) | Bloom filter partitioning | |
| CN105550371A (en) | Big data environment oriented metadata organization method and system | |
| US20030074402A1 (en) | Method and apparatus for providing scalable resource discovery | |
| CN108256076A (en) | Distributed mass data processing method and processing device | |
| CN107483519A (en) | A kind of Memcache load-balancing methods and its system | |
| CN107665246A (en) | Dynamic date migration method and chart database cluster based on chart database | |
| Bhagwat et al. | Content-based document routing and index partitioning for scalable similarity-based searches in a large corpus | |
| Attasena et al. | fvss: A new secure and cost-efficient scheme for cloud data warehouses | |
| Doka et al. | Online querying of d-dimensional hierarchies | |
| CN106250440B (en) | Document management method and device | |
| WO2017045545A1 (en) | Method and apparatus for managing loads of multiple storage disks, file system, and storage network system | |
| JP6055197B2 (en) | Database system | |
| Zhang et al. | Enabling compressed encryption for cloud based big data stores | |
| CN104391931A (en) | Method for efficiently indexing mass data in cloud computing | |
| Batra et al. | A short survey of advantages and applications of skip graphs | |
| Venkateswaran et al. | Handling workload skew in a consistent hashing based partitioning implementation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170531 |
|
| RJ01 | Rejection of invention patent application after publication |