CN105260376A - Method, equipment and system used for cluster node contraction and expansion - Google Patents
Method, equipment and system used for cluster node contraction and expansion Download PDFInfo
- Publication number
- CN105260376A CN105260376A CN201510505399.3A CN201510505399A CN105260376A CN 105260376 A CN105260376 A CN 105260376A CN 201510505399 A CN201510505399 A CN 201510505399A CN 105260376 A CN105260376 A CN 105260376A
- Authority
- CN
- China
- Prior art keywords
- section point
- node
- cluster
- thread
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a method, equipment and a system for cluster node contraction and expansion. The method comprises the following steps: receiving a redis database snapshot generated by a plurality of nodes in parallel; receiving a write operation carried out by a client side after the redis database snapshot is sent; generating attached files which independently correspond to a plurality of second nodes, and independently recording the write operation into the attached files; analyzing the redis database snapshot, and recombining the redis database snapshot into database files which independently correspond to the plurality of second nodes; independently sending the database files and the attached files corresponding to the plurality of second nodes; and indicating a configuration management system which carries out configuration on the client side to send the cluster node topology of a second node cluster to the client side to cause the client side to carry out a read and write operation according to the cluster node topology of the second node cluster. The method, the equipment and the system solve the problems in the prior art that data migration wastes time and is unlikely to carry out on line.
Description
Technical field
The present invention relates to node cluster field, be specifically related to the method, apparatus and system for clustered node reducing and expansion.
Background technology
Redis (long-range dictionary services, RemoteDictionaryService) is a kind of Key/Value (key-value pair) buffer/store system of full internal memory.Due to by individual node place server memory capacity limit, under large-scale data buffer/store scene, generally form cluster by multiple node, each node is only responsible for storage area data, is called sharding.Client, according to the mapping relations of Key (key) and node, is sent to specific node by the request of specific Key.
The mapping relations of Key and node, mode conventional is at present realized by client routing algorithm.Node is numbered in order, using Key as parameter, calculates a numerical value by a Hash function, then carry out delivery (or consistance hash algorithm) to navigate to a node according to the quantity of node.Owing to being carry out route by algorithm, algorithm itself also can ensure that the time complexity of route is O (1), and do not need to store the mapping relations of Key to node anywhere, be therefore mainly used in storage scenarios that is a large amount of, ever-increasing Key/Value.
But client routing algorithm also also exists drawback.Destination node quantity and numbering are once change, the mapping relations of key and node will change, be written to the key/value data of node A by old mapping relations before client, owing to being mapped to Node B by new mapping relations, may go Node B reads and cannot read.Therefore, when needing because data volume increases to increase node, all key (usually directly calling the command interface of redis to perform traversal) must be traveled through, migration data between node, to reflect up-to-date mapping relations.This mode increasing overall cluster capacity by increasing node is called as horizontal extension.The Data Migration that horizontal extension causes usually can be very consuming time and very difficultly to carry out online.
In order to avoid node changes the Data Migration caused.Pre-sharding scheme can be taked, namely pre-estimate data volume and growth rate thereof, set a suitable abundant number of nodes.And, in order to avoid occupying too much physical server, the usual initial stage can at the multiple node of same server deploy, along with the growth of data volume, by by node migrates on the physical server newly increased, make each node have more memory source, thus the memory size of overall cluster is expanded.Migration node is to other servers, and the leader follower replication taking redis itself to provide realizes realizing.Owing to just changing the mapping relations of node and physical machine, number of nodes and numbering do not change, and do not need migration data among the nodes.The mode that this capacity by increase single node, nodes do not change is also referred to as extends perpendicular.
Extends perpendicular needs layout data capacity and growth in advance.Assess suitable abundant node and be not easy.Setting up of number of nodes needs to consider memory consumption and data growth, along with data more than expected increase, even if a physical server disposes a node, expansion cannot be continued owing to being limited by physical server internal memory, unavoidably need horizontal extension to solve the capacity expansion of overall cluster.
And the Data Migration that the horizontal extension spoken of above causes needs to travel through all key, according to new mapping relations, by Data Migration on new node, this whole process usually can be very consuming time.And be difficult to carry out online.
Also there is the horizontal extension scheme of off-line, that is: before expansion, first stop the read-write of all clients, after having moved, then allow client to read and write according to new mapping relations.The horizontal extension of this off-line avoids the data collision and inconsistency that migrator and client-side program carry out reading and writing simultaneously and cause, but needs longer to forbid access time.But business demand does not but usually allow to forbid read-write for a long time, therefore needs a scheme can carrying out online horizontal extension.
Summary of the invention
For solving the problem, embodiments provide a kind of method, apparatus and system of clustered node reducing and expansion.
According to an aspect of the present invention, provide a kind of method for clustered node reducing and expansion, comprising: the redis database snapshot receiving described multiple first node parallel generation from multiple first nodes of first node cluster; From the write operation that described multiple first node reception client is carried out after the transmission of described redis database snapshot; Generate the appended document corresponding respectively with multiple Section Points of Section Point cluster, and according to the mapping relations of set up key to described Section Point cluster, described write operation is recorded in respectively in described appended document; According to the mapping relations of described set up key to described Section Point cluster, analyze described redis database snapshot and described redis database snapshot be recombined in the database file corresponding respectively with described multiple Section Points of described Section Point cluster; The database file corresponding with it and appended document is sent respectively to described multiple Section Point; And the clustered node topology of described Section Point cluster is sent to described client by the configuration management system of the described client of instruction configuration, can carry out read-write operation to make described client according to the clustered node topology of described Section Point cluster.
According to a second aspect of the invention, provide a kind of equipment for clustered node reducing and expansion, comprise: receiving element, for receiving the redis database snapshot of described multiple first node parallel generation from multiple first nodes of first node cluster, and from the write operation that described multiple first node reception client is carried out after the transmission of described redis database snapshot; Appended document generation unit, for generating the appended document corresponding respectively with multiple Section Points of Section Point cluster, and according to the mapping relations of set up key to described Section Point cluster, the described write operation received by described receiving element is recorded in described appended document respectively; Recombination unit, for according to the mapping relations of described set up key to described Section Point cluster, analyze described redis database snapshot that described receiving element receives and described redis database snapshot be recombined in the database file corresponding respectively with described multiple Section Points of described Section Point cluster; Transmitting element, for sending the database file corresponding with it and appended document respectively to described multiple Section Point, and be used to indicate the configuration management system configuring described client the clustered node topology of described Section Point cluster is sent to described client, read-write operation can be carried out according to the clustered node topology of described Section Point cluster to make described client.
According to a third aspect of the invention we, provide a kind of system for clustered node reducing and expansion, comprising: client; Comprise the first node cluster of multiple first node; Comprise the Section Point cluster of multiple Section Point; The above-mentioned equipment for clustered node reducing and expansion; And for configuring the configuration management system of described client, the clustered node topology of described Section Point cluster is sent to described client by described configuration management system, can carry out read-write operation to make described client according to the clustered node topology of described Section Point cluster.
By using above-mentioned method, apparatus and system, solve the Data Migration existed in prior art consuming time and be difficult to the problem of carrying out online.
Accompanying drawing explanation
By the detailed description of carrying out invention below in conjunction with accompanying drawing, above-mentioned feature and advantage of the present invention will be made more obvious, wherein:
Fig. 1 is the schematic diagram that clustered node reducing and expansion system is according to an embodiment of the invention shown;
Fig. 2 is the process flow diagram that clustered node reducing and expansion method is according to an embodiment of the invention shown; And
Fig. 3 is the schematic block diagram that clustered node reducing and expansion equipment is according to an embodiment of the invention shown.
Embodiment
Below, the preferred embodiment of the present invention is described in detail with reference to accompanying drawing.In the accompanying drawings, although be shown in different accompanying drawings, identical Reference numeral is for representing identical or similar assembly.For clarity and conciseness, the detailed description being included in known function and structure here will be omitted, to avoid making theme of the present invention unclear.
In common technical scheme, what the bulk migration of redis node data relied on is redis leader follower replication function.During from node initializing, request host node, by the data snapshot in that moment, is packaged into rdb (binary format of redisdb) and sends to from node, loads rdb initialization self EMS memory from node.Host node have recorded all write operations after that snapshot time simultaneously, and occurs to successively from node by write operation time sequencing.From node on the snapshot data basis loaded, perform all write operations of the follow-up transmission of host node successively, thus reach the data final consistency of master and slave node.
And the scheme that the embodiment of the present invention provides is under the support of redis leader follower replication, a middleware being called redis bridge is provided, by original n node cluster A, by copy with reproduction process in by the new mapping relations of key and node, by the cluster B of Data Migration to m node.In transition process, cluster A continues the read-write requests of service client, by the time B cluster complete copy data and pulled up to the follow-up increment write operation occurring in cluster A, B cluster copy time delay very little time, just allow B cluster service in the read-write requests of client.Particularly, in certain embodiments, according to the time window that business demand allows, by configuration-system, number of nodes new for B cluster and numbering (clustered node topology) are sent to client, forbid that client is write simultaneously.In this time window, client still can carry out a read request according to cluster A node topology.When A and B company-data reaches Complete Synchronization time, configuration-system notice client carries out read-write requests according to B clustered node topology.So far, Data Migration and number of nodes have changed.Old cluster A node resource can discharge.
Redis bridge in the present invention is the middleware achieving redis replication protocol.After redis bridge process initiation, be connected to all nodes on cluster A as the role from node, request copies.Meanwhile, it also allows all nodes of B cluster connect and respond the duplicate requests of the latter as the role of host node.Leader follower replication due to redis is with node (process) for unit, therefore in the embodiment of the present invention can by with connection be replication protocol that granularity realizes redis realize above-mentioned multinode from node and host node.
The concise and to the point system schematic describing the design concept being used for the invention described above below.Fig. 1 shows the schematic diagram of the clustered node reducing and expansion system according to the embodiment of the present invention.
As shown in Figure 1, this system comprises application client (abbreviation client), for configuring the configuration management system of client, the cluster A (in this article also referred to as first node cluster) comprising two nodes, the cluster B (in this article also referred to as Section Point cluster) comprising three nodes that will expand to from cluster A and redis bridge (in this article also referred to as the equipment for clustered node reducing and expansion).
The embodiment of the present invention although it should be noted that in the A of cluster shown in Fig. 1 and comprise two nodes, in cluster B, comprises three nodes, when also can be applied to and there are other various several destination nodes.Such as, in FIG, the number of cluster A interior joint is less than the number of cluster B interior joint, to carry out cluster dilatation.But in certain embodiments, the number of cluster A interior joint can be greater than the number of cluster B interior joint, to realize cluster capacity reducing.Therefore, although in the following description with cluster dilatation for example is described, the method for the embodiment of the present invention also can on an equal basis for the situation of cluster capacity reducing.
System below according to Fig. 1 describes the clustered node reducing and expansion method according to the embodiment of the present invention.Fig. 2 shows the process flow diagram of the clustered node reducing and expansion method according to the embodiment of the present invention.
In the step 210 of Fig. 2, redis database (rdb) snapshot of the plurality of first node parallel generation can be received from multiple first nodes of first node cluster by redis bridge.
Before this step, can at the node topology of redis bridge allocation cluster A and B cluster.
In addition, redis bridge also can set up thread and connection for the node in cluster A and cluster B respectively.
Particularly, B cluster 3 nodes, to be connected to redis bridge from role in master slave relation, initiate synchronous (sync) order.Redis bridge is that 3 of these 3 nodes connect c1, c2, c3 correspondence establishment 3 threads b1, b2, b3, and newly-built 3 isochronous queues (SynchronousQueue) q1, q2, q3.Wherein, b1, b2, b3 consume q1, q2, q3 respectively.When not having data, 3 connect c1, c2, c3 and enter blocked state.
In addition, redis bridge also can newly-built 1 thread t1, is responsible for carrying out rdb conversion.T1 can comprise count synchronization device (CountdownLatch) L1.The initial value of this count synchronization device L1 is identical with the number of cluster A interior joint.Initial count can value be 2 in the example depicted in fig. 1.T1 is blocked in count synchronization device L1 when running, and continues to perform when L1 becomes 0.
Redis bridge also can set up 2 threads a1, a2 for the node of 2 in cluster A respectively.These 2 threads comprise quoting count synchronization device L1.Redis bridge is established to the connection of 2 nodes of cluster A respectively, initiates sync order;
Now 2 nodal parallel of cluster A generate rdb snapshot and issue redis bridge.
In the step 220 of Fig. 2, the write operation that redis bridge carries out after the transmission of redis database snapshot from first node reception client.
Particularly, thread a1 (or a2) receives rdb, and is kept in local file system.Filename for thread a1 and a2 (that is, for first node A1 and A2) is respectively a1.rdb, a2.rdb.According to rdb agreement, ending is the School Affairs of FF+8 byte.After rdb receives, count synchronization device L1 is successively decreased 1 by a1 (or a2).
Next, in the step 230 of Fig. 2, redis bridge generates the appended document corresponding respectively with multiple Section Points of Section Point cluster, and according to the mapping relations of set up key to Section Point cluster, is recorded in respectively in appended document by write operation.
In concrete examples more of the present invention, this has come by thread a1 and a2.After reception rdb, thread a1, a2 continue to receive from the write operation of cluster A, and according to the mapping relations of newly-established key to B clustered node, write operation are kept at b1.aof, b2.aof, b3.aof file of correspondence in additional mode.
Then, in the step 240 of Fig. 2, redis, according to the mapping relations of set up key to Section Point cluster, analyzes redis database snapshot and is recombined into by redis database snapshot in the database file corresponding respectively with the multiple Section Points in Section Point cluster.
In concrete examples more of the present invention, when L1 reduces to 0, represent that the rdb snapshot (number is 2 in this example) of all nodes in cluster A is all received by redis bridge.Now thread t1 continues to run, and reads and resolves two rdb files (such as, travel through key/value), according to the mapping relations of new key to B clustered node, key/value is written to 3 new rdb file b1.rdb, b2.rdb, b3.rdb.Compared with the traversal of carrying out with the command interface directly calling redis in prior art, spendable traversal is called without the need to using tcp (transmission control protocol) protocol stack herein, but directly data are operated, speed is faster, and efficiency is higher.
After rdb snapshot reconfigures, in the step 250 of Fig. 2, redis bridge sends the database file corresponding with it and appended document respectively to Section Point.
Such as, in examples more of the present invention, after thread t1 writes 3 rdb files, send file settling signal toward q1, q2, q3 respectively, comprise corresponding rdb filename.
After thread b1, b2, b3 receive the signal from q1, q2, q3, by the rdb file of correspondence with each self-sustaining connection c1, c2, the c3 with B clustered node of sendfile mechanism write.
Thread b1 reads b1.aof from file header, all write operations is sent to according to this connection c1 with B clustered node, and B cluster corresponding node performs write operation accordingly.Until thread b1 reads the end-of-file of b1.aof, record current offset1.
Thread b2, b3, with reading corresponding b2.aof, b3.aof described in aforesaid operations, send it to corresponding B clustered node, and record last offset2, offset3.
So far, in the step 260 of Fig. 2, by configuration management system, the clustered node topology (interstitial content and numbering etc.) of Section Point cluster is sent to client, read-write operation can be carried out according to the clustered node topology of Section Point cluster to make client.
Particularly, in step 250 to after the clustered node (Section Point) in Section Point cluster sends the database file corresponding with it and appended document respectively, redis bridge, by notifying configuration management system with the communication link (not shown in figure 1) of configuration management system, carries out above-mentioned clustered node topology to indicate configuration management system and sends.
Clustered node reducing and expansion method according to the embodiment of the present invention can terminate in step 260 place.But in some cases, during sending corresponding database file and appended document to cluster B, client may perform some write operations, and be stored in the offset mark in corresponding .aof file according to embodiments of the invention after.Therefore, according to other embodiments of the present invention, some providing this situation are improved.
Particularly, when number of nodes new for B cluster and numbering (clustered node topology) are sent to client by configuration management system, can forbid that client is write simultaneously.
Then thread b1 reads b1.aof from offset1, until end-of-file, all write operations is sent to according to this connection c1, B cluster corresponding node and performs according to this.Thread b2, b3 in like manner perform.
After the operation of thread b1, b2 and b3 all terminates, configuration management system can notify client according to B clustered node topology to carry out read-write operation, that is, restart the write operation of client.
Because in each .aof file, after offset mark, the write operation of record can not be too many, therefore above-mentioned additional operations can not bring delay.This minimize the time window that data are inconsistent, ensure that data final consistency and business write the balance of pausing and affecting
According to said method, can redis storage organization do not revised with under the prerequisite increasing new function, utilize a middleware realizing the replication protocol of redis, by fractionation and the restructuring of data in reproduction process, reach the increase (horizontal dilatation) of back end or reduce (horizontal capacity reducing).
Although it should be noted that convenience in order to describe and be easy to reader understanding and use different Reference numerals to identify each step in fig. 2, the size of Reference numeral does not represent the exact sequence performing correlation step.Such as, the step 230 generating appended document and record write operation can execution after the step 240 analyzed redis database snapshot and reconfigure redis database snapshot, or performs with it simultaneously.Also such as, for each node in first node cluster and Section Point cluster connects and the order of thread can be undertaken etc. by being different from sequencing as herein described.The present invention not by these order limit.
In other embodiments of the present invention, additionally provide clustered node reducing and expansion equipment, such as, redis bridge in Fig. 1.Fig. 3 shows the brief configuration block diagram of this clustered node reducing and expansion equipment.
As shown in Figure 3, this equipment comprises: receiving element 310, for receiving the redis database snapshot of the plurality of first node parallel generation from multiple first nodes of first node cluster, and from the write operation that the plurality of first node reception client is carried out after the transmission of redis database snapshot; Appended document generation unit 320, for generating the appended document corresponding respectively with multiple Section Points of Section Point cluster, and according to the mapping relations of set up key to this Section Point cluster, the described write operation received by receiving element 310 is recorded in appended document respectively; Recombination unit 330, for according to the mapping relations of set up key to Section Point cluster, analyze the redis database snapshot that receiving element 310 receives and also redis database snapshot is recombined in the database file corresponding respectively with the plurality of Section Point of Section Point cluster; And transmitting element 340, for sending the database file corresponding with it and appended document respectively to the plurality of Section Point, and the clustered node topology of Section Point cluster is sent to client by the configuration management system being used to indicate configuration client, can carry out read-write operation to make client according to the clustered node topology of Section Point cluster.
Equipment in Fig. 3 can also comprise write operation and forbid unit 350, for when the clustered node topology of Section Point cluster is sent to client by the configuration management system be configured client, forbids that client carries out write operation by configuration management system.Now, transmitting element 340 also sends the write operation of the client recorded after sending the database file corresponding with it and appended document respectively to the plurality of Section Point in appended document respectively to multiple Section Point.
Equipment in Fig. 3 can also comprise: connection establishment unit 360, is connected for setting up first separately with multiple first nodes of first node cluster, and sets up second separately with multiple Section Points of Section Point cluster and be connected; And thread sets up unit 370, set up the first thread separately for the multiple first nodes for first node cluster, set up the thread being used for rdb conversion, and set up the second thread separately for multiple Section Points of Section Point cluster.
In some instances, as mentioned above, the thread for rdb conversion can comprise count synchronization device.The initial value of count synchronization device equals the number of first node cluster interior joint, and each first thread receives redis database snapshot from the node first node cluster, and count synchronization device successively decreases one.When count synchronization device equals zero, the described thread for rdb conversion can be analyzed redis database snapshot and is recombined into by redis database snapshot in the database file corresponding respectively with the plurality of Section Point of Section Point cluster.
The equipment of Fig. 3 can also comprise queue and set up unit 380, for setting up the isochronous queue of number identical with multiple Section Point.In some instances, after the thread changed for rdb completes and reconfigures, it is informed to the second thread via isochronous queue.Now, transmitting element 340 also utilizes described second thread to send the database file corresponding respectively with it to described multiple Section Point.
In some instances, as mentioned above, the first thread, the second thread and the thread for rdb conversion enter blocked state when not using.
It should be noted that; the clustered node reducing and expansion equipment described in figure 3 is only the figure done to make those skilled in the art more clearly understand the present invention; wherein eliminate some to understanding the unnecessary modules/components of the present invention, protection scope of the present invention should not limit by the detail of these accompanying drawings.Such as, more modules/components can be comprised, as display, Operation and Maintenance interface, IO interface etc. in actual equipment.Again such as, said units 320,330,350-380 etc. can realize by the one or more processor/modules in actual computer.This module can be software, hardware, firmware or other modes well known to those skilled in the art.The present invention does not limit these.
Description is above only for realizing embodiments of the present invention; it should be appreciated by those skilled in the art; the any modification or partial replacement do not departed from the scope of the present invention; the scope that all should belong to claim of the present invention to limit; therefore, protection scope of the present invention should be as the criterion with the protection domain of claims.
Claims (13)
1., for a method for clustered node reducing and expansion, comprising:
The redis database snapshot of described multiple first node parallel generation is received from multiple first nodes of first node cluster;
From the write operation that described multiple first node reception client is carried out after the transmission of described redis database snapshot;
Generate the appended document corresponding respectively with multiple Section Points of Section Point cluster, and according to the mapping relations of set up key to described Section Point cluster, described write operation is recorded in respectively in described appended document;
According to the mapping relations of described set up key to described Section Point cluster, analyze described redis database snapshot and described redis database snapshot be recombined in the database file corresponding respectively with described multiple Section Points of described Section Point cluster;
The database file corresponding with it and appended document is sent respectively to described multiple Section Point; And
The clustered node topology of described Section Point cluster is sent to described client by the configuration management system of the described client of instruction configuration, can carry out read-write operation to make described client according to the clustered node topology of described Section Point cluster.
2. method according to claim 1, when the clustered node topology of described Section Point cluster is sent to described client, also comprises:
Forbid that described client carries out write operation by the configuration management system be configured described client; And
The write operation of the described client recorded after sending the database file corresponding with it and appended document respectively to described multiple Section Point in described appended document is sent respectively to described multiple Section Point.
3. method according to claim 1, also comprises:
Set up second separately with described multiple Section Point of described Section Point cluster to be connected;
For described multiple Section Point of described Section Point cluster sets up the second thread separately;
Set up the thread being used for rdb conversion;
For described multiple first node of described first node cluster sets up the first thread separately; And
Set up first separately with described multiple first node of described first node cluster to be connected.
4. method according to claim 3, wherein, the described thread for rdb conversion comprises count synchronization device, and the initial value of described count synchronization device equals the number of described first node cluster interior joint, and described method also comprises:
Described first thread receives described redis database snapshot from the node described first node cluster at every turn, and described count synchronization device successively decreases one; And
When described count synchronization device equals zero, described redis database snapshot is also recombined in the database file corresponding respectively with the described multiple Section Points of described Section Point cluster by redis database snapshot described in the described Thread Analysis for rdb conversion.
5. method according to claim 4, also comprises:
Set up the isochronous queue with the identical number of described multiple Section Point;
After reconfiguring described in the described thread for rdb conversion completes, it is informed to described second thread via described isochronous queue; And
Described second thread sends the database file corresponding respectively with it to described multiple Section Point.
6. the method according to any one of claim 4 to 6, wherein, described first thread, described second thread and the described thread for rdb conversion enter blocked state when not using.
7., for an equipment for clustered node reducing and expansion, comprising:
Receiving element, for receiving the redis database snapshot of described multiple first node parallel generation from multiple first nodes of first node cluster, and from the write operation that described multiple first node reception client is carried out after the transmission of described redis database snapshot;
Appended document generation unit, for generating the appended document corresponding respectively with multiple Section Points of Section Point cluster, and according to the mapping relations of set up key to described Section Point cluster, the described write operation received by described receiving element is recorded in described appended document respectively;
Recombination unit, for according to the mapping relations of described set up key to described Section Point cluster, analyze described redis database snapshot that described receiving element receives and described redis database snapshot be recombined in the database file corresponding respectively with described multiple Section Points of described Section Point cluster; And
Transmitting element, for sending the database file corresponding with it and appended document respectively to described multiple Section Point, and be used to indicate the configuration management system configuring described client the clustered node topology of described Section Point cluster is sent to described client, read-write operation can be carried out according to the clustered node topology of described Section Point cluster to make described client.
8. equipment according to claim 7, also comprises:
Write operation forbids unit, for when the clustered node topology of described Section Point cluster is sent to described client by the configuration management system be configured described client, forbids that described client carries out write operation by described configuration management system;
Wherein, described transmitting element also sends the write operation of the described client recorded after sending the database file corresponding with it and appended document respectively to described multiple Section Point in described appended document respectively to described multiple Section Point.
9. equipment according to claim 7, also comprises:
Connection establishment unit, is connected for setting up first separately with described multiple first node of described first node cluster, and sets up second separately with described multiple Section Point of described Section Point cluster and be connected; And
Thread sets up unit, for setting up the first thread separately for described multiple first node of described first node cluster, sets up the thread being used for rdb conversion, and sets up the second thread separately for described multiple Section Point of described Section Point cluster.
10. equipment according to claim 9, wherein, the described thread for rdb conversion comprises count synchronization device, the initial value of described count synchronization device equals the number of described first node cluster interior joint, described first thread receives described redis database snapshot from the node described first node cluster at every turn, described count synchronization device successively decreases one, when described count synchronization device equals zero, described redis database snapshot is also recombined in the database file corresponding respectively with the described multiple Section Points of described Section Point cluster by redis database snapshot described in the described Thread Analysis for rdb conversion.
11. equipment according to claim 10, also comprise:
Unit is set up in queue, for setting up the isochronous queue with the identical number of described multiple Section Point,
Wherein, after reconfiguring described in the described thread for rdb conversion completes, it is informed to described second thread via described isochronous queue; And described transmitting element also utilizes described second thread to send the database file corresponding respectively with it to described multiple Section Point.
12. equipment according to any one of claim 7 to 11, wherein, described first thread, described second thread and the described thread for rdb conversion enter blocked state when not using.
13. 1 kinds, for the system of clustered node reducing and expansion, comprising:
Client;
Comprise the first node cluster of multiple first node;
Comprise the Section Point cluster of multiple Section Point;
Equipment according to any one of claim 7 to 11; And
For configuring the configuration management system of described client, the clustered node topology of described Section Point cluster is sent to described client by described configuration management system, can carry out read-write operation to make described client according to the clustered node topology of described Section Point cluster.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510505399.3A CN105260376B (en) | 2015-08-17 | 2015-08-17 | Method, apparatus and system for clustered node reducing and expansion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510505399.3A CN105260376B (en) | 2015-08-17 | 2015-08-17 | Method, apparatus and system for clustered node reducing and expansion |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105260376A true CN105260376A (en) | 2016-01-20 |
| CN105260376B CN105260376B (en) | 2018-08-14 |
Family
ID=55100069
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510505399.3A Active CN105260376B (en) | 2015-08-17 | 2015-08-17 | Method, apparatus and system for clustered node reducing and expansion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105260376B (en) |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105808643A (en) * | 2016-02-25 | 2016-07-27 | 浪潮通信信息***有限公司 | Method for refreshing Redis memory database |
| CN106843755A (en) * | 2017-01-04 | 2017-06-13 | 北京百度网讯科技有限公司 | For the data balancing method and device of server cluster |
| CN107330035A (en) * | 2017-06-26 | 2017-11-07 | 努比亚技术有限公司 | Operation Log synchronous method, mobile terminal and computer-readable recording medium in a kind of database |
| CN107463612A (en) * | 2017-06-29 | 2017-12-12 | 北京北信源软件股份有限公司 | High concurrent REDIS database operation methods and system |
| CN107741963A (en) * | 2017-09-30 | 2018-02-27 | 北京奇虎科技有限公司 | Across company-data merging method and device, electronic equipment, computer-readable storage medium |
| CN107919977A (en) * | 2016-10-11 | 2018-04-17 | 阿里巴巴集团控股有限公司 | A kind of on-line rapid estimation of the distributed consensus system based on Paxos agreements, the method and apparatus of online capacity reducing |
| CN107943589A (en) * | 2017-11-29 | 2018-04-20 | 苏宁云商集团股份有限公司 | The management method and device of a kind of data buffer storage |
| CN108073414A (en) * | 2016-11-15 | 2018-05-25 | 北京亿阳信通科技有限公司 | It is a kind of that multi-thread concurrent is asked to merge the implementation method that batch submits simultaneously Distribution Results based on Jedis |
| WO2019033949A1 (en) * | 2017-08-18 | 2019-02-21 | 阿里巴巴集团控股有限公司 | Data migration method, apparatus and device |
| CN109818767A (en) * | 2017-11-21 | 2019-05-28 | 网宿科技股份有限公司 | A kind of method, apparatus and storage medium of the adjustment of Redis cluster capacity |
| CN110333944A (en) * | 2019-04-19 | 2019-10-15 | 中国联合网络通信集团有限公司 | Call bill data method for processing business and equipment |
| CN111309805A (en) * | 2019-12-13 | 2020-06-19 | 华为技术有限公司 | Data reading and writing method and device for database |
| CN112632189A (en) * | 2020-12-26 | 2021-04-09 | 中国农业银行股份有限公司 | Data synchronization system and method |
| WO2021168618A1 (en) * | 2020-02-24 | 2021-09-02 | 深圳市欢太科技有限公司 | Redis cluster expansion method and apparatus, electronic device, and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102148850A (en) * | 2010-08-09 | 2011-08-10 | 华为软件技术有限公司 | Cluster system and service processing method thereof |
| CN102521297A (en) * | 2011-11-30 | 2012-06-27 | 北京人大金仓信息技术股份有限公司 | Method for achieving system dynamic expansion in shared-nothing database cluster |
| CN103034664A (en) * | 2011-10-10 | 2013-04-10 | 上海盛霄云计算技术有限公司 | Method, system and device for controlling data migration of database |
| CN103078927A (en) * | 2012-12-28 | 2013-05-01 | 合一网络技术(北京)有限公司 | Key-value data distributed caching system and method thereof |
| CN103634375A (en) * | 2013-11-07 | 2014-03-12 | 华为技术有限公司 | Method, device and equipment for cluster node expansion |
| CN104407807A (en) * | 2014-10-24 | 2015-03-11 | 华中科技大学 | Storage and expansion method aiming at RS coding storage cluster |
| CN104484469A (en) * | 2014-12-31 | 2015-04-01 | 天津南大通用数据技术股份有限公司 | Expansion method for supporting non-halt of multi-hash map database cluster system |
-
2015
- 2015-08-17 CN CN201510505399.3A patent/CN105260376B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102148850A (en) * | 2010-08-09 | 2011-08-10 | 华为软件技术有限公司 | Cluster system and service processing method thereof |
| CN103034664A (en) * | 2011-10-10 | 2013-04-10 | 上海盛霄云计算技术有限公司 | Method, system and device for controlling data migration of database |
| CN102521297A (en) * | 2011-11-30 | 2012-06-27 | 北京人大金仓信息技术股份有限公司 | Method for achieving system dynamic expansion in shared-nothing database cluster |
| CN103078927A (en) * | 2012-12-28 | 2013-05-01 | 合一网络技术(北京)有限公司 | Key-value data distributed caching system and method thereof |
| CN103634375A (en) * | 2013-11-07 | 2014-03-12 | 华为技术有限公司 | Method, device and equipment for cluster node expansion |
| CN104407807A (en) * | 2014-10-24 | 2015-03-11 | 华中科技大学 | Storage and expansion method aiming at RS coding storage cluster |
| CN104484469A (en) * | 2014-12-31 | 2015-04-01 | 天津南大通用数据技术股份有限公司 | Expansion method for supporting non-halt of multi-hash map database cluster system |
Cited By (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105808643A (en) * | 2016-02-25 | 2016-07-27 | 浪潮通信信息***有限公司 | Method for refreshing Redis memory database |
| US11271814B2 (en) | 2016-10-11 | 2022-03-08 | Alibaba Group Holding Limited | Online capacity-expanding and online capacity-reducing methods and apparatuses for distributed consensus system |
| CN107919977A (en) * | 2016-10-11 | 2018-04-17 | 阿里巴巴集团控股有限公司 | A kind of on-line rapid estimation of the distributed consensus system based on Paxos agreements, the method and apparatus of online capacity reducing |
| CN108073414A (en) * | 2016-11-15 | 2018-05-25 | 北京亿阳信通科技有限公司 | It is a kind of that multi-thread concurrent is asked to merge the implementation method that batch submits simultaneously Distribution Results based on Jedis |
| CN108073414B (en) * | 2016-11-15 | 2020-07-28 | 北京亿阳信通科技有限公司 | Implementation method for merging multithreading concurrent requests and submitting and distributing results in batches based on Jedis |
| CN106843755A (en) * | 2017-01-04 | 2017-06-13 | 北京百度网讯科技有限公司 | For the data balancing method and device of server cluster |
| CN106843755B (en) * | 2017-01-04 | 2019-10-11 | 北京百度网讯科技有限公司 | Data balancing method and device for server cluster |
| CN107330035B (en) * | 2017-06-26 | 2020-08-07 | 宁波图灵奇点智能科技有限公司 | Operation log synchronization method in database, mobile terminal and computer readable storage medium |
| CN107330035A (en) * | 2017-06-26 | 2017-11-07 | 努比亚技术有限公司 | Operation Log synchronous method, mobile terminal and computer-readable recording medium in a kind of database |
| CN107463612B (en) * | 2017-06-29 | 2019-10-01 | 北京北信源软件股份有限公司 | High concurrent REDIS database operation method and system |
| CN107463612A (en) * | 2017-06-29 | 2017-12-12 | 北京北信源软件股份有限公司 | High concurrent REDIS database operation methods and system |
| WO2019033949A1 (en) * | 2017-08-18 | 2019-02-21 | 阿里巴巴集团控股有限公司 | Data migration method, apparatus and device |
| CN107741963A (en) * | 2017-09-30 | 2018-02-27 | 北京奇虎科技有限公司 | Across company-data merging method and device, electronic equipment, computer-readable storage medium |
| CN109818767B (en) * | 2017-11-21 | 2021-08-03 | 网宿科技股份有限公司 | Method and device for adjusting Redis cluster capacity and storage medium |
| CN109818767A (en) * | 2017-11-21 | 2019-05-28 | 网宿科技股份有限公司 | A kind of method, apparatus and storage medium of the adjustment of Redis cluster capacity |
| CN107943589A (en) * | 2017-11-29 | 2018-04-20 | 苏宁云商集团股份有限公司 | The management method and device of a kind of data buffer storage |
| CN110333944A (en) * | 2019-04-19 | 2019-10-15 | 中国联合网络通信集团有限公司 | Call bill data method for processing business and equipment |
| CN111309805A (en) * | 2019-12-13 | 2020-06-19 | 华为技术有限公司 | Data reading and writing method and device for database |
| CN111309805B (en) * | 2019-12-13 | 2023-10-20 | 华为技术有限公司 | Data reading and writing method and device for database |
| US11868333B2 (en) | 2019-12-13 | 2024-01-09 | Huawei Technologies Co., Ltd. | Data read/write method and apparatus for database |
| WO2021168618A1 (en) * | 2020-02-24 | 2021-09-02 | 深圳市欢太科技有限公司 | Redis cluster expansion method and apparatus, electronic device, and storage medium |
| CN112632189A (en) * | 2020-12-26 | 2021-04-09 | 中国农业银行股份有限公司 | Data synchronization system and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105260376B (en) | 2018-08-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105260376A (en) | Method, equipment and system used for cluster node contraction and expansion | |
| US10579610B2 (en) | Replicated database startup for common database storage | |
| US11860741B2 (en) | Continuous data protection | |
| US12013764B2 (en) | Past-state backup generator and interface for database systems | |
| CN110362381A (en) | HDFS cluster High Availabitity dispositions method, system, equipment and storage medium | |
| CN103905537A (en) | System for managing industry real-time data storage in distributed environment | |
| US11262918B1 (en) | Data storage system with uneven drive wear reduction | |
| US9983823B1 (en) | Pre-forking replicas for efficient scaling of a distribued data storage system | |
| US11720593B2 (en) | Managing identifiers for multinodal master systems of unknown or changing size | |
| CN102983996A (en) | Dynamic allocation method and system for high-availability cluster resource management | |
| CN104468274A (en) | Cluster monitor and management method and system | |
| JP2021524104A (en) | Master / Standby Container System Switching | |
| CN102663017A (en) | Implementation system and implementation method for enhancing availability of MySQL database | |
| CN110442601A (en) | A kind of method and apparatus that Openstack mirror image data accelerates parallel | |
| CN112069152B (en) | Database cluster upgrading method, device, equipment and storage medium | |
| JP2017503422A (en) | Network element data access method and apparatus, and network management system | |
| CN110555064A (en) | data service system and method for insurance business | |
| JP2020191100A (en) | System, control method therefor, and program | |
| CN111352916B (en) | Data storage method, system and storage medium based on NAS storage system | |
| CN113987089A (en) | System cascading method, data processing method and device | |
| CN112527760A (en) | Data storage method, device, server and medium | |
| US10712959B2 (en) | Method, device and computer program product for storing data | |
| CN105760431A (en) | Method and device for transferring file blocks | |
| JP5033095B2 (en) | Storage management intermediary server and control method thereof | |
| CN109947451A (en) | A kind of cluster application file updating method, system, medium and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |