CN108694209B - Distributed index method based on object and client - Google Patents
Distributed index method based on object and client Download PDFInfo
- Publication number
- CN108694209B CN108694209B CN201710233466.XA CN201710233466A CN108694209B CN 108694209 B CN108694209 B CN 108694209B CN 201710233466 A CN201710233466 A CN 201710233466A CN 108694209 B CN108694209 B CN 108694209B
- Authority
- CN
- China
- Prior art keywords
- node
- interface
- pointer
- osd
- key
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a distributed index method and a client based on an object, wherein the method comprises the following steps: the client acquires the content and the name of an object to be stored; the client side carries out normalization processing on the name of the object to be stored to obtain a first normalization result; the client acquires the key of the idle OSD in the OSD cluster through the interface, and stores the first normalization result, the name, the content and the key of the idle OSD in the idle OSD through the interface, so as to generate a new node; the client queries a rear node, which stores a second normalization result larger than the first normalization result, and a front node, which stores a third normalization result smaller than the first normalization result, in the distributed index through an interface; and the client constructs a pointer at the new node, sets the pointer of the new node as the key of the back node, and sets the pointer of the front node as the key of the new node. Keys of the objects are scattered in the distributed index through the pointers, and the objects are located through querying the distributed index, so that the difficulty of deployment and maintenance can be reduced.
Description
Technical Field
The present application relates to the field of information technologies, and in particular, to a distributed index method and a client based on an object.
Background
The Object Storage technology is widely applied at the present stage, in the Object Storage technology, a Storage system sets an OSD (Object-based Storage Device) cluster, the OSD cluster includes a plurality of OSDs, each OSD has a fixed Storage space, objects are stored in the OSDs of the OSD cluster, each OSD corresponds to a key, and a client can find the corresponding OSD through the key and read the Object from the OSD.
In the storage system in the prior art, the storage system further sets an MDS (Metadata Server), the MDS stores a relationship between a key and a network address of the OSD corresponding to the key, the OSD cluster is provided with an interface, and the client queries the network address of the OSD corresponding to the key from the MDS through the interface after acquiring the key, so that an object in the OSD can be accessed according to the network address.
In the prior art, since the storage space of the OSD is fixed, when the object to be stored is a large object whose required space is larger than the storage space of the OSD, the OSD cannot store the object.
Therefore, the storage system in the prior art needs to further set an MDS for the object, store a correspondence between an object name and a key of a corresponding OSD for the MDS of the object, and when the client needs to read the object, the client needs to query the object according to the object name, so as to obtain OSDs in which the object is stored, and read the object from the write OSDs respectively.
However, in the prior art, the client needs to query the MDS to obtain the network address of the OSD corresponding to the key and also needs to query the MDS for the object to obtain the keys of the OSDs for the object, and when the object fails, the client cannot access the object, so that the existence of the MDS for the object increases the difficulty of deployment and maintenance.
Disclosure of Invention
In order to solve the problems in the prior art, embodiments of the present invention provide an object-based distributed index method and a client, where keys of objects are scattered in a distributed index by pointers, and the objects can be located by querying the distributed index, so that difficulty in deployment and maintenance can be reduced.
In a first aspect, an embodiment of the present invention provides an Object-based distributed index method, where the method is applied to a client, and the client interacts with an OSD (Object-based Storage Device) cluster through an interface, where the OSD cluster includes a plurality of OSDs, the client selects the OSDs as nodes through the interface to construct a distributed index, and in each node of the distributed index, a name and a normalization result of the client, a key (cluster) of the client in the OSD cluster, and a pointer are stored, where the normalization result is generated according to the name, and the pointer stores a key of a next node in the distributed index, and the method includes:
the client acquires the content and the name of an object to be stored;
the client side carries out normalization processing on the name of the object to be stored to obtain a first normalization result;
the client acquires the key of the idle OSD in the OSD cluster through the interface, and stores the first normalization result, the name, the content and the key of the idle OSD in the idle OSD through the interface, so as to generate a new node;
the client queries a rear node, which stores a second normalization result larger than the first normalization result, and a front node, which stores a third normalization result smaller than the first normalization result, in the distributed index through an interface;
and the client constructs a pointer at the new node through the interface, sets the pointer of the new node as the key of the back node, and sets the pointer of the front node as the key of the new node.
In summary, in the embodiment of the present invention, the client applies for the OSD from the OSD cluster through the interface, and since the keys of the objects are scattered in the distributed index through the pointer, the objects can be located by querying the distributed index, and the difficulty of deployment and maintenance can be reduced.
In an implementation manner of the embodiment of the present invention, the distributed index includes an entry node, a pointer of the entry node stores a key of a node whose normalization result is the minimum value in the distributed index, and the step of querying, by the client through the interface, a back node in the distributed index, where a second normalization result larger than the first normalization result is stored, and a front node in the distributed index, where a third normalization result smaller than the first normalization result is stored, specifically includes:
and the client queries the normalization results stored by the nodes in the distributed index from the entry node through the interface according to the pointer stored by each node in the distributed index until a back node in which a second normalization result larger than the first normalization result is stored and a front node in which a third normalization result smaller than the first normalization result is stored in the distributed index are queried.
In another implementation manner of the embodiment of the present invention, the step of constructing, by the client, a pointer at the new node through the interface, setting the pointer of the new node as the key of the back node, and setting the pointer of the front node as the key of the new node specifically includes:
the client constructs a pointer in the new node through an interface, and sets the pointer of the new node as a key of the back node;
the client constructs a log at the previous node through an interface, and stores the key of the new node in the log;
when the pointer of the front node is set to the key stored by the new node through the interface, the client reads the key of the new node from the log of the front node and sets the pointer of the front node to the read key of the new node;
the client zeroes the log through the interface.
In another implementation manner of the embodiment of the present invention, the client acquires the key of the idle OSD in the OSD cluster through the interface, and stores the first normalization result, the name, the content, and the key of the idle OSD in the idle OSD through the interface, so that generating the new node specifically includes:
when the client judges that the size of the object to be stored is larger than the storage space of the OSD, dividing the object to be stored into a preset number of parts, acquiring keys of a preset number of idle OSD in the OSD cluster through an interface, and respectively storing the parts of the object to be stored in the idle OSD and the preset number of idle OSD;
and selecting any OSD from the preset number of idle OSD as a new node, and storing the first normalization result, the name, the content and the key of the preset number of idle OSD in the new node through an interface.
In another implementation manner of the embodiment of the present invention, the distributed index is a skip list, and the pointer includes a plurality of pointers, and each pointer points to a different layer of the skip list.
In another implementation manner of the embodiment of the present invention, the method further includes:
the client acquires the name of an object to be deleted;
the client side carries out normalization processing on the name to obtain a fourth normalization result;
the client queries the nodes to be deleted in the distributed index, which store the fourth normalization result, through the interface;
the client sets the pointer of the front node of the node to be deleted as the key of the rear node of the node to be deleted through the interface;
and the client informs the OSD cluster of setting the nodes to be deleted as idle OSD through the interface.
In another implementation manner of the embodiment of the present invention, the step of setting, by the client, the pointer of the front node of the node to be deleted as the key of the back node of the node to be deleted through the interface specifically includes:
the client constructs a log at a front node through an interface, and stores keys stored by the nodes to be deleted in the log;
when the client fails to set the pointer of the front node as the key of the rear node through the interface, reading the key of the node to be deleted from the log of the front node, reading the key of the rear node stored by the pointer of the node to be deleted through the interface according to the key of the node to be deleted, and setting the pointer of the front node as the read key of the rear node;
the client zeroes the log through the interface.
In a second aspect, an embodiment of the present invention further provides a client, where the client interacts with an OSD cluster through an interface, the OSD cluster includes a plurality of OSDs, and the client selects the OSDs as nodes through the interface to construct a distributed index, where in each node of the distributed index, a name and a normalization result of the client, a key of the client in the OSD cluster, and a pointer are stored, where the normalization result is generated according to the name, and the pointer stores a key of a next node in the distributed index, and the client includes:
the device comprises a to-be-stored object acquisition module, a storage module and a storage module, wherein the to-be-stored object acquisition module is used for acquiring the content and the name of an object to be stored;
the normalization module is used for performing normalization processing on the name of the object to be stored to obtain a first normalization result;
the new node generation module is used for acquiring keys of idle OSD in the OSD cluster through the interface, and storing the first normalization result, the name, the content and the keys of the idle OSD in the idle OSD through the interface so as to generate a new node;
the query module is used for querying a rear node in the distributed index, which stores a second normalization result larger than the first normalization result, and a front node in the distributed index, which stores a third normalization result smaller than the first normalization result, through the interface;
and the pointer modification module is used for constructing a pointer in the new node through the interface, setting the pointer of the new node as the key of the back node, and setting the pointer of the front node as the key of the new node.
In another implementation manner of the embodiment of the present invention, the distributed index includes entry nodes, the pointers of the entry nodes store keys of the nodes whose normalization results are the minimum values in the distributed index, the client further includes an entry recording module, the entry recording module is configured to record the keys of the entry nodes,
and the query module is specifically used for querying the normalization results stored by the nodes in the linked list according to the pointer stored by each node in the distributed index from the entry node through the interface until a rear node in the distributed index, which stores a second normalization result larger than the first normalization result, and a front node in the distributed index, which stores a third normalization result smaller than the first normalization result, are queried.
In another implementation manner of the embodiment of the present invention, the pointer modification module is specifically configured to:
constructing a pointer at the new node through an interface, and setting the pointer of the new node as a key of the back node;
constructing a log at a previous node through an interface, and storing a key of a new node in the log;
reading the key of the new node from the log of the previous node and setting the pointer of the previous node as the read key of the new node when the pointer of the previous node is set as the key stored by the new node through the interface;
the log is zeroed through the interface.
In another implementation manner of the embodiment of the present invention, the new node generation module is specifically configured to:
when the size of the object to be stored is judged to be larger than the storage space of the OSD, dividing the object to be stored into a preset number of parts, acquiring keys of a preset number of idle OSD in the OSD cluster through an interface, and respectively storing the parts of the object to be stored in the idle OSD and the preset number of idle OSD;
and selecting any OSD from the preset number of idle OSD as a new node, and storing the first normalization result, the name, the content and the key of the preset number of idle OSD in the new node through an interface.
In another implementation manner of the embodiment of the present invention, the distributed index is a skip list, and the pointer includes a plurality of pointers, and each pointer points to a different layer of the skip list.
In another implementation manner of the embodiment of the present invention, the client further includes a notification module,
the name acquisition module is also used for acquiring the name of the object to be deleted;
the normalization module is also used for carrying out normalization processing on the name to obtain a fourth normalization result;
the query module is further used for querying the nodes to be deleted, which store the fourth normalization result, in the distributed index through the interface;
the pointer modification module is also used for setting a pointer of a front node of the node to be deleted as a key of a rear node of the node to be deleted;
and the notification module is used for notifying the OSD cluster to set the nodes to be deleted as idle OSD through the interface.
In another implementation manner of the embodiment of the present invention, the pointer modification module is specifically configured to:
constructing a log through an interface at a front node, and storing keys stored by nodes to be deleted in the log;
when the pointer of the front node is set to be the key of the rear node through the interface, reading the key of the node to be deleted from the log of the front node, reading the key of the rear node stored by the pointer of the node to be deleted through the interface according to the key of the node to be deleted, and setting the pointer of the front node as the read key of the rear node;
the log is zeroed through the interface.
In a third aspect, an embodiment of the present invention provides a client, including a processor, a memory, and a bus, where the processor and the memory are respectively connected to the bus, and the memory stores program instructions, where:
the processor executes the first program instructions to perform the steps of: acquiring the content and the name of an object to be stored;
the processor executes the second program instructions to perform the steps of: carrying out normalization processing on the name of the object to be stored to obtain a first normalization result;
the processor executes the third program instructions to perform the steps of: acquiring keys of idle OSD in the OSD cluster through an interface, and storing the first normalization result, the name, the content and the keys of the idle OSD in the idle OSD through the interface so as to generate a new node;
the processor executes the fourth program instruction to perform the steps of: querying a back node storing a second normalization result larger than the first normalization result and a front node storing a third normalization result smaller than the first normalization result in the distributed index through an interface;
the processor executes the fifth program instruction to perform the steps of: and constructing a pointer at the new node through the interface, setting the pointer of the new node as the key of the back node, and setting the pointer of the front node as the key of the new node.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments of the present invention will be briefly described below, and it is obvious that the drawings described below are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a schematic structural diagram of a storage system according to an embodiment of the invention;
FIG. 2 is a flow diagram of a method for object-based distributed indexing, according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a distributed index according to an embodiment of the present invention;
FIG. 4 is another schematic diagram of a distributed index according to an embodiment of the present invention;
FIG. 5 is another schematic diagram of a storage system according to an embodiment of the invention;
FIG. 6 is a flow diagram of a method for object-based distributed indexing, according to an embodiment of the present invention;
FIG. 7 is another schematic diagram of a distributed index according to an embodiment of the present invention;
FIG. 8 is another schematic diagram of a distributed index according to an embodiment of the present invention;
fig. 9 is a schematic device structure diagram of a client according to an embodiment of the present invention;
fig. 10 is a schematic structural diagram of another device of the client according to an embodiment of the present invention.
Detailed Description
Referring to fig. 1, fig. 1 is a schematic structural diagram of a storage system according to an embodiment of the present invention, and as shown in fig. 1, the storage system according to the embodiment of the present invention includes a client and an OSD cluster.
For example, the number of the clients may be multiple, as shown in fig. 1, the clients include clients 101 and 102 … … n, and in other examples, the number of the clients may also be 1, which is not limited in this embodiment of the present invention.
The OSD cluster 20 includes an interface (not shown) and a plurality of OSDs, each OSD corresponds to a fixed-size storage space of the physical disk, and the OSD cluster 20 divides the storage space of the physical disk into a plurality of OSDs to store data.
The OSD cluster 20 includes MDSs, each OSD is correspondingly assigned with a key, and the MDS records the correspondence between the key and the storage space, for example, the key can query the MDS for a specific storage space of a specific physical disk mounted on a physical host where the OSD corresponding to the key is located at a specific network address.
The clients 101, 102 … … n access the OSD cluster 20 through the distributed index 201, and the distributed index 201 is set in the OSD of the OSD cluster 20.
It should be noted that, in the embodiment of the present invention, the OSD cluster 20 provides an interface, and the client only needs to access the interface by using the key as an input parameter, so that the client can access the OSD corresponding to the key from the interface.
Specifically, the client accesses the interface by taking the key and the read command as input parameters, namely, the data stored in the OSD can be directly read from the interface; the client side takes the key, the write-in command and the object to be stored as input parameter access interfaces, and can write the object to be stored into the storage space corresponding to the key; further, the client may apply for the idle OSD from the OSD cluster 20 through the interface, for example, the client may input an OSD application command to the interface, and may obtain a key of the idle OSD from the interface; and the client may notify the OSD cluster 20 to recycle the OSD storing the object to be stored through the interface, for example, after the client inputs a key to the interface and cancels the OSD command, the OSD cluster 20 recycles the OSD, and when the subsequent client inputs the key to the interface to try to access the OSD corresponding to the key, an access failure notification sent by the interface is received.
Therefore, the OSD cluster 20 shields the internal working process for the client, and the client only needs to apply for OSD from the OSD cluster 20 through the interface, obtain the key at the interface, and access the OSD at the interface according to the key.
With continuing reference to fig. 1, in fig. 1, it is assumed that the client 101 requests the OSD2, the OSD9, the OSD3, the OSD5, the OSD7, the OSD1, and the OSD8 from the OSD cluster 20, each of the OSDs storing corresponding data, the distributed index 201 includes an OSD2, an OSD9, an OSD3, an OSD5, an OSD7, an OSD1, and an OSD8, in each of the OSDs, a name of itself and a normalization result are stored, a key of itself at the OSD cluster 20 and a pointer are stored, the normalization result is generated according to the name, the pointer is stored at a next node in the distributed index 201, for example, an OSD2 stores a name of itself, a normalization result generated according to the name, and a pointer, wherein the pointer of the OSD2 stores a key of the OSD9, therefore, after the OSD client 101 accesses the OSD2 according to the key of the OSD2, the OSD 6342 may be further accessed through the interface 2, and the pointer 3 … … is set in the client through the OSD 639, so that the OSD forms the distributed index 201, the client 101 only needs to record the key of the node as the entry of the distributed index 201, such as the key of the OSD2, for example, and all the nodes in the distributed index 201 can be queried from the entry of the distributed index 201 through the distributed index 201.
In the embodiment of the present invention, the client 101 applies for the OSD from the OSD cluster 20 through the interface, and manages the applied OSD in the form of the distributed index 201, so that the two-layer MDS can be avoided, and the complexity of deployment and operation and maintenance can be reduced.
For further clarity, referring to fig. 2, fig. 2 is a flowchart of an object distributed index 201 method according to an embodiment of the present invention, and as shown in fig. 2, the object distributed index 201 method includes the following steps:
step S301: the client 101 obtains the content and name of the object to be stored.
The object includes a file and naked data.
For example, the client 101 may provide a web page for a user to access, and the user uploads the object to be stored on the web page, so that the client 101 may obtain the content and the name of the object to be stored.
In this step, it is assumed that the name of the object to be stored is "minuscule".
Step S302: the client 101 performs normalization processing on the name of the object to be stored to obtain a first normalization result.
In the embodiment of the present invention, the normalization process refers to converting the name of the object to be stored into a normalization result that can be compared.
For example, the client 101 may calculate an ASCII (American Standard Code for Information exchange) Code corresponding to the name, and convert the name into a normalization result that can be compared, so as to realize the normalization process.
In some other examples, the client 101 may also perform a hash operation on the name to obtain a normalization result, so as to implement the normalization process.
Assume that after normalization processing is performed on the decimal, the first normalization result obtained is 10.
Step S303: the client 101 obtains keys of the idle OSDs in the OSD cluster 20 through the interface, and stores the first normalization result, the name, the content, and the keys of the idle OSDs in the idle OSDs through the interface, thereby generating new nodes.
In this step, the client 101 may determine whether the size of the object to be stored is larger than the storage space of the OSD, and when it is determined that the size of the object to be stored is larger than the storage space of the OSD, divide the object to be stored into a predetermined number of portions, obtain keys of a predetermined number of idle OSDs in the OSD cluster 20 through the interface, and store the portions of the object to be stored in the idle OSDs and the predetermined number of idle OSDs, respectively; and selecting any OSD from the preset number of idle OSD as a new node, and storing the first normalization result, the name, the content and the key of the preset number of idle OSD in the new node through an interface.
Therefore, in the above manner, the embodiment of the invention can store a larger object in the OSD, thereby realizing capacity expansion.
In other examples, when the client 101 determines that the size of the object to be stored is smaller than or equal to the storage space of the OSD, the object to be stored does not need to be divided, and only the object to be stored needs to be directly stored in the obtained idle OSD.
Step S304: the client 101 queries, through the interface, a back node in the distributed index 201, in which a second normalization result larger than the first normalization result is stored, and a front node in which a third normalization result smaller than the first normalization result is stored.
Referring to fig. 3 in particular, fig. 3 is a schematic diagram of a distributed index 201 according to an embodiment of the present invention, and the distributed index 201 shown in fig. 3 is a single distributed index 201, and in the distributed index 201, 7 nodes are included, which are OSD2, OSD9, OSD3, OSD5, OSD7, OSD1, and OSD8, respectively.
The OSD2 is an entry node of the distributed index 201 that stores pointers storing keys of the OSD9, normalization results being negative infinity,
the OSD8 is the end node of the distributed index 201, and stores a pointer storing the key of the OSD9, the normalization result being positive infinity, and other data.
The other nodes respectively store the pointer, the normalization result, and other data, and the key stored by the pointer is the key of the next node as shown in fig. 3, where the other data includes the object content, the object name, the object length, and the like.
In this step, the client 101 queries the normalization results stored by the nodes in the distributed index 201 from the entry node according to the pointer stored by each node in the distributed index 201 through the interface until querying a back node in the distributed index 201, where a second normalization result larger than the first normalization result is stored, and a front node in the distributed index 201, where a third normalization result smaller than the first normalization result is stored.
Specifically, the client 101 first accesses the entry node OSD2, accesses the OSD9 according to the pointer of the OSD2, knows that the normalization result of the OSD9 is 1, since 10 is greater than 1, the client 101 further accesses the OSD3 pointed by the pointer of the OSD9, knows that the normalization result of the OSD3 is 5, since 10 is greater than 5, the client 101 further accesses the OSD5 pointed by the pointer of the OSD3, since 10 is greater than 9, the client 101 further accesses the OSD11 pointed by the pointer of the OSD5, and since 10 is less than 53, the client 101 queries the back node OSD11 in the distributed index 201, in which the second normalization result (53) greater than the first normalization result (10) is stored, and the front node OSD5, in which the third normalization result (9) less than the first normalization result (10) is stored.
Step S305: the client 101 constructs a pointer at the new node through the interface, sets the pointer of the new node as the key of the back node, and sets the pointer of the front node as the key of the new node.
For example, referring to fig. 4, fig. 4 is another schematic diagram of the distributed index 201 according to an embodiment of the present invention, the client 101 constructs a pointer in the OSD9 such that the pointer of the OSD9 stores a key of the OSD7, and modifies the pointer of the OSD7 such that the pointer of the OSD7 stores a key of the OSD11, so that the OSD7 can be inserted into the single distributed index 201.
It should be noted that, in this step, since the generation of the new node in step S303 and the modification of the pointer of the previous node in this step are two independent steps, atomicity cannot be guaranteed, if the modification of the pointer of the previous node OSD5 in this step fails, the client 101 loses the record of the new node OSD7, in this step, it is necessary to further construct a log at the previous node OSD5, store the key of the new node OSD7 in the log, when the client 101 fails to set the pointer of the previous node OSD5 as the key of the new node OSD7 through the interface, the client 101 may read the key of the new node from the log of the previous node OSD7, set the pointer of the previous node OSD5 as the key of the read new node OSD7, and after the setting is successful, the client 101 sets the log to zero through the interface.
Therefore, after the above steps, the client 101 forms a new distributed index 201 in the OSD cluster 20, specifically as shown in fig. 5, fig. 5 is another schematic diagram of the storage system according to the embodiment of the present invention, and in fig. 5, the OSD2, the OSD9, the OSD3, the OSD5, the OSD11, the OSD7, the OSD1, and the OSD8 form the distributed index 201, wherein the OSD11 is inserted into the distributed index 201 shown in fig. 1 by the above method.
Therefore, in the embodiment of the present invention, the client 101 applies for the OSD from the OSD cluster 20 through the interface, and inserts the applied OSD into the distributed index 201, so that the two-layer MDS can be avoided, and the complexity of deployment and operation and maintenance can be reduced.
An embodiment of deleting a node in the distributed index 201 will be described below, please refer to fig. 6, fig. 6 is a flowchart of a method for object distributed index 201 according to an embodiment of the present invention, and as shown in fig. 6, the method for object distributed index 201 includes the following steps:
step S401: the client 101 obtains the name of the object to be deleted.
For example, the client 101 may provide a web page for the user to access, and the user inputs the name of the object to be deleted on the web page, so that the client 101 may obtain the name of the object to be deleted.
In this step, it is assumed that the name of the object to be deleted is "Xiaoming".
Step S402: the client 101 normalizes the name to obtain a fourth normalization result.
In the embodiment of the present invention, the normalization process refers to converting the name of the object to be stored into a normalization result that can be compared.
For example, the client 101 may calculate an ASCII (American Standard Code for Information exchange) Code corresponding to the name, and convert the name into a normalization result that can be compared, so as to realize the normalization process.
In some other examples, the client 101 may also perform a hash operation on the name to obtain a normalization result, so as to implement the normalization process.
In some other examples, the client 101 may also perform dictionary ordering on the names to achieve the normalization process.
It is assumed that a fourth normalization result obtained after normalization processing for "xiaoming" is 10.
Step S403: the client 101 queries the node to be deleted in the distributed index 201, where the fourth normalization result is stored, through the interface.
Specifically, the distributed index 201 includes entry nodes, where the entry nodes store negative infinite normalization results, and the client 101 queries, through the interface, the normalization results stored by the nodes in the distributed index 201 from the entry nodes according to the pointer stored by each node in the distributed index 201 until querying the node in the distributed index 201 that stores the first normalization result.
According to the above description, the OSD2 is an entry node, the client 101 first accesses the OSD2, knows that the normalization result of the OSD2 is negative infinity, accesses the OSD9 according to the pointer of the OSD2, knows that the normalization result of the OSD9 is 1, since 10 is greater than 1, the client 101 further accesses the OSD3 pointed by the pointer of the OSD9, knows that the normalization result of the OSD3 is 5, since 10 is greater than 5, the client 101 further accesses the OSD5 pointed by the pointer of the OSD3, since 10 is greater than 9, the normalization result of the OSD5 is 9, since 10 is greater than 9, the client 101 further accesses the OSD7 pointed by the pointer of the OSD5, and since 10 is greater than 9, the normalization result of the OSD7 is 10, the client 101 takes the OSD7 as a node to be deleted.
Step S404: the client 101 sets the pointer of the front node of the node to be deleted as the key of the back node of the node to be deleted through the interface.
In this step, the client 101 sets the pointer of the front node OSD5 of the OSD7 to the key of the OSD1 through the interface.
Step S405: the client 101 notifies the OSD cluster 20 through the interface to set the node to be deleted as a free OSD.
In this step, the client 101 notifies the OSD cluster 20 to recover the nodes to be deleted in the above manner, so that the OSD resources can be effectively recovered, and idle OSD is not occupied.
Further, in this step, the client 101 may query whether the node to be deleted further stores the key for storing the OSD of the larger object as described in step S303, and if so, the client 101 further notifies the OSD cluster 20 through the interface to set the OSD for storing the larger object as the idle OSD.
It should be noted that, since the modification of the pointers of the nodes in the step S404 and the previous step is two independent steps, atomicity cannot be guaranteed, that is, if the step S404 fails to be executed, the key of the node to be deleted is lost by the client 101, so that the step S404 and the step S405 fail to be executed.
Therefore, in step S404, the client 101 constructs a log at a previous node through the interface, stores the key stored in the node to be deleted in the log, and when the client 101 fails to set the pointer of the previous node as the key of the next node through the interface, reads the key of the node to be deleted from the log of the previous node, reads the key of the next node stored in the pointer of the new node through the interface according to the key of the node to be deleted, and sets the pointer of the previous node as the read key of the next node, thereby ensuring that the key of the node to be deleted is not lost when the pointer of the previous node is modified and failed, it is worth noting that after the pointer of the previous node is modified successfully, the log of the previous node does not need to record the key of the node to be deleted, and at this time, the client 101 can set the log to zero through the interface.
In summary, in the embodiment of the present invention, the client 101 applies for the OSD from the OSD cluster 20 through the interface, and deletes the applied OSD from the distributed index 201, so that the two-layer MDS can be avoided, and the complexity of deployment and operation and maintenance can be reduced.
While the distributed index 201 described above is a single distributed index 201, it should be noted that in alternative embodiments of the present invention, the distributed index 201 may also be a skip list and a binary tree, where a skip list is particularly suitable for situations where the number of stored objects is large.
Reference may be specifically made to fig. 7 and fig. 8, where fig. 7 is another schematic diagram of the distributed index 201 according to the embodiment of the present invention, and the distributed index 201 shown in fig. 7 is a skip list, and a pointer includes a plurality of pointers, and each pointer points to a different layer of the skip list.
For example, the OSD2 stores a layer 1 pointer, a layer 2 pointer, a layer 3 pointer, and a layer 4 pointer, each of which stores a node corresponding to each layer, when the client 101 is to find the OSD7 (normalization result 10) in the skip list shown in fig. 7, it first finds a node of the same layer pointed by the highest layer of the entry node OSD2, i.e., the OSD8 pointed by the layer 4 pointer, since the normalization result of the OSD8 is positive, it finds a node OSD1 of the same layer pointed by the next highest layer (layer 3), whose normalization result is 78 not consistent with 10, it finds an OSD3 pointed by the layer 2 pointer, and whose normalization result is 5 and less than 10, and thus finds an OSD1 pointed by the layer 2 of the OSD3, since the normalization result of the OSD1 is 78 and greater than 10, it finds an OSD5 pointed by the layer 1 of the OSD3, and, when the normalization result 9 of the query to the OSD5 is less than 10, it further finds a layer 1 st OSD7 of the OSD5, since the normalized result of the OSD7 is 10, the OSD7 is confirmed to be the node to be queried.
Specifically, referring to fig. 8, fig. 8 is another schematic diagram of the distributed index 201 according to an embodiment of the present invention, and unlike fig. 7, in fig. 8, the OSD7 stores 4-level pointers, so that, when the client 101 wants to find the OSD7 (normalized result 10) in the skip list shown in fig. 8, the node of the same layer pointed by the highest level of the entry node OSD2, that is, the OSD7 pointed by the 4-level pointer, is searched first, and since the normalized result of the OSD87 is 10, it is determined that the OSD7 is the node to be queried.
It should be noted that, if each node is provided with multiple layers of pointers, the searching speed of the distributed index 201 is seriously affected, and therefore, the number of layers of each node needs to be limited, specifically, the number of layers of each node can be determined by referring to the existing "coin tossing" algorithm, for example, for OSD2, whether to add a layer 2 pointer is determined by the "coin tossing" algorithm, if the coin tossed is a positive side, the layer 2 pointer is added, if the coin tossed is a negative side, only the layer 1 pointer is retained, and after the layer 2 pointer is added, whether to add a layer 3 pointer … … is further determined by coin tossing, when a new node is inserted, the above processing is used, so that the multiple layers of pointers are uniformly distributed, and the searching speed of the distributed index 201 is not affected.
Further, the client 101 may also cache the queried node, and when the normalized result to be queried exists in the cache, the query does not need to be performed to the distributed index 201, and the cache is read locally.
The nodes with frequent access are cached in each client 101, so that a plurality of copies of the hot nodes are shared by hot spots, and meanwhile, the pressure on OSD is reduced because the service nodes are directly accessed from the internal cache when accessing the hot nodes.
For example, the cache is organized in an LRU (Least recently used) manner, since the nodes are globally shared, the cache may involve a problem of consistency, so the cache may be used when locating the nodes, after finding the relevant node in the cache, the OSD needs to directly access the stored content of the node for confirmation, the cache is deleted if the Key is inconsistent or the node is not present, and the cache is updated if other content is inconsistent. Moreover, the cache can only cache common nodes, and nodes in the transaction of adding and deleting nodes can not be cached, so that the problem of more complex consistency is avoided.
For example, embodiments of the present invention relate to the use of CAS (Compare and swap) operations for OSD intra-data modification, which typically require the introduction of old values, such as:
bool compare_and_set(osd_key_t key,void*old_value,void*new_value)；
where key is metadata indicating the location of the object in the OSD, such as written location offset, written data length, etc., old _ value, the old value, i.e., the data in the OSD object before the write occurred; new _ value, data to be written.
The scheme of the invention improves the CAS operation as follows, and replaces the old value with, but not limited to, checksum (check value), version number and the like:
1. the checksum value of the current content is calculated first when changing, such as using HASH32 (HASH 32 Algorithm) or MD5 (Message-Digest Algorithm 5) to replace the old value, and the change is made if the same as the incoming.
2. And replacing the old value with the version identifier, namely adding a version number identifier in the data structure to be changed, updating the version identifier when the change is successful (such as a self-increasing normalization result, wherein the change is successful by +1) each time, transmitting the old version identifier when the change is performed, and changing when the current content version identifier is consistent with the transmitted version identifier.
Through the mode, the data volume needing to be transmitted can be effectively reduced.
The embodiment of the present invention further provides a client 101, where the client 101 interacts with an OSD cluster 20 through an interface, the OSD cluster 20 includes a plurality of OSDs, the client 101 selects the OSDs as nodes through the interface to construct a distributed index 201, each node of the distributed index 201 stores a name and a normalization result of the client, a key of the client in the OSD cluster 20, and a pointer, the normalization result is generated according to the name, and the pointer stores a key of a next node in the distributed index 201, specifically refer to fig. 9, fig. 9 is a schematic diagram of an apparatus structure of the client according to the embodiment of the present invention, and as shown in fig. 9, the client 101 includes:
a to-be-stored object obtaining module 501, configured to obtain content and a name of an object to be stored;
a normalization module 502, configured to perform normalization processing on a name of an object to be stored to obtain a first normalization result;
a new node generation module 503, configured to obtain keys of idle OSDs in the OSD cluster 20 through the interface, and store the first normalization result, the name, the content, and the keys of the idle OSDs in the idle OSDs through the interface, so as to generate new nodes;
a query module 504, configured to query, through an interface, a back node in the distributed index 201, where a second normalized result that is greater than the first normalized result is stored, and a front node in the distributed index 201, where a third normalized result that is less than the first normalized result is stored;
and the pointer modification module 505 is configured to construct a pointer at the new node through the interface, set the pointer of the new node as the key of the next node, and set the pointer of the previous node as the key of the new node.
Optionally, the distributed index 201 includes an entry node, the entry node stores the negative infinity normalization result, the client 101 further includes an entry recording module, the entry recording module is configured to record a key of the entry node,
the query module 504 is specifically configured to query, through an interface, the normalization result stored in the node in the distributed index 201 from the entry node according to the pointer stored in each node in the distributed index 201 until a back node in the distributed index 201, in which a second normalization result larger than the first normalization result is stored, and a front node in which a third normalization result smaller than the first normalization result is stored, are queried.
Optionally, the pointer modification module 505 is specifically configured to:
constructing a pointer at the new node through an interface, and setting the pointer of the new node as a key of the back node; constructing a log at a previous node through an interface, and storing a key of a new node in the log; reading the key of the new node from the log of the previous node and setting the pointer of the previous node as the read key of the new node when the pointer of the previous node is set as the key stored by the new node through the interface; the log is zeroed through the interface.
Optionally, the new node generating module 503 is specifically configured to:
when the size of the object to be stored is judged to be larger than the storage space of the OSD, dividing the object to be stored into a preset number of parts, acquiring keys of a preset number of idle OSD in the OSD cluster 20 through an interface, and respectively storing the parts of the object to be stored in the idle OSD and the preset number of idle OSD; and selecting any OSD from the preset number of idle OSD as a new node, and storing the first normalization result, the name, the content and the key of the preset number of idle OSD in the new node through an interface.
Optionally, the distributed index 201 is a skip list, and the pointer includes a plurality of pointers, each pointer pointing to a different layer of the skip list.
Optionally, the client 101 of the embodiment of the present invention further includes a notification module 505, where:
the name obtaining module 501 is further configured to obtain a name of the object to be deleted;
a normalization module 502, configured to perform normalization processing on the name to obtain a first normalization result;
the query module 503 is configured to query, through an interface, a node to be deleted in the distributed index 201, where the first normalization result is stored;
the pointer modification module 504 is configured to set a pointer of a front node of the node to be deleted as a key of a rear node of the node to be deleted;
a notification module 505, configured to notify the OSD cluster 20 through the interface to set the node to be deleted as an idle OSD.
Optionally, the distributed index 201 includes an entry node, a pointer of the entry node stores a node of which the normalization result is the minimum value in the distributed index 201, the client 101 further includes an entry recording module, the entry recording module is configured to record a key of the entry node, and the query module 603 is specifically configured to:
and querying the normalization results stored by the nodes in the distributed index 201 from the entry node through the interface according to the pointer stored by each node in the distributed index 201 until the node in the distributed index 201, in which the first normalization result is stored, is queried.
Optionally, the pointer modification module 604 is specifically configured to:
constructing a log at a previous node through an interface, and storing keys stored by a new node in the log;
when the pointer of the front node is set to be the key of the rear node through the interface, the key of the new node is read from the log of the front node, the key of the rear node stored by the pointer of the new node is read through the interface according to the key of the new node, and the pointer of the front node is set to be the read key of the rear node.
Optionally, the pointer modification module 604 is further configured to zero the log through the interface.
Optionally, the distributed index 201 is a skip list, and the pointer includes a plurality of pointers, each pointer pointing to a different layer of the skip list.
Referring to fig. 9, fig. 9 is a schematic structural diagram of another apparatus of the client according to an embodiment of the present invention, as shown in fig. 9, the client 101 includes a processor 701, a memory 702, and a bus 703, the processor 701 and the memory 702 are respectively connected to the bus 703, the memory 702 stores program instructions, and the processor 701 executes the program instructions to complete the functions of the client 101 described in the above embodiments, wherein:
the processor 701 executes the first program instruction to perform step S301: acquiring the content and the name of an object to be stored;
the processor 701 executes the second program instruction to perform step S302: carrying out normalization processing on the name of the object to be stored to obtain a first normalization result;
the processor 701 executes the third program instruction to perform step S303: acquiring keys of idle OSD in the OSD cluster 20 through an interface, and storing the first normalization result, the name, the content and the keys of the idle OSD in the idle OSD through the interface, thereby generating a new node;
the processor 701 executes the fourth program instruction to perform step S304: querying a back node in the distributed index 201, which stores a second normalization result larger than the first normalization result, and a front node in which a third normalization result smaller than the first normalization result is stored through an interface;
the processor 701 executes the fifth program instruction to perform step S305: and constructing a pointer at the new node through the interface, setting the pointer of the new node as the key of the back node, and setting the pointer of the front node as the key of the new node.
Optionally, the distributed index 201 includes an entry node, a pointer stored in the entry node stores a node whose normalization result is the minimum value in the distributed index 201, the memory 702 records a key of the entry node, and the processor 701 executes a fourth program instruction to query, from the entry node through an interface, the normalization result stored in the node in the distributed index 201 according to the pointer stored in each node in the distributed index 201 until a back node in the distributed index 201, which stores a second normalization result larger than the first normalization result, and a front node in which a third normalization result smaller than the first normalization result are stored are queried.
Optionally, the processor 701 executes a fifth program instruction to construct a pointer at the new node through the interface, and set the pointer of the new node as a key of the back node; constructing a log at a previous node through an interface, and storing a key of a new node in the log; reading the key of the new node from the log of the previous node and setting the pointer of the previous node as the read key of the new node when the pointer of the previous node is set as the key stored by the new node through the interface; the log is zeroed through the interface.
Optionally, the processor 701 executes a third program instruction to, when it is determined that the size of the object to be stored is larger than the storage space of the OSD, divide the object to be stored into a predetermined number of parts, obtain keys of a predetermined number of idle OSDs in the OSD cluster 20 through the interface, and store the parts of the object to be stored in the idle OSDs and the predetermined number of idle OSDs, respectively; and selecting any OSD from the preset number of idle OSD as a new node, and storing the first normalization result, the name, the content and the key of the preset number of idle OSD in the new node through an interface.
Optionally, the distributed index 201 is a skip list, and the pointer includes a plurality of pointers, each pointer pointing to a different layer of the skip list.
Referring to fig. 10, fig. 10 is a schematic diagram illustrating another device structure of a client according to an embodiment of the present invention, as shown in fig. 10, the client 101 includes a processor 801, a memory 802, and a bus 803, the processor 801 and the memory 802 are respectively connected to the bus 803, the memory 802 stores program instructions, and the processor 801 executes the program instructions to implement the functions of the client 101 described in the above embodiments, wherein:
the processor 801 executes the first program instruction to perform step S401: acquiring the name of an object to be deleted;
the processor 801 executes the second program instruction to perform step S402: carrying out normalization processing on the name to obtain a fourth normalization result;
the processor 801 executes the third program instruction to perform step S403: querying nodes to be deleted, in which the first normalization result is stored, in the distributed index 201 through an interface;
the processor 801 executes the fourth program instruction to perform step S404: setting a pointer of a front node of a node to be deleted as a key of a rear node of the node to be deleted through an interface;
the processor 801 executes the fifth program instruction to perform step S405: the OSD cluster 20 is notified through the interface to set the node to be deleted as a free OSD.
Optionally, the distributed index 201 includes entry nodes, the entry nodes store negative infinity normalization results, the memory 802 is further configured to record keys of the entry nodes, and the processor 801 executes the third program instruction to query, starting from the entry nodes through the interface, the normalization results stored by the nodes in the distributed index 201 according to the pointer stored by each node in the distributed index 201 until the node in the distributed index 201 storing the first normalization result is queried.
Optionally, the processor 801 executes a fourth program instruction to construct a log by interfacing the previous node, and store the key stored by the new node in the log; when the pointer of the front node is set to be the key of the rear node through the interface, the key of the new node is read from the log of the front node, the key of the rear node stored by the pointer of the new node is read through the interface according to the key of the new node, and the pointer of the front node is set to be the read key of the rear node.
Optionally, processor 801 executes a fourth program instruction to zero the log through the interface.
Optionally, the distributed index 201 is a skip list, and the pointer includes a plurality of pointers, each pointer pointing to a different layer of the skip list.
In summary, in the embodiment of the present invention, the client 101 applies for the OSD from the OSD cluster 20 through the interface, and manages the applied OSD in the form of the distributed index 201, so that the two-layer MDS can be avoided, and the complexity of deployment and operation and maintenance can be reduced.
It should be noted that any of the above-described device embodiments are merely schematic, where the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on multiple network units. Part or all of the processes can be selected according to actual needs to achieve the purpose of the scheme of the embodiment. In addition, in the drawings of the embodiment of the apparatus provided by the present invention, the connection relationship between the processes indicates that there is a communication connection between them, and may be specifically implemented as one or more communication buses or signal lines. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that the present invention may be implemented by software plus necessary general hardware, and may also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components and the like. Generally, functions performed by computer programs can be easily implemented by corresponding hardware, and specific hardware structures for implementing the same functions may be various, such as analog circuits, digital circuits, or dedicated circuits. However, the implementation of a software program is a more preferable embodiment for the present invention. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a readable storage medium, such as a floppy disk, a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk of a computer, and includes instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the methods according to the embodiments of the present invention.
It is clear to those skilled in the art that the specific working process of the above-described system, apparatus or unit may refer to the corresponding process in the foregoing method embodiments, and is not described herein again.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.
Claims (15)
1. An Object-based distributed indexing method applied to a client interacting with an OSD (Object-based Storage Device) cluster through an interface, the OSD cluster including a plurality of OSDs, the client selecting OSDs as nodes through the interface to construct a distributed index, in each of the nodes of the distributed index, a name and a normalization result of itself, a key of itself in the OSD cluster, and a pointer, the normalization result being generated according to the name, the pointer storing a key of a next node in the distributed index, the method comprising:
the client acquires the content and the name of an object to be stored;
the client side carries out normalization processing on the name of the object to be stored to obtain a first normalization result;
the client acquires keys of idle OSD in the OSD cluster through the interface, and stores the first normalization result, the name, the content and the keys of the idle OSD in the idle OSD through the interface so as to generate a new node;
the client queries a rear node, which stores a second normalization result larger than the first normalization result, and a front node, which stores a third normalization result smaller than the first normalization result, in the distributed index through the interface;
and the client constructs a pointer at the new node through the interface, sets the pointer of the new node as the key of the back node, and sets the pointer of the front node as the key of the new node.
2. The method according to claim 1, wherein the distributed index includes an entry node, a pointer of the entry node stores a key of a node with a minimum normalized result in the distributed index, and the step of the client querying, through the interface, a back node in the distributed index that stores a second normalized result that is larger than the first normalized result and a front node that stores a third normalized result that is smaller than the first normalized result specifically includes:
and the client queries the normalization results stored by the nodes in the distributed index from the entry node through the interface according to the pointer stored by each node in the distributed index until a rear node in the distributed index, which stores a second normalization result larger than the first normalization result, and a front node in the distributed index, which stores a third normalization result smaller than the first normalization result, are queried.
3. The method according to claim 1, wherein the step of the client constructing a pointer at the new node through the interface, setting the pointer of the new node as the key of the back node, and setting the pointer of the front node as the key of the new node specifically comprises:
the client constructs a pointer at the new node through the interface, and sets the pointer of the new node as a key of the back node;
the client constructs a log at the front node through the interface, and stores the key of the new node in the log;
when the client fails to set the pointer of the front node as the key stored by the new node through the interface, reading the key of the new node from the log of the front node, and setting the pointer of the front node as the read key of the new node;
and the client sets the log to zero through the interface.
4. The method of claim 3, wherein the client obtains keys of idle OSD in the OSD cluster through the interface, and the storing the first normalization result, the name, the content and the keys of the idle OSD in the idle OSD through the interface, so as to generate a new node specifically comprises:
when the client judges that the size of the object to be stored is larger than the storage space of the OSD, dividing the object to be stored into a preset number of parts, acquiring keys of the preset number of idle OSD in the OSD cluster through the interface, and respectively storing the parts of the object to be stored in the idle OSD and the preset number of idle OSD;
selecting any OSD from the predetermined number of idle OSD as the new node, and storing the first normalization result, the name, the content and the key of the predetermined number of idle OSD in the new node through the interface.
5. The method according to any one of claims 1 to 4, wherein the distributed index is a skip list, and the pointer comprises a plurality of pointers, each of the pointers pointing to a different layer of the skip list.
6. The method of claim 1, further comprising:
the client acquires the name of an object to be deleted;
the client side carries out normalization processing on the name to obtain a fourth normalization result;
the client queries the nodes to be deleted in the distributed index, which store the fourth normalization result, through the interface;
the client sets the pointer of the front node of the node to be deleted as the key of the rear node of the node to be deleted through the interface;
and the client informs the OSD cluster of setting the nodes to be deleted as idle OSD through the interface.
7. The method according to claim 6, wherein the step of setting, by the client, the pointer of the front node of the node to be deleted as the key of the back node of the node to be deleted through the interface specifically includes:
the client constructs a log at the front node through the interface, and stores the key stored by the node to be deleted in the log;
when the client fails to set the pointer of the front node as the key of the rear node through the interface, reading the key of the node to be deleted from the log of the front node, reading the key of the rear node stored by the pointer of the node to be deleted through the interface according to the key of the node to be deleted, and setting the pointer of the front node as the read key of the rear node;
and the client sets the log to zero through the interface.
8. A client interacting with an OSD cluster through an interface, the OSD cluster including a plurality of OSDs, the client selecting OSDs as nodes through the interface to construct a distributed index, in each of the nodes of the distributed index, a name and a normalization result of itself, a key of itself in the OSD cluster, and a pointer, the normalization result being generated according to the name, the pointer storing a key of a next node in the distributed index, the client comprising:
the device comprises a to-be-stored object acquisition module, a storage module and a storage module, wherein the to-be-stored object acquisition module is used for acquiring the content and the name of an object to be stored;
the normalization module is used for performing normalization processing on the name of the object to be stored to obtain a first normalization result;
a new node generation module, configured to obtain keys of idle OSDs in the OSD cluster through the interface, and store the first normalization result, the name, the content, and the keys of the idle OSDs in the idle OSDs through the interface, so as to generate new nodes;
the query module is used for querying a rear node in the distributed index, which stores a second normalization result larger than the first normalization result, and a front node in the distributed index, which stores a third normalization result smaller than the first normalization result, through the interface;
and the pointer modification module is used for constructing a pointer on the new node through the interface, setting the pointer of the new node as the key of the back node, and setting the pointer of the front node as the key of the new node.
9. The client according to claim 8, wherein the distributed index includes entry nodes, pointers of the entry nodes store keys of nodes with minimum normalized results in the distributed index, the client further includes an entry recording module, the entry recording module is configured to record the keys of the entry nodes,
the query module is specifically configured to query, through the interface, normalization results stored in nodes in a linked list from the entry node according to the pointer stored in each node in the distributed index until a back node in which a second normalization result larger than the first normalization result is stored and a front node in which a third normalization result smaller than the first normalization result is stored in the distributed index are queried.
10. The client according to claim 8, wherein the pointer modification module is specifically configured to:
constructing a pointer at the new node through the interface, and setting the pointer of the new node as a key of the back node;
constructing a log at the front node through the interface, and storing the key of the new node in the log;
when the pointer of the front node is set to the key stored by the new node through the interface, reading the key of the new node from the log of the front node, and setting the pointer of the front node to the read key of the new node;
zeroing the log through the interface.
11. The client according to claim 10, wherein the new node generation module is specifically configured to:
when the size of the object to be stored is judged to be larger than the storage space of the OSD, dividing the object to be stored into a preset number of parts, acquiring keys of the preset number of idle OSD in the OSD cluster through the interface, and respectively storing the parts of the object to be stored in the idle OSD and the preset number of idle OSD;
selecting any OSD from the predetermined number of idle OSD as the new node, and storing the first normalization result, the name, the content and the key of the predetermined number of idle OSD in the new node through the interface.
12. The client according to any one of claims 8 to 11, wherein the distributed index is a skip list, and the pointer comprises a plurality of pointers, each of the pointers pointing to a different layer of the skip list.
13. The client of claim 8, further comprising a notification module,
the name acquisition module is also used for acquiring the name of the object to be deleted;
the normalization module is further configured to perform normalization processing on the name to obtain a fourth normalization result;
the query module is further configured to query, through the interface, the node to be deleted in the distributed index, where the fourth normalization result is stored;
the pointer modification module is further configured to set a pointer of a front node of the node to be deleted as a key of a back node of the node to be deleted;
and the notification module is used for notifying the OSD cluster to set the node to be deleted as an idle OSD through the interface.
14. The client of claim 13, wherein the pointer modification module is specifically configured to:
constructing a log at the front node through the interface, and storing the key stored by the node to be deleted in the log;
when the pointer of the front node is failed to be set as the key of the rear node through the interface, reading the key of the node to be deleted from the log of the front node, reading the key of the rear node stored by the pointer of the node to be deleted through the interface according to the key of the node to be deleted, and setting the pointer of the front node as the read key of the rear node;
zeroing the log through the interface.
15. A client, comprising a processor, a memory and a bus, wherein the processor and the memory are respectively connected to the bus, and the memory stores program instructions, and wherein:
the processor executes a first program instruction to perform the steps of: acquiring the content and the name of an object to be stored;
the processor executes second program instructions to perform the steps of: carrying out normalization processing on the name of the object to be stored to obtain a first normalization result;
the processor executes third program instructions to perform the steps of: acquiring keys of idle OSD in an OSD cluster through an interface, and storing the first normalization result, the name, the content and the keys of the idle OSD in the idle OSD through the interface so as to generate a new node;
the processor executes the fourth program instruction to perform the steps of: querying a rear node in the distributed index, which stores a second normalization result larger than the first normalization result, and a front node in the distributed index, which stores a third normalization result smaller than the first normalization result, through the interface;
the processor executes fifth program instructions to perform the steps of: and constructing a pointer at the new node through the interface, setting the pointer of the new node as the key of the back node, and setting the pointer of the front node as the key of the new node.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710233466.XA CN108694209B (en) | 2017-04-11 | 2017-04-11 | Distributed index method based on object and client |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710233466.XA CN108694209B (en) | 2017-04-11 | 2017-04-11 | Distributed index method based on object and client |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108694209A CN108694209A (en) | 2018-10-23 |
| CN108694209B true CN108694209B (en) | 2021-11-19 |
Family
ID=63843322
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710233466.XA Active CN108694209B (en) | 2017-04-11 | 2017-04-11 | Distributed index method based on object and client |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108694209B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109669822B (en) * | 2018-11-28 | 2023-06-06 | 平安科技（深圳）有限公司 | Electronic device, method for creating backup storage pool, and computer-readable storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1197570A (en) * | 1995-08-02 | 1998-10-28 | 汤姆森消费电子有限公司 | Linked list structure onscreen display |
| CN1514976A (en) * | 1998-07-24 | 2004-07-21 | �ָ��� | Distributed computer data base system and method for object searching |
| CN101996217A (en) * | 2009-08-24 | 2011-03-30 | 华为技术有限公司 | Method for storing data and memory device thereof |
| CN102779185B (en) * | 2012-06-29 | 2014-11-12 | 浙江大学 | High-availability distribution type full-text index method |
| CN106095920A (en) * | 2016-06-07 | 2016-11-09 | 四川大学 | Distributed index method towards extensive High dimensional space data |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR101266358B1 (en) * | 2008-12-22 | 2013-05-22 | 한국전자통신연구원 | A distributed index system based on multi-length signature files and method thereof |
-
2017
- 2017-04-11 CN CN201710233466.XA patent/CN108694209B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1197570A (en) * | 1995-08-02 | 1998-10-28 | 汤姆森消费电子有限公司 | Linked list structure onscreen display |
| CN1514976A (en) * | 1998-07-24 | 2004-07-21 | �ָ��� | Distributed computer data base system and method for object searching |
| CN101996217A (en) * | 2009-08-24 | 2011-03-30 | 华为技术有限公司 | Method for storing data and memory device thereof |
| CN102779185B (en) * | 2012-06-29 | 2014-11-12 | 浙江大学 | High-availability distribution type full-text index method |
| CN106095920A (en) * | 2016-06-07 | 2016-11-09 | 四川大学 | Distributed index method towards extensive High dimensional space data |
Non-Patent Citations (1)
| Title |
|---|
| "对象存储文件***的设计与实现";胡永奎;《中国优秀硕士学位论文全文数据库 信息科技辑》;20120215(第2期);第I137-28页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108694209A (en) | 2018-10-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110998557B (en) | High availability database system and method via distributed storage | |
| US8429134B2 (en) | Distributed database recovery | |
| CN110018998B (en) | File management method and system, electronic equipment and storage medium | |
| CN110196818B (en) | Data caching method, caching device and storage system | |
| US20200334292A1 (en) | Key value append | |
| CN106326229B (en) | File storage method and device of embedded system | |
| US20190087130A1 (en) | Key-value storage device supporting snapshot function and operating method thereof | |
| US20150006478A1 (en) | Replicated database using one sided rdma | |
| EP3570182B1 (en) | Sparse infrastructure for tracking ad-hoc operation timestamps | |
| CN111381779A (en) | Data processing method, device, equipment and storage medium | |
| US11249866B1 (en) | Snapshot-based data corruption detection | |
| CN111831691A (en) | Data reading and writing method and device, electronic equipment and storage medium | |
| CN111639087B (en) | Data updating method and device in database and electronic equipment | |
| CN108694209B (en) | Distributed index method based on object and client | |
| CN111309799A (en) | Method, device and system for realizing data merging and storage medium | |
| CN111611223B (en) | Non-volatile data access method, system, electronic device and medium | |
| CN112115521B (en) | Data access method and device | |
| CN116303267A (en) | Data access method, device, equipment and storage medium | |
| US10073874B1 (en) | Updating inverted indices | |
| CN112130747A (en) | Distributed object storage system and data read-write method | |
| CN114490540A (en) | Data storage method, medium, device and computing equipment | |
| CN112084141A (en) | Full-text retrieval system capacity expansion method, device, equipment and medium | |
| CN112181899A (en) | Metadata processing method and device and computer readable storage medium | |
| CN116909490B (en) | Data processing method, device, storage system and computer readable storage medium | |
| CN115113798B (en) | Data migration method, system and equipment applied to distributed storage |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220221 Address after: 550025 Huawei cloud data center, jiaoxinggong Road, Qianzhong Avenue, Gui'an New District, Guiyang City, Guizhou Province Patentee after: Huawei Cloud Computing Technology Co.,Ltd. Address before: 518129 Bantian HUAWEI headquarters office building, Longgang District, Guangdong, Shenzhen Patentee before: HUAWEI TECHNOLOGIES Co.,Ltd. |