CN117081931B - Online capacity expansion method and device for heterogeneous distributed storage system - Google Patents
Online capacity expansion method and device for heterogeneous distributed storage system Download PDFInfo
- Publication number
- CN117081931B CN117081931B CN202311344093.5A CN202311344093A CN117081931B CN 117081931 B CN117081931 B CN 117081931B CN 202311344093 A CN202311344093 A CN 202311344093A CN 117081931 B CN117081931 B CN 117081931B
- Authority
- CN
- China
- Prior art keywords
- sub
- cluster
- capacity
- target
- allocation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003860 storage Methods 0.000 title claims abstract description 287
- 238000000034 method Methods 0.000 title claims abstract description 65
- 238000004590 computer program Methods 0.000 claims description 15
- 230000000694 effects Effects 0.000 claims description 11
- 238000012790 confirmation Methods 0.000 claims description 4
- 238000009826 distribution Methods 0.000 claims description 2
- 230000009545 invasion Effects 0.000 abstract 1
- 238000010586 diagram Methods 0.000 description 13
- 230000006870 function Effects 0.000 description 9
- 238000005516 engineering process Methods 0.000 description 8
- 230000006872 improvement Effects 0.000 description 8
- 230000008569 process Effects 0.000 description 8
- 238000012545 processing Methods 0.000 description 7
- 238000013500 data storage Methods 0.000 description 5
- 230000004044 response Effects 0.000 description 5
- 230000008859 change Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 229920001296 polysiloxane Polymers 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 239000010979 ruby Substances 0.000 description 1
- 229910001750 ruby Inorganic materials 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0896—Bandwidth or capacity management, i.e. automatically increasing or decreasing capacities
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0894—Policy-based network configuration management
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The specification discloses an online capacity expansion method and device for a heterogeneous distributed storage system, and relates to the technical field of storage capacity expansion, wherein the method comprises the following steps: the method comprises the steps of firstly organizing newly added storage nodes into a single distributed storage cluster, registering the single distributed storage cluster as a storage sub-cluster, then generating a pre-allocation result by a control module based on a capacity allocation rule and user storage requirements, then allocating user namespaces in a target storage sub-cluster according to the pre-allocation result, finally pushing and configuring the namespaces to an access gateway by the namespaces module, enabling the access gateway to be configured and effective, and distributing IO operation requests to the corresponding storage sub-clusters according to the user namespaces. The invention breaks the limitation of software and hardware, provides the storage service to the outside through the access gateway uniformly, solves the problem of business invasion, and achieves the purposes of integrating heterogeneous storage resources and providing uniform storage service.
Description
Technical Field
The present disclosure relates to the technical field of distributed storage systems, and in particular, to an online capacity expansion method and device for a heterogeneous distributed storage system.
Background
Under the drive of new technologies such as big data, cloud computing, artificial intelligence and the like, the data is fused with each industry, and the data volume is increased in an explosive manner. With the increasing number of scenarios for large-scale data storage and access, it is difficult for a single storage system to meet the requirements for large-scale data storage and access, and thus distributed storage systems are increasingly being widely used. As a key technology in the distributed storage field, the online capacity expansion technology is mainly aimed at adding hard disks and nodes into an existing distributed storage system under the condition of normally providing services so as to realize expansion of storage capacity and throughput.
According to the capacity expansion mode, the online capacity expansion technology of distributed storage can be divided into two types: horizontal expansion and vertical expansion. In contrast, horizontal expansion solves the problem of the upper limit of the specification of the machine, and the storage capacity and performance linearly increase with the number of nodes, and has become a technical hotspot in recent years. The existing hot distributed storage systems Ceph, minio, HDFS and the like all support horizontal capacity expansion, but most online capacity expansion methods including the above all face the following problems: 1. the software and hardware limitation is that the existing distributed storage system generally adopts the same set of software to organize storage nodes and has requirements on server configuration (CPU, memory, disk types, disk number and the like), but in the current data center, public cloud and other scenes, multi-manufacturer storage software and multi-type servers exist objectively, and the integration of heterogeneous storage resources and the provision of uniform storage services are the problems to be solved urgently at present. 2. The expansion is used for invading the service, and the horizontal expansion involves the problems of data splitting, data migration, routing change and the like, so that the online expansion needs to thoroughly shield the change of the underlying data storage, and the user transformation is avoided on the premise of ensuring the service stability and the storage performance, which is also a main challenge.
Disclosure of Invention
The present disclosure provides an online capacity expansion method and device for a heterogeneous distributed storage system, so as to partially solve the above-mentioned problems in the prior art.
The technical scheme adopted in the specification is as follows:
the specification provides an online capacity expansion method of a heterogeneous distributed storage system, which comprises the following steps:
responding to a capacity expansion request aiming at a distributed storage system, registering a target storage sub-cluster in the distributed storage system, wherein manufacturers corresponding to all devices in the target storage sub-cluster are unified;
performing capacity pre-allocation on the target storage sub-cluster according to a preset capacity allocation rule to obtain a pre-allocation result, wherein the pre-allocation result is used for representing the capacity condition of the target storage sub-cluster allocated to a user;
sending an allocation namespace instruction to the target storage sub-cluster according to the pre-allocation result, so that the target storage sub-cluster allocates a namespace for a user based on the allocation namespace instruction, and returns an allocation result, wherein the namespace is used for representing the storage space allocated to the user in the target storage sub-cluster;
based on the allocation result, if the IO request of the user for the target storage sub-cluster is determined to be received, the IO request is forwarded to the target storage sub-cluster.
Optionally, performing capacity pre-allocation on the target storage subset group according to a preset capacity allocation rule to obtain a pre-allocation result, which specifically includes:
determining capacity related parameters of other sub-clusters;
and determining other target sub-clusters and corresponding target users in the other target sub-clusters according to the capacity related parameters of the other sub-clusters based on the capacity allocation rule, and performing capacity pre-allocation for the target users aiming at the target storage sub-clusters to obtain a pre-allocation result.
Optionally, the capacity-related parameters of the other sub-clusters include: at least one of available capacity, used capacity, and QPS indicator.
Optionally, determining the other sub-clusters of the target according to the capacity related parameters of the other sub-clusters specifically includes:
taking other sub-clusters with the available capacity ratio not meeting the preset ratio as target other sub-clusters; and/or
And taking other sub-clusters with the QPS index not smaller than a preset threshold value as target other sub-clusters.
Optionally, determining the corresponding target user in the other target sub-clusters specifically includes:
determining users with activity higher than preset activity in other target sub-clusters as target users, wherein the activity comprises at least one of QOS and capacity growth rate; and/or
And determining the users with the capacity demand degree higher than the preset capacity demand degree in the other target sub-clusters as target users.
Optionally, responding to the capacity expansion request for the distributed storage system specifically includes:
responding to an expansion request of an administrator for the distributed storage system;
performing capacity pre-allocation on the target storage subset group according to a preset capacity allocation rule, and after obtaining a pre-allocation result, the method further comprises:
and returning the pre-allocation result to the administrator, and responding to a confirmation instruction sent by the administrator based on the pre-allocation result, and sending an allocation namespace instruction to the target storage subset group according to the pre-allocation result.
Optionally, if it is determined that an IO instruction for the target storage sub-cluster is received by a user, forwarding the IO instruction to the target storage sub-cluster specifically includes:
and configuring the allocation result at an access gateway so as to determine that when an IO request of a user for a name space in the target storage sub-cluster is received through the access gateway, the IO request is forwarded to the target storage sub-cluster.
The present specification provides an online capacity expansion device of a heterogeneous distributed storage system, including:
the control module is used for responding to the capacity expansion request aiming at the distributed storage system, registering a target storage sub-cluster in the distributed storage system, and the manufacturers corresponding to all the devices in the target storage sub-cluster are unified;
the control module is used for carrying out capacity pre-allocation on the target storage sub-cluster according to a preset capacity allocation rule to obtain a pre-allocation result, wherein the pre-allocation result is used for representing the capacity condition of the target storage sub-cluster allocated to a user;
the naming space module is used for sending an allocation naming space instruction to the target storage sub-cluster according to the pre-allocation result, so that the target storage sub-cluster allocates a naming space for a user based on the allocation naming space instruction, returns an allocation result, and the naming space is used for representing the storage space allocated to the user in the target storage sub-cluster;
and the access gateway module is used for forwarding the IO request to the target storage sub-cluster if the IO request of the user for the target storage sub-cluster is determined to be received based on the distribution result.
The present specification provides a computer readable storage medium storing a computer program which when executed by a processor implements the above-described heterogeneous distributed storage system online capacity expansion method.
The present disclosure provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the above-mentioned method for online capacity expansion of a heterogeneous distributed storage system when executing the program.
The above-mentioned at least one technical scheme that this specification adopted can reach following beneficial effect:
according to the online capacity expansion method of the heterogeneous distributed storage system, the target storage sub-cluster is registered in the distributed storage system in response to the capacity expansion request aiming at the distributed storage system, manufacturers corresponding to all devices in the target storage sub-cluster are unified, capacity pre-allocation can be carried out on the target storage sub-cluster according to a preset capacity allocation rule to obtain a pre-allocation result, the pre-allocation result is used for representing the capacity condition in the target storage sub-cluster allocated to the user, a naming space allocation instruction is sent to the target storage sub-cluster according to the pre-allocation result, so that the target storage sub-cluster allocates a naming space for the user based on the naming space allocation instruction, and an allocation result is returned, wherein the naming space is used for representing the storage space allocated to the user in the target storage sub-cluster; based on the allocation result, if the IO request of the user for the target storage sub-cluster is determined to be received, the IO request is forwarded to the target storage sub-cluster.
As can be seen from the above, in the distributed storage system in the method, there may be multiple storage sub-clusters, and when one storage sub-cluster needs to be expanded to enter the distributed storage system, a storage space in the storage sub-cluster may be allocated to a user according to the above method, so when an IO request of the user for the storage space in the storage sub-cluster is received, the IO request is forwarded to the storage sub-cluster to be processed. Therefore, the method can maintain the storage sub-clusters using different manufacturer devices, and the different storage sub-clusters can adopt respective corresponding storage software to store and process own data.
The method reduces maintenance and management cost of users and service loss caused by suspending service while ensuring online expansion of the distributed storage system; the online capacity expansion is realized through the newly added storage subsystem, the old data is not influenced, the rapid expansion and the reduction of the distributed storage system are ensured, the storage capacity can be flexibly allocated according to the service requirement, and the method has high expansibility and flexibility; and the load is shared by expanding the new storage sub-cluster, so that the single-point fault risk is reduced, and the overall performance and throughput of the system are improved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the specification, illustrate and explain the exemplary embodiments of the present specification and their description, are not intended to limit the specification unduly. In the drawings:
FIG. 1 is a schematic flow chart of an online capacity expansion method of a heterogeneous distributed storage system provided in the present specification;
FIG. 2 is a schematic diagram of steps for capacity pre-allocation provided in the present specification;
FIG. 3 is a schematic diagram of a distributed storage system according to the present disclosure;
FIG. 4 is a schematic diagram of an online capacity expansion device of a heterogeneous distributed storage system provided in the present specification;
fig. 5 is a schematic view of the electronic device corresponding to fig. 1 provided in the present specification.
Detailed Description
For the purposes of making the objects, technical solutions and advantages of the present specification more apparent, the technical solutions of the present specification will be clearly and completely described below with reference to specific embodiments of the present specification and corresponding drawings. It will be apparent that the described embodiments are only some, but not all, of the embodiments of the present specification. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are intended to be within the scope of the present disclosure.
The following describes in detail the technical solutions provided by the embodiments of the present specification with reference to the accompanying drawings.
Fig. 1 is a schematic flow chart of an online capacity expansion method of a heterogeneous distributed storage system provided in the present specification, which specifically includes the following steps:
s100: and in response to the capacity expansion request aiming at the distributed storage system, registering the target storage sub-cluster in the distributed storage system, wherein vendors corresponding to all devices in the target storage sub-cluster are unified.
S102: and performing capacity pre-allocation on the target storage sub-cluster according to a preset capacity allocation rule to obtain a pre-allocation result, wherein the pre-allocation result is used for representing the capacity condition of the target storage sub-cluster allocated to a user.
S104: and responding to the received allocation instruction, sending an allocation namespace instruction to the target storage sub-cluster according to the pre-allocation result, so that the target storage sub-cluster allocates a namespace for a user based on the allocation namespace instruction, and returning an allocation result, wherein the namespace is used for representing the storage space allocated to the user in the target storage sub-cluster.
S106: based on the allocation result, if the IO request of the user for the target storage sub-cluster is determined to be received, the IO request is forwarded to the target storage sub-cluster.
In this specification, the online capacity expansion of the distributed storage system may be performed, where the distributed storage system may include a plurality of storage sub-clusters, where vendors corresponding to devices of one storage sub-cluster are identical, and vendors corresponding to devices used by different storage sub-clusters may be different, that is, one storage sub-cluster may use an identical device, so that one storage sub-cluster may be maintained by using storage software provided by the vendor corresponding to the device of the storage sub-cluster.
It should be noted that, the distributed storage system may provide a user with a storage service, and the user may perform online data storage through the distributed storage system, where the distributed storage system may uniformly manage each storage sub-cluster, and since devices of different storage sub-clusters may be different, storage software adopted by different storage sub-clusters may be different, so that the distributed storage system needs to schedule an IO request (a request for storing and reading) of the user to be distributed to different storage sub-clusters.
Assume that M storage sub-clusters are registered in the distributed storage system, and in this case, expansion needs to be performed on the distributed storage system (it should be noted that, the M storage sub-clusters may also be expanded by the following online expansion method to enter the distributed storage system), and the target storage sub-cluster is added into the distributed storage system.
Based on this, when the distributed storage system performs capacity expansion, the target storage sub-cluster may be registered in the distributed storage system in response to a capacity expansion request for the distributed storage system, where the registration may refer to recording an identification, a resource condition, a device condition, and the like of the target storage sub-cluster in the distributed storage system.
And then, performing capacity pre-allocation on the target storage sub-cluster according to a preset capacity allocation rule to obtain a pre-allocation result, wherein the pre-allocation result is used for representing the capacity condition of the target storage sub-cluster allocated to the user.
The capacity allocation rule may be set according to actual requirements, specifically, capacity related parameters of other sub-clusters may be determined, based on the capacity allocation rule, a target other sub-cluster is determined according to the capacity related parameters of the other sub-clusters, a corresponding target user in the target other sub-cluster is determined, and capacity pre-allocation is performed for the target user for the target storage sub-cluster, so as to obtain a pre-allocation result, as shown in fig. 2.
Fig. 2 is a schematic diagram of steps for capacity pre-allocation provided in the present specification.
The other sub-clusters mentioned above may refer to storage sub-clusters that have been registered in the distributed storage system, the capacity-related parameters of the other sub-clusters may refer to available capacity, used capacity, QPS indexes, and the like, and the capacity-related parameters of corresponding users in each other sub-cluster (e.g., the available capacity, used capacity, QPS indexes, and the like of the users in the distributed storage system) may also be determined, and the corresponding users in the other sub-clusters may refer to users that have storage spaces of the other sub-clusters.
That is, the above-described manner is to determine whether a new namespace needs to be allocated for the user. That is, some users have stored a certain amount of data in other existing sub-clusters, but the capacity level of other sub-clusters storing data may be urgent, and assuming that the capacity perceived by the user to be owned in the distributed storage system is 100T, and 50T of capacity storage data is used in other sub-clusters, a 50T namespace may be allocated for the user in the target storage sub-cluster, and data storage may be reserved for the user.
The following is an example of selecting other sub-clusters of the target and the target user by the capacity allocation rule: other sub-clusters with the available capacity ratio not meeting the preset ratio can be used as target other sub-clusters; or other sub-clusters with the QPS index not smaller than a preset threshold are used as target other sub-clusters. Users with activity higher than preset activity in other target sub-clusters can be used as target users; and/or users with capacity demand higher than the preset capacity demand in other target sub-clusters are used as target users, wherein the activity degree includes QOS and capacity increase speed.
For example, if the capacity water level of the storage sub-cluster a is found to be greater than 50%, the storage sub-cluster a may be used as a target other storage sub-cluster, and the capacity may be allocated to the user 1 with the highest capacity demand in the storage sub-cluster a, and the capacity of 15TB in the target storage sub-cluster may be allocated to the user 1 according to the available capacity and the remaining capacity of the user 1 in the distributed storage system.
And then, in response to the received allocation instruction, sending an allocation namespace instruction to the target storage sub-cluster according to the pre-allocation result, so that the target storage sub-cluster allocates a namespace for a user based on the allocation namespace instruction, and returns an allocation result, wherein the namespace is used for representing the storage space allocated to the user in the target storage sub-cluster. Then, based on the allocation result, if it is determined that the IO request of the user for the target storage sub-cluster is received, the IO request is forwarded to the target storage sub-cluster.
That is, the capacity pre-allocation is to determine how much capacity needs to be allocated to which users in the target storage sub-cluster, and the target storage sub-cluster allocates namespaces for users, which may mean that the target storage sub-cluster establishes a specific correspondence between namespaces (storage spaces) in the sub-cluster and users, for example, needs to allocate 15TB capacity for user 1 in the target storage sub-cluster, then the target storage sub-cluster may determine a namespaces ns3 of 15TB capacity in the target storage sub-cluster, allocate the namespaces ns3 to user 1, and record the correspondence between namespaces ns3 and user 1.
Then, the target storage sub-cluster may return the allocation result recorded with the correspondence between the namespace ns3 and the user 1 to the distributed storage system, so that when the distributed storage system receives the IO request of the user 1 for the namespace ns3, the IO request may be forwarded to the target storage sub-cluster, and the target storage sub-cluster may perform data processing (data reading, data writing, data deleting, etc.) for the user 1 according to the IO request.
The allocation result (naming space allocation result) may be pushed to the access gateway, so that when the access gateway determines that an IO request of a user for a naming space in the target storage sub-cluster is received, the IO request is forwarded to the target storage sub-cluster.
It should be noted that, the administrator may control the capacity expansion of the distributed storage system by sending an instruction through its own terminal or client, that is, when the administrator sends a capacity expansion request for the distributed storage system, the distributed storage system may start to perform steps such as registration of the target storage sub-cluster, and may return the pre-allocation result to the administrator, determine that the administrator sends a confirmation instruction based on the pre-allocation result, and then send an instruction for allocating a namespace to the target storage sub-cluster according to the pre-allocation result.
And then an administrator performs the allocation result of the naming space through the target storage sub-cluster, after determining that the target storage sub-cluster finishes the allocation of the naming space, the administrator can configure the access gateway according to the allocation result, so that the access gateway determines the corresponding relation among the user, the naming space and the target storage sub-cluster, and the corresponding relation is forwarded to the target storage sub-cluster when the user sends an IO request aiming at the naming space in the target storage sub-cluster.
It should be noted that the access gateway may be a software module in a distributed storage system, and will be referred to as an access gateway module in the following.
For convenience of description, the execution body for executing the method is described as a distributed storage system, and the execution body of the method may be a server, a large service platform, or the like, which is not limited herein.
As can be seen from the above, in the distributed storage system in the method, there may be multiple storage sub-clusters, and when one storage sub-cluster needs to be expanded to enter the distributed storage system, a storage space in the storage sub-cluster may be allocated to a user according to the above method, so when an IO request of the user for the storage space in the storage sub-cluster is received, the IO request is forwarded to the storage sub-cluster to be processed. Therefore, the method can maintain the storage sub-clusters using different manufacturer devices, and the different storage sub-clusters can adopt respective corresponding storage software to store and process own data.
The method reduces maintenance and management cost of users and service loss caused by suspending service while ensuring online expansion of the distributed storage system; the online capacity expansion is realized through the newly added storage subsystem, the old data is not influenced, the rapid expansion and the reduction of the distributed storage system are ensured, the storage capacity can be flexibly allocated according to the service requirement, and the method has high expansibility and flexibility; and the load is shared by expanding the new storage sub-cluster, so that the single-point fault risk is reduced, and the overall performance and throughput of the system are improved.
Fig. 3 is a schematic structural diagram of a distributed storage system provided in the present specification, which can be understood by combining the following on-line capacity expansion devices of heterogeneous distributed storage systems.
In fig. 3, the distributed storage system includes an a cluster, a B cluster and a C cluster, where the C cluster is a storage sub-cluster that is newly expanded, and a namespace ns3 in the C cluster is allocated to a user 1 through pre-allocation and allocation of namespaces, so when an IO request (including a request for data reading, data writing, etc.) about the ns3 namespace of the user 1 is received, the IO request may be forwarded to the C cluster, so that the C cluster processes through internal storage software.
The above method for online capacity expansion of a heterogeneous distributed storage system provided in one or more embodiments of the present disclosure, based on the same concept, further provides an online capacity expansion device of a heterogeneous distributed storage system, as shown in fig. 4.
Fig. 4 is a schematic diagram of an online capacity expansion device of a heterogeneous distributed storage system provided in the present specification, including:
a control module 401, configured to register a target storage sub-cluster in a distributed storage system in response to a capacity expansion request for the distributed storage system, where vendors corresponding to devices in the target storage sub-cluster are unified;
the control module 401 is configured to pre-allocate the capacity of the target storage sub-cluster according to a preset capacity allocation rule, so as to obtain a pre-allocation result, where the pre-allocation result is used to represent a capacity condition in the target storage sub-cluster allocated to a user;
a namespace module 402, configured to send an allocation namespace instruction to the target storage sub-cluster according to the pre-allocation result, so that the target storage sub-cluster allocates a namespace for a user based on the allocation namespace instruction, and returns an allocation result, where the namespace is used to represent a storage space allocated to the user in the target storage sub-cluster;
and the access gateway module 403 is configured to forward the IO request to the target storage sub-cluster if it is determined that the IO request of the user for the target storage sub-cluster is received based on the allocation result.
Optionally, the control module 401 is specifically configured to determine a capacity related parameter of another sub-cluster, determine, based on the capacity allocation rule, another sub-cluster of the target and a corresponding target user in the other sub-cluster of the target according to the capacity related parameter of the other sub-cluster, and pre-allocate the capacity of the target user for the target storage sub-cluster, so as to obtain a pre-allocation result.
Optionally, the control module 401 is specifically configured to include: at least one of available capacity, used capacity, and QPS indicator.
Optionally, the control module 401 is specifically configured to use, as the target other sub-cluster, the other sub-clusters whose available capacity ratio does not meet the preset ratio; and/or taking other sub-clusters with the QPS index not smaller than a preset threshold value as target other sub-clusters.
Optionally, the control module 401 is specifically configured to determine, as the target user, a user whose activity level in the target other sub-cluster is higher than a preset activity level; and/or determining the users with the capacity demand degree higher than the preset capacity demand degree in the other target sub-clusters as target users.
Optionally, the control module 401 is specifically configured to respond to a capacity expansion request of an administrator for the distributed storage system;
and performing capacity pre-allocation on the target storage subset group according to a preset capacity allocation rule, wherein after a pre-allocation result is obtained, the control module 401 is further configured to return the pre-allocation result to the administrator, and respond to a confirmation instruction sent by the administrator based on the pre-allocation result, send an allocation namespace instruction to the target storage subset group according to the pre-allocation result.
Optionally, the access gateway module 403 is specifically configured to configure the allocation result at an access gateway, so as to determine, through the access gateway, that when an IO request of a user for a namespace in the target storage sub-cluster is received, forward the IO request to the target storage sub-cluster.
The present specification also provides a computer readable storage medium storing a computer program operable to perform the above-described heterogeneous distributed storage system online capacity expansion method.
The present specification also provides a schematic structural diagram of the electronic device shown in fig. 5. At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile storage, as illustrated in fig. 5, although other hardware required by other services may be included. The processor reads the corresponding computer program from the nonvolatile memory to the memory and then operates the memory to realize the on-line capacity expansion method of the heterogeneous distributed storage system.
Of course, other implementations, such as logic devices or combinations of hardware and software, are not excluded from the present description, that is, the execution subject of the following processing flows is not limited to each logic unit, but may be hardware or logic devices.
In the 90 s of the 20 th century, improvements to one technology could clearly be distinguished as improvements in hardware (e.g., improvements to circuit structures such as diodes, transistors, switches, etc.) or software (improvements to the process flow). However, with the development of technology, many improvements of the current method flows can be regarded as direct improvements of hardware circuit structures. Designers almost always obtain corresponding hardware circuit structures by programming improved method flows into hardware circuits. Therefore, an improvement of a method flow cannot be said to be realized by a hardware entity module. For example, a programmable logic device (Programmable Logic Device, PLD) (e.g., field programmable gate array (Field Programmable Gate Array, FPGA)) is an integrated circuit whose logic function is determined by the programming of the device by a user. A designer programs to "integrate" a digital system onto a PLD without requiring the chip manufacturer to design and fabricate application-specific integrated circuit chips. Moreover, nowadays, instead of manually manufacturing integrated circuit chips, such programming is mostly implemented by using "logic compiler" software, which is similar to the software compiler used in program development and writing, and the original code before the compiling is also written in a specific programming language, which is called hardware description language (Hardware Description Language, HDL), but not just one of the hdds, but a plurality of kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), lava, lola, myHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog are currently most commonly used. It will also be apparent to those skilled in the art that a hardware circuit implementing the logic method flow can be readily obtained by merely slightly programming the method flow into an integrated circuit using several of the hardware description languages described above.
The controller may be implemented in any suitable manner, for example, the controller may take the form of, for example, a microprocessor or processor and a computer readable medium storing computer readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable logic controllers, and embedded microcontrollers, examples of which include, but are not limited to, the following microcontrollers: ARC 625D, atmel AT91SAM, microchip PIC18F26K20, and Silicone Labs C8051F320, the memory controller may also be implemented as part of the control logic of the memory. Those skilled in the art will also appreciate that, in addition to implementing the controller in a pure computer readable program code, it is well possible to implement the same functionality by logically programming the method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. Such a controller may thus be regarded as a kind of hardware component, and means for performing various functions included therein may also be regarded as structures within the hardware component. Or even means for achieving the various functions may be regarded as either software modules implementing the methods or structures within hardware components.
The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. One typical implementation is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
For convenience of description, the above devices are described as being functionally divided into various units, respectively. Of course, the functions of each element may be implemented in one or more software and/or hardware elements when implemented in the present specification.
It will be appreciated by those skilled in the art that embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, the present specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present description can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
The present description is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the specification. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.
It will be appreciated by those skilled in the art that embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, the present specification may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present description can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
The description may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.
The foregoing is merely exemplary of the present disclosure and is not intended to limit the disclosure. Various modifications and alterations to this specification will become apparent to those skilled in the art. Any modifications, equivalent substitutions, improvements, or the like, which are within the spirit and principles of the present description, are intended to be included within the scope of the claims of the present description.
Claims (10)
1. An online capacity expansion method of a heterogeneous distributed storage system is characterized by comprising the following steps:
responding to a capacity expansion request aiming at a distributed storage system, registering a target storage sub-cluster in the distributed storage system, wherein manufacturers corresponding to all devices in the target storage sub-cluster are unified;
performing capacity pre-allocation on the target storage sub-cluster according to a preset capacity allocation rule to obtain a pre-allocation result, wherein the pre-allocation result is used for representing the capacity condition of the target storage sub-cluster allocated to a user;
sending an allocation namespace instruction to the target storage sub-cluster according to the pre-allocation result, so that the target storage sub-cluster allocates a namespace for a user based on the allocation namespace instruction, and returns an allocation result, wherein the namespace is used for representing the storage space allocated to the user in the target storage sub-cluster;
based on the allocation result, if the IO request of the user for the target storage sub-cluster is determined to be received, the IO request is forwarded to the target storage sub-cluster.
2. The method of claim 1, wherein the pre-allocation of the target storage subset group according to a preset capacity allocation rule to obtain a pre-allocation result specifically comprises:
determining capacity related parameters of other sub-clusters;
and determining other target sub-clusters and corresponding target users in the other target sub-clusters according to the capacity related parameters of the other sub-clusters based on the capacity allocation rule, and performing capacity pre-allocation for the target users aiming at the target storage sub-clusters to obtain a pre-allocation result.
3. The method of claim 2, wherein the capacity-related parameters of the other sub-clusters comprise: at least one of available capacity, used capacity, and QPS indicator.
4. A method according to claim 3, wherein determining the target other sub-clusters based on the capacity-related parameters of the other sub-clusters comprises:
taking other sub-clusters with the available capacity ratio not meeting the preset ratio as target other sub-clusters; and/or
And taking other sub-clusters with the QPS index not smaller than a preset threshold value as target other sub-clusters.
5. The method of claim 2, wherein determining the corresponding target user in the target other sub-cluster specifically comprises:
determining users with activity higher than preset activity in other target sub-clusters as target users, wherein the activity comprises at least one of QOS and capacity growth rate; and/or
And determining the users with the capacity demand degree higher than the preset capacity demand degree in the other target sub-clusters as target users.
6. The method of claim 1, wherein responding to the capacity expansion request for the distributed storage system comprises:
responding to an expansion request of an administrator for the distributed storage system;
performing capacity pre-allocation on the target storage subset group according to a preset capacity allocation rule, and after obtaining a pre-allocation result, the method further comprises:
and returning the pre-allocation result to the administrator, and responding to a confirmation instruction sent by the administrator based on the pre-allocation result, and sending an allocation namespace instruction to the target storage subset group according to the pre-allocation result.
7. The method of claim 1, wherein based on the allocation result, if it is determined that an IO instruction for the target storage sub-cluster is received by a user, forwarding the IO instruction to the target storage sub-cluster specifically comprises:
and configuring the allocation result at an access gateway so as to determine that when an IO request of a user for a name space in the target storage sub-cluster is received through the access gateway, the IO request is forwarded to the target storage sub-cluster.
8. An online capacity expansion device of a heterogeneous distributed storage system, comprising:
the control module is used for responding to the capacity expansion request aiming at the distributed storage system, registering a target storage sub-cluster in the distributed storage system, and the manufacturers corresponding to all the devices in the target storage sub-cluster are unified;
the control module is used for carrying out capacity pre-allocation on the target storage sub-cluster according to a preset capacity allocation rule to obtain a pre-allocation result, wherein the pre-allocation result is used for representing the capacity condition of the target storage sub-cluster allocated to a user;
the naming space module is used for sending an allocation naming space instruction to the target storage sub-cluster according to the pre-allocation result, so that the target storage sub-cluster allocates a naming space for a user based on the allocation naming space instruction, returns an allocation result, and the naming space is used for representing the storage space allocated to the user in the target storage sub-cluster;
and the access gateway module is used for forwarding the IO request to the target storage sub-cluster if the IO request of the user for the target storage sub-cluster is determined to be received based on the distribution result.
9. A computer readable storage medium, characterized in that the storage medium stores a computer program which, when executed by a processor, implements the method of any of the preceding claims 1-7.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of any of the preceding claims 1-7 when executing the program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311344093.5A CN117081931B (en) | 2023-10-17 | 2023-10-17 | Online capacity expansion method and device for heterogeneous distributed storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311344093.5A CN117081931B (en) | 2023-10-17 | 2023-10-17 | Online capacity expansion method and device for heterogeneous distributed storage system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117081931A CN117081931A (en) | 2023-11-17 |
| CN117081931B true CN117081931B (en) | 2024-01-09 |
Family
ID=88713866
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311344093.5A Active CN117081931B (en) | 2023-10-17 | 2023-10-17 | Online capacity expansion method and device for heterogeneous distributed storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117081931B (en) |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017189620A1 (en) * | 2016-04-25 | 2017-11-02 | Netlist, Inc. | Method and apparatus for uniform memory access in a storage cluster |
| CN108287669A (en) * | 2018-01-26 | 2018-07-17 | 平安科技（深圳）有限公司 | Date storage method, device and storage medium |
| CN110275681A (en) * | 2019-06-27 | 2019-09-24 | 深圳前海微众银行股份有限公司 | A kind of date storage method and data-storage system |
| US10656862B1 (en) * | 2017-05-02 | 2020-05-19 | Amzetta Technologies, Llc | Systems and methods for implementing space consolidation and space expansion in a horizontally federated cluster |
| CN112162707A (en) * | 2020-10-15 | 2021-01-01 | 科大讯飞股份有限公司 | Storage method, electronic device and storage medium for distributed storage system |
| RU2019123157A (en) * | 2019-07-23 | 2021-01-26 | ИЭмСи АйПи ХОЛДИНГ КОМПАНИ, ЛЛС | SYSTEM AND METHOD OF NETWORK IDENTIFICATION ARCHITECTURE FOR CLUSTER AND FEDERATIVE STORAGE SYSTEMS |
| CN112463372A (en) * | 2020-11-23 | 2021-03-09 | 中国建设银行股份有限公司 | Target system capacity expansion method, device, equipment and storage medium |
| CN113505177A (en) * | 2021-07-08 | 2021-10-15 | 中国工商银行股份有限公司 | Distributed system cluster online fragmentation and capacity expansion method and device |
| CN113918647A (en) * | 2021-09-18 | 2022-01-11 | 上海浦东发展银行股份有限公司 | Distributed database elastic expansion method, device, equipment and storage medium |
| CN114089917A (en) * | 2021-11-19 | 2022-02-25 | 中国电信集团***集成有限责任公司 | Distributed object storage cluster, capacity expansion method and device thereof, and electronic equipment |
| CN114721719A (en) * | 2022-04-20 | 2022-07-08 | 上海道客网络科技有限公司 | Method and system for containerized deployment of heterogeneous applications in cluster |
| CN116418826A (en) * | 2022-11-25 | 2023-07-11 | 中移(苏州)软件技术有限公司 | Object storage system capacity expansion method, device and system and computer equipment |
| CN116483605A (en) * | 2023-04-20 | 2023-07-25 | 中国工商银行股份有限公司 | Data processing method, device, system, storage medium and electronic equipment |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9805054B2 (en) * | 2011-11-14 | 2017-10-31 | Panzura, Inc. | Managing a global namespace for a distributed filesystem |
| US10503404B2 (en) * | 2017-10-23 | 2019-12-10 | Micron Technology, Inc. | Namespace management in non-volatile memory devices |
-
2023
- 2023-10-17 CN CN202311344093.5A patent/CN117081931B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017189620A1 (en) * | 2016-04-25 | 2017-11-02 | Netlist, Inc. | Method and apparatus for uniform memory access in a storage cluster |
| US10656862B1 (en) * | 2017-05-02 | 2020-05-19 | Amzetta Technologies, Llc | Systems and methods for implementing space consolidation and space expansion in a horizontally federated cluster |
| CN108287669A (en) * | 2018-01-26 | 2018-07-17 | 平安科技（深圳）有限公司 | Date storage method, device and storage medium |
| CN110275681A (en) * | 2019-06-27 | 2019-09-24 | 深圳前海微众银行股份有限公司 | A kind of date storage method and data-storage system |
| RU2019123157A (en) * | 2019-07-23 | 2021-01-26 | ИЭмСи АйПи ХОЛДИНГ КОМПАНИ, ЛЛС | SYSTEM AND METHOD OF NETWORK IDENTIFICATION ARCHITECTURE FOR CLUSTER AND FEDERATIVE STORAGE SYSTEMS |
| CN112162707A (en) * | 2020-10-15 | 2021-01-01 | 科大讯飞股份有限公司 | Storage method, electronic device and storage medium for distributed storage system |
| CN112463372A (en) * | 2020-11-23 | 2021-03-09 | 中国建设银行股份有限公司 | Target system capacity expansion method, device, equipment and storage medium |
| CN113505177A (en) * | 2021-07-08 | 2021-10-15 | 中国工商银行股份有限公司 | Distributed system cluster online fragmentation and capacity expansion method and device |
| CN113918647A (en) * | 2021-09-18 | 2022-01-11 | 上海浦东发展银行股份有限公司 | Distributed database elastic expansion method, device, equipment and storage medium |
| CN114089917A (en) * | 2021-11-19 | 2022-02-25 | 中国电信集团***集成有限责任公司 | Distributed object storage cluster, capacity expansion method and device thereof, and electronic equipment |
| CN114721719A (en) * | 2022-04-20 | 2022-07-08 | 上海道客网络科技有限公司 | Method and system for containerized deployment of heterogeneous applications in cluster |
| CN116418826A (en) * | 2022-11-25 | 2023-07-11 | 中移(苏州)软件技术有限公司 | Object storage system capacity expansion method, device and system and computer equipment |
| CN116483605A (en) * | 2023-04-20 | 2023-07-25 | 中国工商银行股份有限公司 | Data processing method, device, system, storage medium and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117081931A (en) | 2023-11-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10235047B2 (en) | Memory management method, apparatus, and system | |
| US10356150B1 (en) | Automated repartitioning of streaming data | |
| CN110597614B (en) | Resource adjustment method and device | |
| CN110955720B (en) | Data loading method, device and system | |
| CN108845876B (en) | Service distribution method and device | |
| CN116305298B (en) | Method and device for managing computing power resources, storage medium and electronic equipment | |
| CN116048800A (en) | Data processing method and device, storage medium and electronic equipment | |
| CN115328924B (en) | Method and device for creating table space, storage medium and electronic equipment | |
| CN114594914A (en) | Control method and system for distributed storage system | |
| US8443369B1 (en) | Method and system for dynamically selecting a best resource from each resource collection based on resources dependencies, prior selections and statistics to implement an allocation policy | |
| JP2015114913A (en) | Storage device, storage system, and data management program | |
| CN117081931B (en) | Online capacity expansion method and device for heterogeneous distributed storage system | |
| CN115964181B (en) | Data processing method and device, storage medium and electronic equipment | |
| CN116737345A (en) | Distributed task processing system, distributed task processing method, distributed task processing device, storage medium and storage device | |
| TWI706343B (en) | Sample playback data access method, device and computer equipment | |
| CN115604294A (en) | Method and device for managing storage resources | |
| CN114780201A (en) | Resource adjusting method and device, electronic equipment and storage medium | |
| CN114296891A (en) | Task scheduling method, system, computing device, storage medium and program product | |
| CN110083680B (en) | Method and device for managing context data in distributed system | |
| CN115827745A (en) | Memory database cluster and implementation method and device thereof | |
| CN117041980B (en) | Network element management method and device, storage medium and electronic equipment | |
| CN117555697B (en) | Distributed training-oriented cache loading system, method, device and equipment | |
| KR101594555B1 (en) | Method for processing distributed prosonalized data and cloud server | |
| CN116185576A (en) | Application migration method and device, electronic equipment and storage medium | |
| CN117093161B (en) | Memory management system, method, medium and equipment based on optical transceiver chip |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |