CN107729185A - A kind of fault handling method and device - Google Patents

A kind of fault handling method and device Download PDF

Info

Publication number
CN107729185A
CN107729185A CN201711015125.1A CN201711015125A CN107729185A CN 107729185 A CN107729185 A CN 107729185A CN 201711015125 A CN201711015125 A CN 201711015125A CN 107729185 A CN107729185 A CN 107729185A
Authority
CN
China
Prior art keywords
osd
target storage
storage node
node
threshold value
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201711015125.1A
Other languages
Chinese (zh)
Other versions
CN107729185B (en
Inventor
顾雷雷
乔辉
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
New H3C Technologies Co Ltd
Original Assignee
New H3C Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by New H3C Technologies Co Ltd filed Critical New H3C Technologies Co Ltd
Priority to CN201711015125.1A priority Critical patent/CN107729185B/en
Publication of CN107729185A publication Critical patent/CN107729185A/en
Application granted granted Critical
Publication of CN107729185B publication Critical patent/CN107729185B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14Error detection or correction of the data by redundancy in operation
    • G06F11/1402Saving, restoring, recovering or retrying
    • G06F11/1446Point-in-time backing up or restoration of persistent data
    • G06F11/1458Management of the backup or restore process
    • G06F11/1464Management of the backup or restore process for networked environments
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/30Monitoring
    • G06F11/3003Monitoring arrangements specially adapted to the computing system or computing system component being monitored
    • G06F11/3034Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a storage system, e.g. DASD based or network based

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Quality & Reliability (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The present invention, which provides a kind of fault handling method and device, this method, to be included:Obtain the free memory capacity of memory node and the information of object storage device OSD quantity;When it is determined that the free memory capacity of target storage node and the ratio of OSD quantity are less than the first predetermined threshold value, OSD increase operation of the refusal for the target storage node, and the part OSD of the target storage node is arranged to first kind inoperative Down states, so that the free memory capacity of the target storage node and the ratio of remaining OSD quantity are more than or equal to first predetermined threshold value;Wherein, remaining described OSD refers to the available OSD in addition to the part OSD for being arranged to first kind Down states in the OSD of the target storage node.The risk for being likely to occur OSD performance bottlenecks and data recovery failure on memory node after memory failure can be evaded using the present invention.

Description

A kind of fault handling method and device
Technical field
The present invention relates to network communication technology field, more particularly to a kind of fault handling method and device.
Background technology
Ceph (distributed memory system) is an open source projects, there is provided software definition, unified storage solution, Possess can extend on a large scale, high-performance, without Single Point of Faliure the advantages of.
One typical Ceph clustered deploy(ment), can be that each piece of physical hard disk creates an OSD in clustered node (Object Storage Device, object storage device).
The failure domain of Ceph clusters generally includes disk, node (i.e. server), frame, power circuit etc..Work as failure domain After interior any component failure, the corresponding OSD that can cause to be deployed in thereon can break down, and Ceph clusters can be by these OSD is labeled as Down (inoperative) state, and carries out initialization operation, reorganizes number impacted on the node to break down According to.
But practice is found, in existing Ceph clusters implementation, because the partial memory of memory node is unavailable simultaneously OSD state will not be directly affected, therefore, when the partial memory of memory node is unavailable, Ceph clusters can't be to OSD's State is adjusted, if but now data recovery or releveling event occur for Ceph clusters, it can have a strong impact on Ceph clusters Process performance, even result in data recovery failure situation occur.
The content of the invention
The present invention provides a kind of fault handling method and device, may to solve memory node memory failure in the prior art Ceph cluster process performances can be reduced, even result in the problem of Ceph company-datas recover failure.
First aspect according to embodiments of the present invention, there is provided a kind of fault handling method, this method include:
Obtain the free memory capacity of memory node and the information of object storage device OSD quantity;
When it is determined that the free memory capacity of target storage node and the ratio of OSD quantity are less than first in the memory node During predetermined threshold value, refusal for the target storage node OSD increase operations, and by the part of the target storage node OSD is arranged to first kind inoperative Down states, so that the free memory capacity of the target storage node and remaining OSD number The ratio of amount is more than or equal to first predetermined threshold value;Wherein, remaining described OSD refers in the OSD of the target storage node Available OSD in addition to the part OSD for being arranged to first kind Down states.
Second aspect according to embodiments of the present invention, there is provided a kind of fault handling method, this method include:
Obtain the information of itself free memory capacity and object storage object equipment OSD quantity;
The information of itself free memory capacity and OSD quantity is reported to monitor.
The third aspect according to embodiments of the present invention, there is provided a kind of fault treating apparatus, the device include:
Acquiring unit, for obtaining the free memory capacity of memory node and the information of object storage device OSD quantity;
Determining unit, for determining that memory size and object in the memory node with the presence or absence of target storage node are deposited The ratio of storage equipment OSD quantity is less than the first predetermined threshold value;
Processing unit, for determining that the free memory of target storage node in the memory node holds when the determining unit When the ratio of amount and OSD quantity is less than the first predetermined threshold value, refusal increases operation for the OSD of the target storage node, and The part OSD of the target storage node is arranged to first kind inoperative Down states, so that the target storage node Free memory capacity and the ratio of remaining OSD quantity be more than or equal to first predetermined threshold value;Wherein, remaining described OSD is Refer to the available OSD in addition to the part OSD for being arranged to first kind Down states in the OSD of the target storage node.
Fourth aspect according to embodiments of the present invention, there is provided a kind of fault treating apparatus, the device include:
Acquiring unit, for obtaining the information of itself free memory capacity and object storage object equipment OSD quantity;
Transmitting element, for reporting the information of itself free memory capacity and OSD quantity to monitor.
Using the embodiment of the present invention, the first default threshold of the memory source deficiency by being provided for indicating memory node Value, when it is determined that the free memory capacity of target storage node and the ratio of OSD quantity in memory node are less than the first default threshold During value, refusal is operated for the OSD increases of target storage node, and the part OSD of target storage node is arranged into the first kind Type Down states, preset so that the free memory capacity of target storage node and the ratio of remaining OSD quantity are more than or equal to first Threshold value, the risk for being likely to occur OSD performance bottlenecks and data recovery failure on memory node after memory failure is evaded.
Brief description of the drawings
Fig. 1 is a kind of schematic flow sheet of fault handling method provided in an embodiment of the present invention;
Fig. 2 is a kind of schematic flow sheet of fault handling method provided in an embodiment of the present invention;
Fig. 3 is a kind of structural representation of fault treating apparatus provided in an embodiment of the present invention;
Fig. 4 is the structural representation of another fault treating apparatus provided in an embodiment of the present invention;
Fig. 5 is a kind of structural representation of fault treating apparatus provided in an embodiment of the present invention;
Fig. 6 is the hardware configuration that a kind of monitor and memory node provided in an embodiment of the present invention are located at same physical host Schematic diagram.
Embodiment
In order that those skilled in the art more fully understand the technical scheme in the embodiment of the present invention, and make of the invention real Apply the above-mentioned purpose of example, feature and advantage can be more obvious understandable, below in conjunction with the accompanying drawings to technical side in the embodiment of the present invention Case is described in further detail.
Fig. 1 is referred to, is a kind of schematic flow sheet of fault handling method provided in an embodiment of the present invention, wherein, the event Barrier processing method can apply to the memory node of Ceph clusters, as shown in figure 1, the fault handling method can include following step Suddenly:
It should be noted that in embodiments of the present invention, if non-specified otherwise, memory size is in units of G, memory node OSD quantity be that the OSD of UP (work) state is in memory node, the embodiment of the present invention subsequently no longer repeats.
Step 101, the information for obtaining itself free memory capacity and OSD quantity.
Step 102, the information for reporting to monitor itself free memory capacity and OSD quantity.
In the embodiment of the present invention, in order to ensure the OSD of each memory node in Ceph clusters energy normal works, it is necessary to ensure to deposit Each OSD is configured with enough internal memories in storage node, for memory node generally speaking, it is necessary to ensures the available interior of memory node The ratio for depositing capacity and OSD quantity is sufficiently large.
When memory node, which initializes, to be run, it will usually sufficient memory source is reserved for OSD, still, works as memory node When inside points internal memory breaks down, such as memory bar loosen, memory bar interface fault, the free memory capacity of memory node and The ratio of OSD quantity can decline, and even result in low free memory to meet the needs of OSD normal works.
Therefore, in embodiments of the present invention, memory node can report itself available interior according to preset strategy to monitor The information of capacity and OSD quantity is deposited, such as free memory capacity and the ratio of OSD quantity.
For example, memory node regularly can report itself free memory capacity and OSD numbers (as periodically) to monitor The ratio of amount;Or memory node can when the ratio of itself free memory capacity and OSD quantity meets specified requirements, to Monitor reports itself free memory capacity and the ratio of OSD quantity.
It is above-mentioned to report itself free memory capacity and OSD quantity to monitor in the one of embodiment of the present invention Information, it can include:
When itself free memory capacity and the ratio of OSD quantity are less than the first predetermined threshold value, first is sent to monitor Type of report.
In this embodiment it is possible to a threshold value (the referred to herein as first default threshold is set in each memory node in advance Value), the threshold value is used to be judged as whether the reserved memory sources of OSD meet that OSD is capable of the Minimum requirements of normal work, i.e., ought deposit When the free memory capacity of storage node and the ratio of OSD quantity are more than or equal to first predetermined threshold value, show that the memory node is Memory source reserved OSD meets that OSD is capable of the Minimum requirements of normal work, otherwise, shows what the memory node was reserved for OSD Memory source is unsatisfactory for the Minimum requirements that OSD is capable of normal work.Wherein, first predetermined threshold value can be set according to actual scene It is fixed, such as determined according to the actual hard-disk capacity of memory node.
Alternatively, first predetermined threshold value can be the arbitrary value in the range of 0.5~1.5.
Correspondingly, in this embodiment, memory node can obtain itself free memory capacity and the ratio of OSD quantity in real time Value, to determine whether the ratio of itself free memory capacity and OSD quantity is less than the first predetermined threshold value, and when determination itself is available When the ratio of memory size and OSD quantity is less than the first predetermined threshold value, report (the referred to herein as first kind is sent to monitor Report), to notify the ratio of the free memory capacity of the monitor memory node and OSD quantity to be less than the first predetermined threshold value, i.e., The free memory capacity of the memory node performs step to be not enough to the Minimum requirements for meeting OSD normal works, to trigger monitor Rapid 202.
For example, memory node can obtain itself free memory capacity and the ratio of OSD quantity in real time, and by OSD Finger daemon periodically (cycle can set according to actual scene, such as 300 seconds) poll free memory capacity and OSD numbers The ratio of amount, and whether the ratio of the free memory capacity and OSD quantity is judged more than the first predetermined threshold value, and when OSD is guarded When process determines that the ratio of the free memory capacity and OSD quantity is less than the first predetermined threshold value, the first kind is sent to monitor Report.
Wherein, it is contemplated that memory node usually there will be multiple OSD finger daemons (consistent with OSD quantity), in order to avoid Multiple OSD finger daemons in same memory node repeat to send first kind report to monitor, and each OSD finger daemons can be with The poll of free memory capacity and the ratio of OSD quantity is carried out with different cycle, or, with the identical cycle but it is different when Carve and start poll.
For example, it is assumed that memory node create have 32 OSD, can correspondingly, in the memory node run 32 OSD guard into Journey (OSD finger daemons 1~32), each OSD finger daemons can use with the cycle of 300 seconds to memory node poll in memory node The ratio of memory size and OSD quantity, for example, OSD finger daemons 2 can be at 5 seconds of the Poll starting time of OSD finger daemons 1 Start poll afterwards, OSD finger daemons 3 can start poll ... OSD after 5 seconds of the Poll starting time of OSD finger daemons 2 Finger daemon 32 can start poll after 5 seconds of the Poll starting time of OSD finger daemons 31.
When any OSD finger daemons find that the free memory capacity of memory node and the ratio of OSD quantity are pre- less than first If during threshold value, the OSD finger daemons can send first kind report to monitor;Monitor receives first kind report When, a confirmation message can be responded, such as ACK (confirmation) message;OSD finger daemons receive the confirmation message of monitor return Afterwards, can be sent a notification message to other OSD finger daemons so that other OSD finger daemons receive the notification message it Afterwards, silent status can be in certain time length, i.e., without free memory capacity and the ratio of OSD quantity in preset duration Poll.Wherein, the silent duration of each OSD finger daemons needs to be more than each OSD finger daemons progress free memory capacity and OSD The polling cycle of the ratio of quantity, such as when polling cycle is 300 seconds, silent duration can be 900 seconds or 1800 seconds etc..When OSD finger daemons receive the notification message that other OSD finger daemons are sent in the duration of silence, then OSD finger daemons can It is zeroed with the timing to the silence period, restarts the timing of silence period;Otherwise, i.e. OSD finger daemons are in silent duration When not receiving the notification message that other OSD finger daemons are sent, OSD finger daemons can start reach poll next time During start time, the poll of free memory capacity and the ratio of OSD quantity is carried out.
In the embodiment of the present invention, when monitor receives the first kind report that memory node reports, on the one hand, can refuse Absolutely for the OSD increase operations of the memory node;On the other hand, the part OSD on memory node can be carried out at work Reason, i.e., the part OSD on memory node is arranged to first kind Down states and (is referred to as managing Down shapes herein State) so that the free memory capacity of memory node and the ratio of remaining OSD quantity are more than or equal to the first predetermined threshold value, its is specific The associated description that may refer in method flow shown in Fig. 2 is realized, the embodiment of the present invention will not be described here.
Wherein, remaining OSD refers to remove the part OSD for being arranged to first kind Down states in the OSD of memory node Outside available OSD (i.e. the OSD of UP states);First kind Down and OSD Down in existing Ceph clusters difference are: The former OSD not failure, i.e. OSD has in harness ability, but is set to off working state;The latter OSD breaks down, i.e., OSD does not have the ability of work, and is arranged to Down states.
It can be seen that in method flow shown in Fig. 1, itself free memory capacity and OSD quantity are obtained by memory node Ratio, and when reporting itself free memory capacity and the ratio of OSD quantity to monitor, so that monitor is it is determined that memory node Free memory capacity and the ratio of OSD quantity when being less than the first predetermined threshold value, the part OSD on memory node is arranged to One type Down states, the free memory capacity of memory node and the ratio of remaining OSD quantity is set to be more than or equal to the first default threshold Value, that is, it is that OSD is reserved with enough free memory capacity to ensure memory node, and having evaded may after memory failure on memory node There is the risk of OSD performance bottlenecks and data recovery failure.
Fig. 2 is referred to, is a kind of schematic flow sheet of fault handling method provided in an embodiment of the present invention, wherein, the party The monitor (Monitor) that method can apply in Ceph clusters, as shown in Fig. 2 the fault handling method can include it is following Step:
It should be noted that in embodiments of the present invention, monitor can be deployed in different physical hosts from memory node On, same physical host can also be deployed in.Memory node can be understood as comprising at least the collection for the OSD for playing store function Close, certain memory node can also include the processor for possessing control function.
The information of step 201, the free memory capacity for obtaining memory node and OSD quantity.
In the embodiment of the present invention, in order to avoid being likely to occur OSD performance bottlenecks and data after memory failure on memory node Recover the risk of failure, monitor can obtain the free memory capacity of memory node and the information of OSD quantity, so as to it is determined that When the free memory capacity of memory node and the too low ratio of OSD quantity, corresponding strategy is taken to improve the available of memory node The ratio of memory size and OSD quantity.Wherein, the information of the available storage of the memory node of above-mentioned acquisition and OSD quantity It is directed to whole cluster.But the process of above- mentioned information is obtained, obtained, that is, obtained in units of physical host Parameter is for being same physical host, that is, (the total of OSD set can for the free memory capacity of the memory node obtained With memory size) and OSD quantity be same physical host parameter.
In the one of embodiment of the present invention, the free memory capacity of above-mentioned acquisition memory node and the letter of OSD quantity Breath, can include:
The free memory capacity and the information of OSD quantity that reception memory node reports.
In this embodiment, memory node reports the specific reality of the information of free memory capacity and OSD quantity to monitor The associated description in embodiment of the method shown in Fig. 1 is now may refer to, the embodiment of the present invention will not be repeated here.
In another embodiment, the information of the free memory capacity of above-mentioned acquisition memory node and OSD quantity, It can include:
Detect the active volume of the memory node and the information of OSD quantity.
In this embodiment, monitor can be with the active volume of active probe memory node and the information of OSD quantity, example Such as, monitor can the active volume of timing (as periodically) detection memory node and the information of OSD quantity.
It should be noted that in this embodiment it is possible to institute in Ceph clusters is detected by any monitor in Ceph clusters Have the free memory capacity of memory node and the ratio of OSD quantity, or, can according to the quantity of monitor in Ceph clusters, All memory nodes in Ceph clusters are divided into multiple packets, the quantity of the packet can be consistent with the quantity of monitor, and one Individual monitor detects the free memory capacity of memory node and the ratio of OSD quantity in a packet, and it is implemented herein Do not repeat.
Step 202, when it is determined that the free memory capacity of target storage node in memory node and the ratio of OSD quantity are small When the first predetermined threshold value, refusal for target storage node OSD increase operations, and by the part OSD of target storage node First kind inoperative Down states are arranged to, so that the free memory capacity of target storage node and the ratio of remaining OSD quantity Value is more than or equal to the first predetermined threshold value.
In the embodiment of the present invention, target storage node and the memory node for being not specific to a certain fixation, but may refer to Any memory node in Ceph clusters.
In the one of embodiment of the present invention, the free memory capacity of target storage node and the ratio of OSD quantity are determined Value is less than the first predetermined threshold value, can include:
Receive the first kind report of target storage node transmission.
Wherein, target storage node may refer to side shown in Fig. 1 to the specific implementation of monitor transmission first kind report Associated description in method flow, the embodiment of the present invention will not be repeated here.
In this embodiment, when monitor receives the first kind report of target storage node transmission, it may be determined that The free memory capacity of target storage node and the ratio of OSD quantity are less than the first predetermined threshold value.
In another embodiment, the free memory capacity of target storage node and the ratio of OSD quantity are determined Less than the first predetermined threshold value, can include:
Detect the free memory capacity of target storage node and the ratio of OSD quantity is less than the first predetermined threshold value.
In this embodiment, monitor can with the free memory capacity of each memory node in detecting periodically Ceph clusters and The ratio of OSD quantity, and judge whether the free memory capacity of each memory node and the ratio of OSD quantity are less than the first default threshold Value.
In the embodiment of the present invention, the free memory capacity of target storage node and the ratio of OSD quantity are determined when monitor During less than the first predetermined threshold value, on the one hand, monitor needs to refuse the OSD increase operations for the target storage node, that is, works as Monitor, which receives, to be needed, when increasing OSD notification message on target storage node, to forbid this OSD increase operations, avoid The increase of quantity further reduces the free memory capacity of target storage node and the ratio of OSD quantity on target storage node, There is the risk of OSD performance bottlenecks and data recovery failure in aggravation target storage node.
On the other hand, monitor can carry out work disposal to the part OSD of target storage node, i.e., store target The part OSD of node is arranged to first kind Down states, so that the free memory capacity of target storage node and remaining OSD number The ratio of amount is more than or equal to the first predetermined threshold value, that is, ensures that the memory source of target storage node disclosure satisfy that in UP states The Minimum requirements of OSD normal work.
As an example it is assumed that the first predetermined threshold value is h1, the initial free memory capacity of target storage node is M0, initially OSD quantity is N0, M0/N0 >=h1;If a certain moment, target storage node inside points memory failure, cause free memory capacity Change turns to M1, and M1/N0 < h1, and target storage node can report the first kind to report to monitor, and now, monitor can be from N1 OSD is selected to be arranged to first kind Down states in target storage node so that M1/ (N0-N1) >=h1.Optionally, select The quantity that selects out be arranged to the OSD of first kind Down states can be the minimum value for the N1 for meeting M1/ (N0-N1) >=h1.
Further, in embodiments of the present invention, after monitor has carried out above-mentioned processing to memory node, if storage The memory failure of node is repaired, then the free memory capacity of memory node can increase, and correspondingly, can use for memory node is interior Depositing capacity and the ratio of OSD quantity can also increase, and now, monitor can be arranged to by the memory node upper part or all The OSD of first kind Down states reverts to UP states.
And in order to avoid monitor is extensive by the OSD for being partly or entirely arranged to first kind Down states in memory node It is that the free memory capacity of memory node is again lowered to less than the first default threshold with the ratio of OSD quantity after UP states again Value, results in the need for that the part OSD of memory node is arranged into first kind Down states again, can be pre- so as to form concussion Another threshold value (referred to herein as the second predetermined threshold value) is first set, second predetermined threshold value is used to determine whether to store The OSD for being arranged to first kind Down states on node reverts to UP states.Wherein, it is pre- to be more than first for second predetermined threshold value If threshold value.
Preferably, second predetermined threshold value is free memory capacity and OSD ratio ideally, you can uses internal memory Optimal or preferred proportion between capacity and OSD.
Optionally, the second predetermined threshold value can be the arbitrary value in the range of 1.5~2.0, wherein, when the first predetermined threshold value is When 1.5, the second predetermined threshold value needs to be more than 1.5.
Correspondingly, it is above-mentioned that the part OSD of target storage node is arranged to first in the one of embodiment of the present invention After type Down states, it can also include:
When it is determined that the free memory capacity of target storage node and the ratio of OSD quantity are more than the second predetermined threshold value, permit Perhaps the OSD increase operations of target storage node are directed to, and be more than with the free memory capacity after recovery and the ratio of OSD quantity etc. In the principle of the second predetermined threshold value, the portion in the part OSD of first kind Down states will be arranged in target storage node Divide or whole OSD revert to UP states.
In the embodiment, monitor determines that the free memory capacity of target storage node and the ratio of OSD quantity are more than the Two predetermined threshold values can include:
The Second Type report that target storage node reports is received, or, detects the free memory of target storage node Capacity and the ratio of OSD quantity are more than the second predetermined threshold value.
Wherein, Second Type report is that target storage node determines itself free memory capacity and the ratio of OSD quantity Turn to what is sent during more than the second predetermined threshold value by becoming less than the first predetermined threshold value, allow to be directed to target to trigger monitor and perform The OSD increase operations of memory node, and be more than or equal to second with the free memory capacity after recovery and the ratio of OSD quantity and preset The principle of threshold value, the part or all of OSD in the part OSD of first kind Down states will be arranged in target storage node Revert to the operation of UP states.
Wherein, target storage node sends specific implementation and the target storage node of Second Type report to prison to monitor Realizing for visual organ transmission first kind report is similar, and the embodiment of the present invention will not be described here.
In this embodiment, when monitor determine target storage node free memory capacity and OSD quantity ratio it is big When the second predetermined threshold value, monitor can determine that target storage node is reserved with enough memory sources for OSD, now, can To allow target storage node to increase OSD quantity.
Correspondingly, on the one hand, monitor can allow the OSD increase operations for target storage node;On the other hand, supervise The OSD for being partly or entirely arranged to first kind Down states in target storage node can be reverted to UP states by visual organ.
Wherein, monitor is carried out when above-mentioned recovery is handled to target storage node, it is necessary to ensure that target storage node recovers The ratio of free memory capacity and OSD quantity after processing is more than or equal to Second Threshold.
For example, still exemplified by one example of the above, it is assumed that the second predetermined threshold value is h2, if a certain moment, target storage section Point free memory capacity restoration is M2 (M1 < M2≤M0), and M2/ (N0-N1) > h2, and now, monitor can store from target It is arranged to select N2 (0 < N2 < N1) individual OSD to revert to UP states in the OSD of first kind Down states in node, wherein, M2/(N0-N1+N2)≥h2。
Further, in the one of embodiment of the present invention, when monitor determines the free memory of target storage node Capacity and the ratio of OSD quantity are more than or equal to the first predetermined threshold value, and when being less than the second predetermined threshold value, refusal is directed to memory node OSD increase operation.
In this embodiment, monitor determines that the free memory capacity of target storage node and the ratio of OSD quantity are more than Equal to the first predetermined threshold value, and the 3rd type report that can be sent less than the second predetermined threshold value including receiving target storage node Accuse, or, detect the free memory capacity of target storage node and the ratio of OSD quantity is more than or equal to the first predetermined threshold value, and Less than the second predetermined threshold value.
Wherein, the 3rd type of report is that target storage node determines itself free memory capacity and the ratio of OSD quantity More than or equal to the first predetermined threshold value, and it is less than what is sent during the second predetermined threshold value, and then triggers monitor refusal for storage section The OSD increase operations of point.
Wherein, target storage node sends specific implementation and the target storage node of the 3rd type of report to prison to monitor Realizing for visual organ transmission first kind report is similar, and the embodiment of the present invention will not be described here.
It should be noted that when the free memory capacity of target storage node and the ratio of OSD quantity are by pre- less than first If changes of threshold is more than or equal to the first predetermined threshold value, and when being less than the second predetermined threshold value, illustrates that now memory node can be tieed up OSD normal work is held, but has been unsatisfactory for the optimal or preferable distribution between OSD and free memory capacity, is now supervised Visual organ can keep OSD increase operation of the refusal for target storage node, without carrying out other specially treateds.
Further, in the one of embodiment of the present invention, when monitor determines the free memory of target storage node Capacity and the ratio of OSD quantity are less than the first predetermined threshold value, or, more than or equal to the first predetermined threshold value and are less than the second predetermined threshold value When, monitor can generate system health daily record, and the system health daily record is used to record target storage node generation memory failure, So as to, keeper can know that the situation of memory failure occurs for target storage node according to the system health daily record, and according to need Seek carry out respective handling.
It should be noted that in embodiments of the present invention, after being changed due to the OSD states of memory node, Ceph collection The Leader (leader) of group needs to update cluster Map (mapping), therefore, if not Leader monitor is by the middle part of memory node Point OSD is arranged to after first kind Down states, it is also necessary to Leader nodes are notified, by Leader node updates cluster Map, Therefore, in order to improve cluster Map renewal efficiency, above-mentioned processing can be performed by the Leader of Ceph clusters and operated.
In order that those skilled in the art more fully understand that the embodiment of the present invention provides technical scheme, with reference to specific reality Example illustrates to technical scheme provided in an embodiment of the present invention.
In this embodiment it is assumed that the first predetermined threshold value (h1) is 1, the second predetermined threshold value (h2) is 1.5;Assuming that target is deposited The initial free memory capacity of storage node is 64G (assuming that being provided by 16 4G internal memories), and OSD quantity is 32.The then embodiment Middle fault handling method flow is as follows:
1st, when 5 root memories break down in target storage node, the free memory capacity and OSD numbers of target storage node The ratio (being depicted below as h3) of amount is h3=(64-4*5)/32=1.375, i.e. h1 < h3 < h2, now target storage node It can be reported by OSD finger daemons to monitor, i.e., send the 3rd type of report to monitor;
When monitor receives three type of report of target storage node transmission, determine that target storage node is in h1 < H3 < h2 state, now, monitor can perform following processing:
A, system health daily record is generated, memory failure occurs for record target storage node;
B, OSD increase operation of the refusal for target storage node;
2nd, when other 4 root memory of target storage node breaks down (i.e. totally 9 root memories break down), target storage section H3=(64-4*9)/32=0.875, i.e. the h3 < h1 of point, now, target storage node can be by OSD finger daemons to prison Visual organ reports, i.e., sends first kind report to monitor;
When monitor receives the first kind report of target storage node transmission, determine that target storage node is in h3 < H1 state, now, monitor can perform following processing:
A, system health daily record is generated, serious memory failure occurs for record target storage node;
B, OSD increase operation of the refusal for target storage node;
C, work disposal is carried out to target storage node upper part OSD, its realization is as follows:
I, the quantity o1 for the OSD for needing work disposal is calculated, the h3 of the target storage node after being handled with guarantee is more than H1, i.e. 1≤(64-4*9)/(32-o1), therefore, o1 minimum value is 4;
Random 4 OSD on target storage node are carried out work disposal by ii, monitor, will select 4 OSD It is arranged to manage Down states (i.e. first kind Down states), and is moved the data on this 4 OSD by data recovery action Move on other normal OSD of Ceph clusters;
Wherein, monitor, can be periodically after having carried out going work disposal to the 4 of target storage node OSD Detect the h3 values of target storage node;
3rd, when keeper has found system health daily record and to find target storage node according to the system health daily record, and carry out During internal memory recovery processing (assuming that the free memory capacity after recovering is M), the h3 of target storage node can increase, when monitor is visited When measuring target storage node and being in h3 > h2 state (i.e. h3 > 1.5), monitor can perform following processing:
A, the OSD increase operations for target storage node are allowed;
B, the OSD of target storage node upper part or cura generalis Down states is reverted into UP states, its realization is as follows:
I, the quantity o2 for the OSD for needing to revert to UP states is calculated, the h3 of the target storage node after being handled with guarantee is big In equal to h2, i.e. 1.5≤M/ (28+o2);
The OSD of o2 quantity reverts to UP states in ii, monitor will manage Down states OSD on target storage node, And carry out releveling processing.
Wherein, the difference that monitor performs according to the change of the free memory capacity of memory node and the ratio of OSD quantity The schematic diagram of operation can be as shown in Figure 3.
By above description as can be seen that in technical scheme provided in an embodiment of the present invention, by being provided for indicating First predetermined threshold value of the memory source deficiency of memory node, when it is determined that the free memory of the target storage node in memory node When capacity and the ratio of OSD quantity are less than the first predetermined threshold value, refusal operates for the OSD increases of target storage node, and will The part OSD of target storage node is arranged to first kind Down states so that the free memory capacity of target storage node and The ratio of remaining OSD quantity is more than or equal to the first predetermined threshold value, has evaded and has been likely to occur OSD on memory node after memory failure Energy bottleneck and the risk of data recovery failure.
Fig. 4 is referred to, is a kind of structural representation of fault treating apparatus provided in an embodiment of the present invention, wherein, the dress The monitor that can apply in above method embodiment is put, as shown in figure 4, the fault treating apparatus can include:
Acquiring unit 410, for obtaining the free memory capacity of memory node and the letter of object storage device OSD quantity Breath;
Determining unit 420, for determining in the memory node with the presence or absence of the memory size of target storage node and right As the ratio of storage device OSD quantity is less than the first predetermined threshold value;
Processing unit 430, for determining the available of target storage node in the memory node when the determining unit 420 When the ratio of memory size and OSD quantity is less than the first predetermined threshold value, OSD increase behaviour of the refusal for the target storage node Make, and the part OSD of the target storage node is arranged to first kind inoperative Down states, so that the target stores The free memory capacity of node and the ratio of remaining OSD quantity are more than or equal to first predetermined threshold value;Wherein, it is described remaining OSD refers to available in addition to the part OSD for being arranged to first kind Down states in the OSD of the target storage node OSD。
In an alternative embodiment, the acquiring unit 410, the free memory reported specifically for receiving the memory node The information of capacity and OSD quantity;Or, the active volume of the detection memory node and the information of OSD quantity.
In an alternative embodiment, the processing unit 430, it is additionally operable to when the determining unit 420 is when the determination target When the free memory capacity of memory node and the ratio of OSD quantity are more than the second predetermined threshold value, it is allowed to stored for the target The OSD increase operations of node, and free memory capacity after being handled with recovery and the ratio of OSD quantity are more than or equal to described second The principle of predetermined threshold value, the part in the part OSD of first kind Down states will be arranged in the target storage node Or whole OSD revert to work UP states;Wherein, first predetermined threshold value is less than second predetermined threshold value.
In an alternative embodiment, the acquiring unit 410, the free memory for being additionally operable to detect the target storage node hold The ratio of amount and OSD quantity;
The determining unit 420, specifically for when the free memory capacity and OSD numbers that detect the target storage node When the ratio of amount is more than the second predetermined threshold value, the free memory capacity of the target storage node and the ratio of OSD quantity are determined More than the second predetermined threshold value.
In an alternative embodiment, the processing unit 430, it is additionally operable to when the memory size for determining the target storage node When being more than or equal to first predetermined threshold value with the ratio of OSD quantity, and being less than second predetermined threshold value, refusal is for described The OSD increase operations of target storage node.
Fig. 5 is referred to, is a kind of structural representation of fault treating apparatus provided in an embodiment of the present invention, wherein, the dress The memory node that can apply in above method embodiment is put, as shown in figure 5, the fault treating apparatus can include:
Acquiring unit 510, for obtaining the information of itself free memory capacity and object storage object equipment OSD quantity;
Transmitting element 520, for reporting the information of itself free memory capacity and OSD quantity to monitor.
The hardware configuration that Fig. 6 is located at same physical host for a kind of monitor and memory node that disclosure example provides shows It is intended to.But it is to be understood that monitor and memory node can be located at different physical hosts.The present embodiment is with Fig. 6 situation Illustrate.The physical host may include processor 601, be stored with the machinable medium 602 of machine-executable instruction And metadata and/or the OSD603 of copy for data storage object.
Processor 601 can communicate with machinable medium 602 and OSD603 via system bus 604.It is also, logical Cross and read and perform machine-executable instruction corresponding with fault handling logic, processor 601 in machinable medium 602 It can perform above-described fault handling method.
Machinable medium 602 referred to herein can be any electronics, magnetic, optics or other physical stores Device, can be included or storage information, such as executable instruction, data, etc..For example, machinable medium can be: RAM (Radom Access Memory, random access memory), volatile memory, nonvolatile memory, flash memory, storage are driven Dynamic device (such as hard disk drive), solid state hard disc, any kind of storage dish (such as CD, dvd), or similar storage are situated between Matter, or combinations thereof.OSD603 can include but is not limited to physical disk.
The function of unit and the implementation process of effect specifically refer to and step are corresponded in the above method in said apparatus Implementation process, it will not be repeated here.
For device embodiment, because it corresponds essentially to embodiment of the method, so related part is real referring to method Apply the part explanation of example.Device embodiment described above is only schematical, wherein described be used as separating component The unit of explanation can be or may not be physically separate, can be as the part that unit is shown or can also It is not physical location, you can with positioned at a place, or can also be distributed on multiple NEs.Can be according to reality Need to select some or all of module therein to realize the purpose of the present invention program.Those of ordinary skill in the art are not paying In the case of going out creative work, you can to understand and implement.
Those skilled in the art will readily occur to the present invention its after considering specification and putting into practice invention disclosed herein Its embodiment.The application be intended to the present invention any modification, purposes or adaptations, these modifications, purposes or Person's adaptations follow the general principle of the present invention and including undocumented common knowledges in the art of the invention Or conventional techniques.Description and embodiments are considered only as exemplary, and true scope and spirit of the invention are by following Claim is pointed out.
It should be appreciated that the invention is not limited in the precision architecture for being described above and being shown in the drawings, and And various modifications and changes can be being carried out without departing from the scope.The scope of the present invention is only limited by appended claim.

Claims (12)

1. a kind of fault handling method, it is characterised in that this method includes:
Obtain the free memory capacity of memory node and the information of object storage device OSD quantity;
When it is determined that the free memory capacity of target storage node and the ratio of OSD quantity are default less than first in the memory node During threshold value, refusal is operated for the OSD increases of the target storage node, and the part OSD of the target storage node is set First kind inoperative Down states are set to, so that the free memory capacity of the target storage node and remaining OSD quantity Ratio is more than or equal to first predetermined threshold value;Wherein, remaining described OSD refers to remove quilt in the OSD of the target storage node The available OSD being arranged to outside the part OSD of first kind Down states.
2. according to the method for claim 1, it is characterised in that the free memory capacity and OSD for obtaining memory node The information of quantity, including:
Receive the information of free memory capacity that the memory node reports and OSD quantity;Or,
Detect the active volume of the memory node and the information of OSD quantity.
3. according to the method for claim 1, it is characterised in that the part OSD by the target storage node is set After first kind Down states, in addition to:
When it is determined that the free memory capacity of the target storage node and the ratio of OSD quantity are more than the second predetermined threshold value, permit Perhaps the OSD increase operations of the target storage node, and free memory capacity and the ratio of OSD quantity after being handled with recovery are directed to Value will be arranged to first kind Down states more than or equal to the principle of second predetermined threshold value in the target storage node Part OSD in part or all of OSD revert to work UP states;Wherein, first predetermined threshold value is less than described second Predetermined threshold value.
4. according to the method for claim 3, it is characterised in that the free memory for determining the target storage node holds The ratio of amount and OSD quantity is more than the second predetermined threshold value, including:
Detect the free memory capacity of the target storage node and the ratio of OSD quantity;
When the free memory capacity and the ratio of OSD quantity that detect the target storage node are more than the second predetermined threshold value, Determine that the free memory capacity of the target storage node and the ratio of OSD quantity are more than the second predetermined threshold value.
5. according to the method for claim 3, it is characterised in that methods described also includes:
When it is determined that the memory size of the target storage node and the ratio of OSD quantity are more than or equal to first predetermined threshold value, And when being less than second predetermined threshold value, OSD increase operation of the refusal for the target storage node.
6. a kind of fault handling method, it is characterised in that this method includes:
Obtain the information of itself free memory capacity and object storage object equipment OSD quantity;
The information of itself free memory capacity and OSD quantity is reported to monitor.
7. a kind of fault treating apparatus, it is characterised in that the device includes:
Acquiring unit, for obtaining the free memory capacity of memory node and the information of object storage device OSD quantity;
Determining unit, for determining to set with the presence or absence of the memory size of target storage node and object storage in the memory node The ratio of standby OSD quantity is less than the first predetermined threshold value;
Processing unit, for determined when the determining unit target storage node in the memory node free memory capacity and When the ratio of OSD quantity is less than the first predetermined threshold value, refusal for the target storage node OSD increase operations, and by institute The part OSD for stating target storage node is arranged to first kind inoperative Down states so that the target storage node can It is more than or equal to first predetermined threshold value with the ratio of memory size He remaining OSD quantity;Wherein, remaining described OSD refers to institute State the available OSD in addition to the part OSD for being arranged to first kind Down states in the OSD of target storage node.
8. device according to claim 7, it is characterised in that
The acquiring unit, the free memory capacity and the information of OSD quantity reported specifically for the reception memory node; Or, the active volume of the detection memory node and the information of OSD quantity.
9. device according to claim 7, it is characterised in that
The processing unit, be additionally operable to when the determining unit when the free memory capacity for determining the target storage node and When the ratio of OSD quantity is more than the second predetermined threshold value, it is allowed to increase operation for the OSD of the target storage node, and with extensive The ratio of free memory capacity and OSD quantity after multiple processing is more than or equal to the principle of second predetermined threshold value, by the mesh The part or all of OSD being arranged in mark memory node in the part OSD of first kind Down states reverts to work UP shapes State;Wherein, first predetermined threshold value is less than second predetermined threshold value.
10. device according to claim 9, it is characterised in that
The acquiring unit, it is additionally operable to detect the free memory capacity of the target storage node and the ratio of OSD quantity;
The determining unit, specifically for when the free memory capacity and the ratio of OSD quantity that detect the target storage node Value is when being more than the second predetermined threshold value, determines the free memory capacity of the target storage node and the ratio of OSD quantity more than the Two predetermined threshold values.
11. device according to claim 9, it is characterised in that
The processing unit, it is additionally operable to when the memory size and the ratio of OSD quantity that determine the target storage node are more than etc. In first predetermined threshold value, and when being less than second predetermined threshold value, OSD increase of the refusal for the target storage node Operation.
12. a kind of fault treating apparatus, it is characterised in that the device includes:
Acquiring unit, for obtaining the information of itself free memory capacity and object storage object equipment OSD quantity;
Transmitting element, for reporting the information of itself free memory capacity and OSD quantity to monitor.
CN201711015125.1A 2017-10-26 2017-10-26 Fault processing method and device Active CN107729185B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201711015125.1A CN107729185B (en) 2017-10-26 2017-10-26 Fault processing method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201711015125.1A CN107729185B (en) 2017-10-26 2017-10-26 Fault processing method and device

Publications (2)

Publication Number Publication Date
CN107729185A true CN107729185A (en) 2018-02-23
CN107729185B CN107729185B (en) 2020-12-04

Family

ID=61213886

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201711015125.1A Active CN107729185B (en) 2017-10-26 2017-10-26 Fault processing method and device

Country Status (1)

Country Link
CN (1) CN107729185B (en)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109101357A (en) * 2018-07-20 2018-12-28 广东浪潮大数据研究有限公司 A kind of detection method and device of OSD failure
CN109213637A (en) * 2018-11-09 2019-01-15 浪潮电子信息产业股份有限公司 Data reconstruction method, device and the medium of distributed file system clustered node
CN109614276A (en) * 2018-11-28 2019-04-12 平安科技(深圳)有限公司 Fault handling method, device, distributed memory system and storage medium
CN109669822A (en) * 2018-11-28 2019-04-23 平安科技(深圳)有限公司 The creation method and computer readable storage medium of electronic device, spare memory pool
CN109710456A (en) * 2018-12-10 2019-05-03 新华三技术有限公司 A kind of data reconstruction method and device
CN115543862A (en) * 2022-09-27 2022-12-30 超聚变数字技术有限公司 Memory management method and related device
CN118035924A (en) * 2024-04-11 2024-05-14 国网山东省电力公司电力科学研究院 Power equipment fault data processing method and system based on power big data
CN118035924B (en) * 2024-04-11 2024-06-21 国网山东省电力公司电力科学研究院 Power equipment fault data processing method and system based on power big data

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102693177A (en) * 2011-03-23 2012-09-26 ***通信集团公司 Fault diagnosing and processing methods of virtual machine as well as device and system thereof
US20150149423A1 (en) * 2013-10-18 2015-05-28 Hitachi Data Systems Engineering UK Limited Data redundancy in a cluster system
US20150254003A1 (en) * 2014-03-10 2015-09-10 Futurewei Technologies, Inc. Rdma-ssd dual-port unified memory and network controller
CN105119737A (en) * 2015-07-16 2015-12-02 浪潮软件股份有限公司 Method for monitoring Ceph cluster through Zabbix
CN105930103A (en) * 2016-05-10 2016-09-07 南京大学 Distributed storage CEPH based erasure correction code overwriting method
CN106302717A (en) * 2016-08-12 2017-01-04 浪潮(北京)电子信息产业有限公司 The method for optimizing resources of a kind of CEPH system and device
CN107045469A (en) * 2017-03-28 2017-08-15 北京精强远科技有限公司 A kind of intelligent sound warning system and method

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102693177A (en) * 2011-03-23 2012-09-26 ***通信集团公司 Fault diagnosing and processing methods of virtual machine as well as device and system thereof
US20150149423A1 (en) * 2013-10-18 2015-05-28 Hitachi Data Systems Engineering UK Limited Data redundancy in a cluster system
US20150254003A1 (en) * 2014-03-10 2015-09-10 Futurewei Technologies, Inc. Rdma-ssd dual-port unified memory and network controller
CN105119737A (en) * 2015-07-16 2015-12-02 浪潮软件股份有限公司 Method for monitoring Ceph cluster through Zabbix
CN105930103A (en) * 2016-05-10 2016-09-07 南京大学 Distributed storage CEPH based erasure correction code overwriting method
CN106302717A (en) * 2016-08-12 2017-01-04 浪潮(北京)电子信息产业有限公司 The method for optimizing resources of a kind of CEPH system and device
CN107045469A (en) * 2017-03-28 2017-08-15 北京精强远科技有限公司 A kind of intelligent sound warning system and method

Cited By (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109101357A (en) * 2018-07-20 2018-12-28 广东浪潮大数据研究有限公司 A kind of detection method and device of OSD failure
CN109213637B (en) * 2018-11-09 2022-03-04 浪潮电子信息产业股份有限公司 Data recovery method, device and medium for cluster nodes of distributed file system
CN109213637A (en) * 2018-11-09 2019-01-15 浪潮电子信息产业股份有限公司 Data reconstruction method, device and the medium of distributed file system clustered node
CN109614276A (en) * 2018-11-28 2019-04-12 平安科技(深圳)有限公司 Fault handling method, device, distributed memory system and storage medium
CN109669822A (en) * 2018-11-28 2019-04-23 平安科技(深圳)有限公司 The creation method and computer readable storage medium of electronic device, spare memory pool
CN109669822B (en) * 2018-11-28 2023-06-06 平安科技(深圳)有限公司 Electronic device, method for creating backup storage pool, and computer-readable storage medium
WO2020107829A1 (en) * 2018-11-28 2020-06-04 平安科技(深圳)有限公司 Fault processing method, apparatus, distributed storage system, and storage medium
CN109614276B (en) * 2018-11-28 2021-09-21 平安科技(深圳)有限公司 Fault processing method and device, distributed storage system and storage medium
CN109710456B (en) * 2018-12-10 2021-03-23 新华三技术有限公司 Data recovery method and device
CN109710456A (en) * 2018-12-10 2019-05-03 新华三技术有限公司 A kind of data reconstruction method and device
CN115543862A (en) * 2022-09-27 2022-12-30 超聚变数字技术有限公司 Memory management method and related device
CN115543862B (en) * 2022-09-27 2023-09-01 超聚变数字技术有限公司 Memory management method and related device
CN118035924A (en) * 2024-04-11 2024-05-14 国网山东省电力公司电力科学研究院 Power equipment fault data processing method and system based on power big data
CN118035924B (en) * 2024-04-11 2024-06-21 国网山东省电力公司电力科学研究院 Power equipment fault data processing method and system based on power big data

Also Published As

Publication number Publication date
CN107729185B (en) 2020-12-04

Similar Documents

Publication Publication Date Title
CN107729185A (en) A kind of fault handling method and device
CN109407976B (en) Distributed storage method and distributed storage device
CN106170971A (en) Arbitration process method, arbitration storage device and system after a kind of cluster Schizencephaly
EP2606607B1 (en) Determining equivalent subsets of agents to gather information for a fabric
CN107817950A (en) A kind of data processing method and device
CN106911728A (en) The choosing method and device of host node in distributed system
CN108984107A (en) Improve the availability of storage system
CN109274544A (en) A kind of fault detection method and device of distributed memory system
US20060117101A1 (en) Node discovery and communications in a network
JP2015530639A5 (en)
CN104572344B (en) A kind of method and system of cloudy data backup
US10999131B2 (en) Method and system for detecting abnormalities in network element operation
CN106933659A (en) The method and apparatus of managing process
CN109710456B (en) Data recovery method and device
CN101631048A (en) Method, device and system for monitoring managed object
CN106330531A (en) Node fault recording and processing method and device
CN107065616A (en) Inter-linked controlling method and device, computer-readable recording medium
US7840725B2 (en) Capture of data in a computer network
CN102981939B (en) Disk monitoring method
CN106911519A (en) A kind of data acquisition monitoring method and device
KR20120030938A (en) Method of data replication in a distributed data storage system and corresponding device
CN110532096B (en) System and method for multi-node grouping parallel deployment
US8612524B2 (en) Cessation of sending network status messages to a server
CN107547301A (en) A kind of master/slave device switching method and device
JP2010160660A (en) Network interface, computer system, operation method therefor, and program

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant