CN111142801B - Distributed storage system network sub-health detection method and device - Google Patents
Distributed storage system network sub-health detection method and device Download PDFInfo
- Publication number
- CN111142801B CN111142801B CN201911371153.6A CN201911371153A CN111142801B CN 111142801 B CN111142801 B CN 111142801B CN 201911371153 A CN201911371153 A CN 201911371153A CN 111142801 B CN111142801 B CN 111142801B
- Authority
- CN
- China
- Prior art keywords
- network
- storage devices
- object storage
- osd
- sub
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/22—Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing
- G06F11/2205—Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing using arrangements specific to the hardware being tested
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0631—Management of faults, events, alarms or notifications using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
- H04L41/064—Management of faults, events, alarms or notifications using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis involving time analysis
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0677—Localisation of faults
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/14—Network analysis or design
- H04L41/147—Network analysis or design for predicting network behaviour
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Abstract
The invention discloses a distributed storage system network sub-health detection method and device. Wherein, the method comprises the following steps: determining heartbeat messages among data falling processes of a plurality of object storage devices OSD in a distributed system and communication messages among object storage devices OSD and copies of the object storage devices OSD, wherein the heartbeat messages and the communication messages are transmitted through a network among the object storage devices OSD, and the distributed system comprises the plurality of object storage devices OSD; and determining whether the network between the object storage devices OSD is sub-healthy or not according to the heartbeat message and the communication message. The invention solves the technical problems that the system problems can be detected only after the network in the distributed system is abnormal in the related technology, and the network abnormality cannot be predetermined.
Description
Technical Field
The invention relates to the field of system monitoring, in particular to a distributed storage system network sub-health detection method and device.
Background
In a distributed storage system, the health of a network has a great influence on the stability and performance of the system, but in actual situations, the network is often unstable, and the performance and stability of the distributed storage system are affected by network card deceleration, large network time and network packet loss. In the distributed storage system, judgment and isolation of network sub-health are very complex, and the network sub-health is mostly judged through ping between every two links or through a network diagnosis tool, but the pressure of the distributed storage system is increased certainly, so that the service is greatly influenced, and in addition, if the judgment is not accurate, wrong isolation is caused, and greater risks are brought.
In the related technology, nodes affected by network sub-health are located by detecting the state change of network resources, then detection packets are sent aiming at the nodes, the problems of abnormal link time, packet loss and the like are identified, and network port isolation, link isolation, node isolation and the like are carried out according to diagnosis results. The prior art can detect the sub-health of the network, and has the main defects that when the sub-health of the network is not timely, the judgment is carried out according to the state of the network resource, the network resource changes, the situation that the time or packet loss and other relatively large problems are serious, the stability problem of system performance reduction or system abnormity and the like is caused, and the judgment is late.
In view of the above problems, no effective solution has been proposed.
Disclosure of Invention
The embodiment of the invention provides a method and a device for detecting sub-health of a distributed storage system network, which are used for at least solving the technical problems that system problems can be detected and network abnormalities cannot be predetermined after the network in a distributed system in the related technology is abnormal.
According to an aspect of an embodiment of the present invention, there is provided a distributed system determination method, including: determining heartbeat messages among data falling processes of a plurality of object storage devices OSD in a distributed system and communication messages among the object storage devices OSD and copies of the object storage devices OSD, wherein the heartbeat messages and the communication messages are transmitted through a network among the object storage devices OSD, and the distributed system comprises a plurality of object storage devices OSD; and determining whether the network between the object storage devices OSD is sub-healthy or not according to the heartbeat message and the communication message.
Optionally, determining whether the network between the object storage devices OSD is sub-healthy according to the heartbeat packet and the communication packet includes: determining whether the number of times that the network delay of the communication message or the heartbeat message exceeds first preset time exceeds the preset number of times within a first preset time period in the past; and determining the sub-health of the network between the object storage devices OSD under the condition that the network delay of the communication message or the heartbeat message exceeds the first preset time and the preset times.
Optionally, determining whether the network between the object storage devices OSD is sub-healthy according to the heartbeat packet and the communication packet includes: determining whether the network delay of the communication message exceeds a second preset time and a first preset multiple of the average delay of the historical data of the communication message in a third past preset time period within a second past preset time period; and determining the sub-health of the network between the object storage devices OSD under the condition that the network delay of the communication message exceeds a second preset time within the second preset time period and the network delay of the communication message exceeds a first preset multiple of the average delay of the historical data of the communication message within the third preset time period.
Optionally, determining whether the network delay of the communication packet exceeds the second preset time and exceeds a preset multiple of the delay of the third preset time period in the second preset time period includes: determining the communication message with the same sending address as the communication message and the same file size grade as the historical data, wherein the file size grade is multiple, and the multiple grades are multiple preset ranges of the file size respectively.
Optionally, determining whether the network between the object storage devices OSD is sub-healthy according to the heartbeat packet and the communication packet includes: determining network delays of all heartbeat messages in a storage pool where the object storage device OSD is located within a fourth preset time period, and determining an average network delay of the heartbeat messages according to the network delays of all the heartbeat messages in the storage pool; determining whether the average network delay of the heartbeat message exceeds a third preset time and a second preset multiple of the average network delay of the storage pool within a fourth preset time period; and in the fourth preset time period, determining that the network between the object storage devices OSD is sub-healthy under the condition that the average network delay of the heartbeat message exceeds a third preset time and a second preset multiple of the average network delay of the storage pool in the fourth preset time period.
Optionally, after determining whether the network between the object storage devices OSD is sub-healthy according to the heartbeat message and the communication message, the method further includes: and under the condition that the sub-health of the network between the object storage devices OSD is detected, sending a detection packet for detecting the network abnormity to the position of the abnormal network, and determining whether the network between the object storage devices OSD is actually abnormal or not according to the receiving condition of the detection packet.
Optionally, in a case that it is determined that the network ownership between the object storage devices OSD is abnormal, the method further includes: isolating the network between the object storage devices OSD in the distributed system, wherein the object storage devices OSD are abnormal.
According to another aspect of the embodiments of the present invention, there is also provided a distributed system detection apparatus, including: the system comprises a determining module, a judging module and a processing module, wherein the determining module is used for determining heartbeat messages among data falling processes of a plurality of object storage devices OSD in a distributed system and communication messages among the object storage devices OSD and a copy of the object storage devices OSD, the heartbeat messages and the communication messages are transmitted through a network among the object storage devices OSD, and the distributed system comprises a plurality of object storage devices OSD; and the detection module is used for determining whether the network between the object storage devices OSD is sub-healthy or not according to the heartbeat message and the communication message.
According to another aspect of the embodiments of the present invention, a storage medium is further provided, where the storage medium includes a stored program, and when the program runs, a device where the storage medium is located is controlled to execute any one of the above-mentioned methods for detecting sub-health of a distributed storage system network.
According to another aspect of the embodiments of the present invention, there is further provided a processor, where the processor is configured to execute a program, where the program executes the method for detecting sub-health of a distributed storage system network described in any one of the above.
In the embodiment of the invention, a heartbeat message between data falling processes of a plurality of object storage devices OSD in a distributed system and a communication message between the object storage devices OSD and a copy of the object storage devices OSD are determined, wherein the heartbeat message and the communication message are transmitted through a network between the object storage devices OSD, and the distributed system comprises a plurality of object storage devices OSD; the method for determining whether the network between the object storage devices OSD is sub-healthy or not according to the heartbeat message and the communication message achieves the purpose of predicting the abnormity of the distributed system by determining whether the network between the object storage devices OSD is sub-healthy or not through the heartbeat message between the object storage devices OSD of the distributed system, the copy of the object storage devices OSD and the communication message of the programming language EC, thereby achieving the technical effect of predetermining the abnormity of the distributed system, and further solving the technical problems that the system problem can be detected and the network abnormity cannot be predetermined after the network in the distributed system in the related technology is abnormal.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
FIG. 1 is a flow chart of a distributed storage system network sub-health detection method according to an embodiment of the invention;
fig. 2 is a schematic diagram of a distributed storage system network sub-health detection apparatus according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
The related techniques in this embodiment are explained as follows:
briefly, the distributed storage system is to store data on a plurality of storage servers in a distributed manner, and form the distributed storage resources into a virtual storage device, which can provide data storage service
Network sub-health: network subhealth refers to network card speed reduction, time, packet loss and the like, and the network subhealth can cause cluster network performance degradation
OSD: the object Storage device OSD (object Storage device) has a process responsible for data dropping, and the copy and EC are also implemented by the service of the OSD. There is one OSD process per hard disk.
MON: and a monitoring service (Monitor) for monitoring the state of the whole cluster.
In accordance with an embodiment of the present invention, there is provided a method embodiment of a distributed storage system network sub-health detection method, it should be noted that the steps illustrated in the flowchart of the accompanying drawings may be performed in a computer system such as a set of computer-executable instructions, and that while a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different than here.
Fig. 1 is a flowchart of a distributed storage system network sub-health detection method according to an embodiment of the present invention, as shown in fig. 1, the method includes the following steps:
step S102, determining heartbeat messages among data falling processes of a plurality of object storage devices OSD in the distributed system and communication messages among object storage devices OSD and copy of the object storage devices OSD, wherein the heartbeat messages and the communication messages are transmitted through a network among the object storage devices OSD, and the distributed system comprises a plurality of object storage devices OSD;
and step S104, determining whether the network between the object storage devices OSD is sub-healthy or not according to the heartbeat message and the communication message.
Determining a heartbeat message between object storage devices OSD of the distributed system, a copy of the object storage devices OSD and a communication message of a programming language EC, wherein the heartbeat message and the communication message are transmitted through a network between the object storage devices OSD, and the distributed system comprises a plurality of object storage devices OSD; the method for predicting the network between the object storage devices OSD according to the heartbeat message and the communication message predicts the network between the object storage devices OSD through the heartbeat message between the object storage devices OSD of the distributed system, the copy of the object storage devices OSD and the communication message of the programming language EC, and achieves the purpose of determining whether the distributed system is sub-healthy, thereby achieving the technical effect of predetermining the abnormity of the distributed system, and further solving the technical problems that the system problem can not be detected and the network abnormity cannot be predetermined after the network in the distributed system in the related technology is abnormal.
The distributed system may be a software system based on a distributed system architecture, and the distributed system architecture includes various network architectures, such as a high concurrency architecture, a long-distance multi-activity architecture, a container flower architecture, a micro service architecture, a high availability architecture, an elasticization architecture, and the like.
The Object-based Storage device OSD may be a device based on an Object-based Storage OS (OS) network Storage architecture. The object storage device OSD can have only one access interface provided by its CPU, memory, network and disk system, and the OSD is different from the storage medium in the same device. The main functions of the OSD include data storage and secure access.
The data dropping process is a process that each object storage device OSD has, and specifically may be a system operation process of storing data in the object storage device OSD. The heartbeat message is a heartbeat message between a plurality of data landing processes.
The object storage device OSD is provided with a copy in the distributed system, the object storage device OSD and the copy are communicated, and the communication message is a communication message between the object storage device OSD and the copy of the object storage device OSD.
In this embodiment, whether the system is sub-healthy can be determined by detecting the heartbeat message and the communication message in the distributed system, and the system has a high probability of generating an abnormality in the sub-healthy state. The problem of network sub-health is found after the problem of network resource state in the related technology is solved, system performance is reduced, or other abnormalities are caused, and the abnormality can be wiped out, so that the method is not prospective.
The sub-health state of the network before the fault occurs is detected, wherein under the condition that the network is detected to be in the sub-health state, a system fault is developed after a period of time with a high probability. The determining whether the network between the object storage devices OSD is sub-healthy may be determining whether the network between the object storage devices OSD is in a sub-healthy state before a failure occurs.
Optionally, determining whether the network between the object storage devices OSD is sub-healthy according to the heartbeat message and the communication message includes: determining whether the times of network delay of communication messages or heartbeat messages exceeding first preset time exceeds preset times within a first preset time period in the past or not; and determining the network abnormality between the object storage devices OSD under the condition that the network delay of the communication message or the heartbeat message exceeds the first preset time times and exceeds the preset times.
The heartbeat message and the communication message are both file packets used for reflecting network transmission conditions between the object storage devices OSD in a distributed system, and the network of the heartbeat message and the communication message between the object storage devices OSD is in a sub-health state before a fault occurs, so that the time delay of the heartbeat message and the communication message is increased, and therefore whether the network between the object storage devices OSD is in the sub-health state or not can be determined through the time delay of the heartbeat message and/or the communication message.
Optionally, determining whether the network between the object storage devices OSD is sub-healthy according to the heartbeat message and the communication message includes: determining whether the network delay of the communication message exceeds a second preset time and exceeds a first preset multiple of the average delay of the historical data of the communication message in a third past preset time period within a second past preset time period; and determining the sub-health of the network between the object storage devices OSD under the condition that the network delay of the communication message exceeds the second preset time within the second preset time period and the network delay of the communication message exceeds the first preset multiple of the average delay of the historical data of the communication message within the third preset time period.
The communication messages are usually communicated according to requirements, so that whether the currently detected communication message is in a state before a fault occurs is determined by comparing the currently detected communication message with the historical communication messages. Specifically, according to comparison between historical message data, which are sent to the same address and have the same message size, and historical message data in historical data, when the communication message exceeds a second preset time and exceeds a first preset multiple of the average delay of the historical data of the communication message in a third past preset time period, it is determined that the network sub-health corresponding to the message may cause a system fault.
Optionally, determining whether the network delay of the communication packet exceeds the second preset time and exceeds a preset multiple of the delay of the third preset time period in the second preset time period includes: determining that the communication message which has the same sending address as the communication message and the same level of the file size is historical data, wherein the number of the levels of the file size is multiple, and the multiple levels are multiple preset ranges of the file size respectively.
Because the size of the communication message and the length of the communication path are different, the delay of the communication message is influenced, and therefore the communication message which has the same sending address as the communication message and the same file size and grade is selected as historical data. Thereby improving the accuracy of network failure prediction. The levels of the file sizes can be divided according to the file sizes, different communication messages are certainly different, and the probability that the communication messages are completely the same in size is small, so that two communication messages in the same level can be considered to be the same in magnitude by dividing the communication messages in different sizes in a level manner, and the delay influence of the sizes on the two communication messages can be ignored. Thereby, the history data can be effectively selected. So as to effectively determine whether the communication message is in sub-health state through historical data.
Optionally, determining whether the network between the object storage devices OSD is sub-healthy according to the heartbeat message and the communication message includes: determining network delays of all heartbeat messages in a storage pool where the object storage device OSD is located within a fourth preset time period, and determining an average network delay of the heartbeat messages according to the network delays of all the heartbeat messages in the storage pool; determining whether the average network delay of the heartbeat message exceeds a third preset time and a second preset multiple of the average network delay of the storage pool within a fourth preset time period; and in a fourth preset time period, determining the network sub-health between the object storage devices OSD under the condition that the average network delay of the heartbeat message exceeds the third preset time and exceeds a second preset multiple of the average network delay of the storage pool in the fourth preset time period.
The sizes of the heartbeat messages are the same, and the heartbeat messages of the object storage devices OSD in the same storage pool can be sent at the same time, so that whether the network corresponding to the heartbeat messages is in a sub-health state before failure can be determined by comparing the time delay of the heartbeat messages of the object storage devices OSD in the same storage pool with the time delay of the detected heartbeat messages. Specifically, the average network delay of all heartbeat messages in the storage pool where the heartbeat messages are located is determined, and under the condition that the delay of the heartbeat messages exceeds a third preset time and exceeds the average network delay, the sub-health of the network between the object storage devices OSD is determined.
Optionally, after determining whether the network between the object storage devices OSD is sub-healthy according to the heartbeat message and the communication message, the method further includes: and under the condition that the sub-health of the network between the object storage devices OSD is detected, sending a detection packet for detecting the network abnormity to the position of the abnormal network, and determining whether the network between the object storage devices OSD is actually abnormal or not according to the receiving condition of the detection packet.
The receiving condition of the detection packet includes whether the detection packet is delayed or not, the delay time, whether packet loss occurs or not and the like, and whether the network between the object storage devices OSD is really abnormal or not is determined under the conditions that delay occurs or the delay time exceeds the preset time or other faults such as packet loss occur. Therefore, whether the network between the OSD of the object storage equipment is subjected to one-field experimental detection or not is detected according to the detection packet, so that the accuracy and reliability of abnormal detection are improved.
Optionally, in a case that it is determined that the network ownership between the object storage devices OSD is abnormal, the method further includes: and isolating the network between the abnormal object storage devices OSD in the distributed system.
When determining that the network between the OSD of the object storage devices is abnormal, the network can be customized to avoid the influence on other networks in the system when the network fails.
It should be noted that this embodiment also provides an alternative implementation, and the implementation is described in detail below.
The embodiment solves the problem that the sub-health of the network is detected in time in the distributed system, but the sub-health of the network is not found until the network resource state is out of question, and the performance of the storage system is reduced or other abnormalities are caused at this time.
The distributed storage system ceph has heartbeat and IO communication between OSD and MON, and uses the communication mechanism of OSD and MON, mainly the heartbeat detection mechanism and the duplicate communication mechanism. The scheme has the advantages that the heartbeat and the communication of the distributed system are utilized, no additional network pressure is added, no pressure is brought to the distributed system, and the stability and the performance of the distributed system are not influenced.
Heartbeat mechanism of distributed system ceph: the two OSD are detected by periodic heartbeat, the OSD detects the survival state of the other side by periodically sending ping messages, for example, a heartbeat message is sent every 2 seconds, if no response message is received within a certain time, the OSD considers that the OSD at the other side is overtime, and the OSD is down after a plurality of times.
Copy communication mechanism of distributed system ceph: the distributed system supports multiple copies and ECs to provide data redundancy and protection functions. Ceph is strong consistency, for example, for multiple copies, after IO writes to the master OSD, the master OSD will write data to other OSDs at the same time; the same is true for EC, and the main OSD would be EC sliced, writing other slices to other OSDs.
Network model of distributed system ceph: the OSD and other OSD which are backups of each other have network links, ping messages can be sent regularly, and data copies can be sent through the network when IO exists.
By utilizing the heartbeat and communication mechanism of the distributed system, the network time of the packet is counted, and whether the link has network sub-health can be judged.
The time of transmission of different packet sizes on the network may be different, so the packet sizes are divided into different size sections, 0-16 k,16 k-64 k,64 k-256 k,256 k-1M, 1M-4M, and more than 4M. For the heartbeat message, the size is fixed, and the size sections do not need to be distinguished.
The OSD process periodically collects current network time data.
Comparing the collected packet network time with transverse, longitudinal and threshold values, and judging the sub-health of the network in multiple directions, specifically as follows:
1) judging a threshold value: if the number of times that the time is greater than a threshold, such as 1 second, exceeds 3 times over a past period of time, such as 1 minute, the link is deemed to be sub-healthy
2) And (3) longitudinal comparison: during comparison, comparison is carried out according to (ip address, packet size range and delay), namely the comparison is carried out to the same ip address and the same packet size range. A network is considered sub-healthy if the delay of the last period of time, e.g. 1 minute, exceeds a certain threshold, e.g. 256ms, and is 10 times or more higher than the delay of the last period of time, e.g. 1 hour.
3) And (3) transverse comparison: comparison between OSDs in the same memory pool. Generally, the OSD pressure in the same pool is comparable, otherwise there is an imbalance. The OSD process will periodically send its own network time data to the MON for some period of time, such as 1min,5min,15 min. The MON stores network delay data of 3 time periods of 1min,5min,15min and the like of each OSD, calculates an average value of the network delay of 1min,5min and 15min in the storage pool, traverses the network delay data of 1min,5min and 15min of all the OSD in the storage pool during comparison, and considers that the network is sub-healthy if the time of a certain OSD exceeds a certain threshold value, such as 256ms, and is 10 times higher than the average value of the network delay of the storage pool after the data of the OSD is removed.
4) After the network sub-health is detected by the means, the detection packet is sent according to the network link between the two OSD which detect the network sub-health in a targeted manner, the link time or the packet loss is identified, if the link time or the packet loss is confirmed, the sub-health network is isolated according to a network networking model, and the influence on the system is reduced.
The embodiment utilizes a mechanism of a ceph distributed system to count the time of heartbeat messages and copy/EC communication messages between OSD; dividing the IO size into different intervals to be counted respectively; comparing the link size with the OSD historical data, and judging that the network is sub-healthy if the link size and the packet size are the same; transverse comparison is carried out, comparison with other OSD in the storage pool is carried out, the OSD loads in the slave pool in the same storage are equivalent, and if the difference is large and exceeds a threshold value, the fact that the network is sub-healthy can be judged; and according to the preliminarily judged network sub-health, the detection packet of the network link is sent in a targeted manner to confirm whether the time is the time or the packet is lost.
Compared with the related technology, the embodiment can utilize the heartbeat and copy/EC communication messages of the ceph distributed storage system, does not increase any extra IO load, does not increase the pressure of the system, and does not influence the service; through transverse comparison and longitudinal comparison, the sub-health of the network is detected more accurately; the method can isolate sub-healthy network links, reduce the influence on a distributed system, and improve the robustness of the system, namely the robustness of the system.
Fig. 2 is a schematic diagram of a distributed storage system network sub-health detection apparatus according to an embodiment of the present invention, and as shown in fig. 2, according to another aspect of the embodiment of the present invention, there is also provided a distributed storage system network sub-health detection apparatus, including: a determination module 22 and a detection module 24, which are described in detail below.
The determining module 22 is configured to determine a heartbeat message between data dropdown processes of a plurality of object storage devices OSD in the distributed system, and a communication message between the object storage devices OSD and a copy of the object storage devices OSD, where the heartbeat message and the communication message are both transmitted through a network between the object storage devices OSD, and the distributed system includes the plurality of object storage devices OSD; and the detection module 24 is connected with the determination module 22 and is used for determining whether the network between the object storage devices OSD is sub-healthy or not according to the heartbeat message and the communication message.
By the device, heartbeat messages among data falling processes of a plurality of object storage devices OSD in the distributed system and communication messages among object storage devices OSD and copies of the object storage devices OSD are determined, wherein the heartbeat messages and the communication messages are transmitted through a network among the object storage devices OSD; the method for determining whether the network between the object storage devices OSD is sub-healthy or not according to the heartbeat message and the communication message achieves the purpose of predicting the abnormity of the distributed system by determining whether the network between the object storage devices OSD is sub-healthy or not through the heartbeat message between the object storage devices OSD of the distributed system, the copy of the object storage devices OSD and the communication message of the programming language EC, thereby achieving the technical effect of predetermining the abnormity of the distributed system, and further solving the technical problems that the system problem can be detected and the network abnormity cannot be predetermined after the network in the distributed system in the related technology is abnormal.
According to another aspect of the embodiments of the present invention, there is also provided a storage medium, where the storage medium includes a stored program, and when the program runs, a device in which the storage medium is located is controlled to execute the distributed system prediction method of any one of the above.
According to another aspect of the embodiments of the present invention, there is also provided a processor, where the processor is configured to execute a program, where the program executes a distributed system prediction method in any one of the above methods.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
In the above embodiments of the present invention, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In the embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units may be a logical division, and in actual implementation, there may be another division, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (9)
1. A distributed storage system network sub-health detection method is characterized by comprising the following steps:
determining heartbeat messages among data falling processes of a plurality of object storage devices OSD in a distributed system and communication messages among the object storage devices OSD and copies of the object storage devices OSD, wherein the heartbeat messages and the communication messages are transmitted through a network among the object storage devices OSD, and the distributed system comprises a plurality of object storage devices OSD;
determining whether the network between the object storage devices OSD is sub-healthy or not according to the heartbeat message and the communication message;
determining whether the network between the object storage devices OSD is sub-healthy according to the heartbeat message and the communication message comprises: determining whether the network delay of the communication message exceeds a second preset time and a first preset multiple of the average delay of a third past preset time period within a second past preset time period, wherein the average delay is the average network delay of the historical data of the communication message within the third past preset time period; and under the condition that the network delay of the communication message exceeds second preset time within the second preset time period and the network delay of the communication message exceeds a first preset multiple of the average delay of the historical data of the communication message within the third preset time period, determining that the network between the object storage devices OSD is sub-healthy.
2. The method of claim 1, wherein determining whether the network between the object storage devices OSDs is sub-healthy according to the heartbeat message and the communication message comprises:
determining whether the number of times that the network delay of the communication message or the heartbeat message exceeds first preset time exceeds preset number of times within a first preset time period in the past;
and determining the sub-health of the network between the object storage devices OSD under the condition that the network delay of the communication message or the heartbeat message exceeds the first preset time and the preset times.
3. The method of claim 1, wherein determining whether the network delay of the communication packet exceeds a second predetermined time and exceeds a predetermined multiple of the delay of a third predetermined time period within the second predetermined time period comprises:
determining the communication message with the same sending address as the communication message and the same file size grade as the historical data, wherein the file size grade is multiple, and the multiple grades are multiple preset ranges of the file size respectively.
4. The method of claim 1, wherein determining whether the network between the object storage devices OSDs is sub-healthy according to the heartbeat message and the communication message comprises:
determining network delays of all heartbeat messages in a storage pool where the object storage device OSD is located within a fourth preset time period, and determining an average network delay of the heartbeat messages according to the network delays of all the heartbeat messages in the storage pool;
determining whether the average network delay of the heartbeat message exceeds a third preset time and a second preset multiple of the average network delay of the storage pool within a fourth preset time period;
and determining the network sub-health between the object storage devices OSD under the condition that the average network delay of the heartbeat message exceeds a third preset time and a second preset multiple of the average network delay of the storage pool within a fourth preset time period.
5. The method of claim 1, wherein after determining whether the network between the object storage devices OSDs is sub-healthy according to the heartbeat message and the communication message, the method further comprises:
and under the condition that the sub-health of the network between the object storage devices OSD is detected, sending a detection packet for detecting the network abnormity to the position of the abnormal network, and determining whether the network between the object storage devices OSD is actually abnormal or not according to the receiving condition of the detection packet.
6. The method of claim 5, wherein in case it is determined that the network ownership between the object storage devices OSDs is abnormal, further comprising:
isolating the network between the object storage devices OSD in the distributed system, wherein the object storage devices OSD are abnormal.
7. A distributed storage system network sub-health detection device, comprising:
the system comprises a determining module, a judging module and a processing module, wherein the determining module is used for determining heartbeat messages among data falling processes of a plurality of object storage devices OSD in a distributed system and communication messages among the object storage devices OSD and a copy of the object storage devices OSD, the heartbeat messages and the communication messages are transmitted through a network among the object storage devices OSD, and the distributed system comprises a plurality of object storage devices OSD;
the detection module is used for determining whether the network between the object storage devices OSD is sub-healthy or not according to the heartbeat message and the communication message;
determining whether the network between the object storage devices OSD is sub-healthy according to the heartbeat message and the communication message comprises: determining whether the network delay of the communication message exceeds a second preset time and a first preset multiple of the average delay of a third past preset time period within a second past preset time period, wherein the average delay is the average network delay of the historical data of the communication message within the third past preset time period; and under the condition that the network delay of the communication message exceeds second preset time within the second preset time period and the network delay of the communication message exceeds a first preset multiple of the average delay of the historical data of the communication message within the third preset time period, determining that the network between the object storage devices OSD is sub-healthy.
8. A storage medium, characterized in that the storage medium comprises a stored program, wherein when the program runs, a device in which the storage medium is located is controlled to execute the distributed storage system network sub-health detection method according to any one of claims 1 to 6.
9. A processor, configured to execute a program, wherein the program executes to perform the distributed storage system network sub-health detection method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911371153.6A CN111142801B (en) | 2019-12-26 | 2019-12-26 | Distributed storage system network sub-health detection method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911371153.6A CN111142801B (en) | 2019-12-26 | 2019-12-26 | Distributed storage system network sub-health detection method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111142801A CN111142801A (en) | 2020-05-12 |
| CN111142801B true CN111142801B (en) | 2021-05-04 |
Family
ID=70520734
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911371153.6A Active CN111142801B (en) | 2019-12-26 | 2019-12-26 | Distributed storage system network sub-health detection method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111142801B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112328461A (en) * | 2020-10-29 | 2021-02-05 | 无锡先进技术研究院 | Performance bottleneck prediction method, equipment and storage medium based on distributed storage |
| CN113132160B (en) * | 2021-04-08 | 2023-02-28 | 山东英信计算机技术有限公司 | Method and system for detecting network sub-health state of client node |
| CN113687782B (en) * | 2021-07-30 | 2023-12-22 | 济南浪潮数据技术有限公司 | Storage pool time delay determining method and device, electronic equipment and readable storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2012110897A2 (en) * | 2011-02-17 | 2012-08-23 | Peretz Shlomo | Verifying the location of a mobile communication device |
| CN107547252A (en) * | 2017-06-29 | 2018-01-05 | 新华三技术有限公司 | A kind of network failure processing method and device |
| CN108235751A (en) * | 2017-12-18 | 2018-06-29 | 华为技术有限公司 | Identify the method, apparatus and data-storage system of object storage device inferior health |
| CN109828868A (en) * | 2019-01-04 | 2019-05-31 | 新华三技术有限公司成都分公司 | Date storage method, device, management equipment and dual-active data-storage system |
| JP2019531563A (en) * | 2017-04-14 | 2019-10-31 | 華為技術有限公司Ｈｕａｗｅｉ Ｔｅｃｈｎｏｌｏｇｉｅｓ Ｃｏ．，Ｌｔｄ． | Data processing method, storage system, and switching device |
-
2019
- 2019-12-26 CN CN201911371153.6A patent/CN111142801B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2012110897A2 (en) * | 2011-02-17 | 2012-08-23 | Peretz Shlomo | Verifying the location of a mobile communication device |
| JP2019531563A (en) * | 2017-04-14 | 2019-10-31 | 華為技術有限公司Ｈｕａｗｅｉ Ｔｅｃｈｎｏｌｏｇｉｅｓ Ｃｏ．，Ｌｔｄ． | Data processing method, storage system, and switching device |
| CN107547252A (en) * | 2017-06-29 | 2018-01-05 | 新华三技术有限公司 | A kind of network failure processing method and device |
| CN108235751A (en) * | 2017-12-18 | 2018-06-29 | 华为技术有限公司 | Identify the method, apparatus and data-storage system of object storage device inferior health |
| CN109828868A (en) * | 2019-01-04 | 2019-05-31 | 新华三技术有限公司成都分公司 | Date storage method, device, management equipment and dual-active data-storage system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111142801A (en) | 2020-05-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111142801B (en) | Distributed storage system network sub-health detection method and device | |
| US9009523B2 (en) | Method and apparatus for isolating a fault in a controller area network | |
| CN108989135B (en) | Network equipment fault detection method and device | |
| CN106789306B (en) | Method and system for detecting, collecting and recovering software fault of communication equipment | |
| EP3142011A1 (en) | Anomaly recovery method for virtual machine in distributed environment | |
| CN111385107B (en) | Main/standby switching processing method and device for server | |
| CN109245926B (en) | Intelligent network card, intelligent network card system and control method | |
| CN103761180A (en) | Method for preventing and detecting disk faults during cluster storage | |
| CN110601945A (en) | Dual-redundancy CAN bus communication system and communication method | |
| CN110677480A (en) | Node health management method and device and computer readable storage medium | |
| US20140298076A1 (en) | Processing apparatus, recording medium storing processing program, and processing method | |
| CN110740064A (en) | Distributed cluster node fault processing method, device, equipment and storage medium | |
| US9499174B2 (en) | Method and apparatus for isolating a fault-active controller in a controller area network | |
| US8208370B1 (en) | Method and system for fast link failover | |
| CN104994173A (en) | Message processing method and system | |
| CN109219105B (en) | Route switching method and route switching system | |
| JP2007280155A (en) | Reliability improving method in dispersion system | |
| CN112367386B (en) | Ignite-based automatic operation and maintenance method and device and computer equipment | |
| WO2014040470A1 (en) | Alarm message processing method and device | |
| CN110321261B (en) | Monitoring system and monitoring method | |
| CN114884803A (en) | Method, device, equipment and medium for processing multiple redundant states | |
| CN104348676A (en) | Link detection method and device based on operation administration and maintenance | |
| KR101909264B1 (en) | System and method for fault recovery of controller in separated SDN controller | |
| CN101729349A (en) | RRPP-based detection method and device for connectivity of main ring access | |
| CN111934909A (en) | Method and device for switching IP (Internet protocol) resources of host and standby machine, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address | ||
| CP03 | Change of name, title or address |
Address after: 100094 101, floors 1-5, building 7, courtyard 3, fengxiu Middle Road, Haidian District, Beijing Patentee after: Beijing Xingchen Tianhe Technology Co.,Ltd. Address before: 100097 room 806-1, block B, zone 2, Jinyuan times shopping center, indigo factory, Haidian District, Beijing Patentee before: XSKY BEIJING DATA TECHNOLOGY Corp.,Ltd. |