CN115167757A - Acceleration card distributed storage access method, device, equipment and storage medium - Google Patents

Acceleration card distributed storage access method, device, equipment and storage medium Download PDF

Info

Publication number
CN115167757A
CN115167757A CN202210660002.8A CN202210660002A CN115167757A CN 115167757 A CN115167757 A CN 115167757A CN 202210660002 A CN202210660002 A CN 202210660002A CN 115167757 A CN115167757 A CN 115167757A
Authority
CN
China
Prior art keywords
server
virtual
accelerator card
access path
distributed storage
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210660002.8A
Other languages
Chinese (zh)
Inventor
管树发
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
New H3C Technologies Co Ltd
Original Assignee
New H3C Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by New H3C Technologies Co Ltd filed Critical New H3C Technologies Co Ltd
Priority to CN202210660002.8A priority Critical patent/CN115167757A/en
Publication of CN115167757A publication Critical patent/CN115167757A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/061—Improving I/O performance
    • G06F3/0611—Improving I/O performance in relation to response time
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
    • G06F3/0647—Migration mechanisms
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0662—Virtualisation aspects
    • G06F3/0665—Virtualisation aspects at area level, e.g. provisioning of virtual or logical volumes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G06F2009/45562—Creating, deleting, cloning virtual machine instances
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G06F2009/4557—Distribution of virtual machine instances; Migration and load balancing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G06F2009/45575—Starting, stopping, suspending or resuming virtual machine instances
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G06F2009/45591—Monitoring or debugging support
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G06F2009/45595—Network integration; Enabling network access in virtual machine instances

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Software Systems (AREA)
  • Hardware Redundancy (AREA)

Abstract

The invention provides an accelerator card distributed storage access method, device, equipment and storage medium, which are used for improving the continuous effectiveness of distributed storage service of an accelerator card. The invention configures the main and standby access paths of distributed storage for a virtualization system, wherein the main path accesses the distributed storage through an accelerator card, and the standby path accesses the distributed storage through a common network card. The virtualization system detects a local accelerator card fault through a heartbeat detection mechanism. When the local accelerator card fails, the virtual volume controller bound with the virtual IP is migrated to a node where other virtual IPs in the protection group are located, and the virtualization system switches the access path of the distributed storage from the main access path to the standby access path. The invention can reduce the service interruption time caused by the failure of the accelerator card and improve the continuous effectiveness of the distributed storage service based on the accelerator card.

Description

Acceleration card distributed storage access method, device, equipment and storage medium
Technical Field
The invention relates to the technical field of communication and cloud computing, in particular to a distributed storage access method, a distributed storage access device, distributed storage access equipment and a storage medium for an accelerator card.
Background
Distributed storage is the decentralized storage of data on a plurality of independent devices. The traditional network storage system adopts a centralized storage server to store all data, the storage server becomes the bottleneck of the system performance, is also the focus of reliability and safety, and cannot meet the requirement of large-scale storage application.
The distributed storage system adopts an expandable system structure, utilizes a plurality of storage servers to share the storage load, and utilizes the position server to position the storage information, thereby not only improving the reliability, the availability and the access efficiency of the system, but also being easy to expand.
The accelerator card is also called a smart card, and compared with a common network card, the accelerator card has an independent central processing unit CPU, and is generally called a System On a Chip (SoC) in the industry. The accelerator card has certain network and storage acceleration (including offload) capabilities, and has different implementation modes such as an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), and the like.
In the prior art, a server where a virtualization system is located may be configured with a Solid State Disk (SSD), and an accelerator card may be directly connected to an SSD data Disk by using miniSAS (Serial Attachment SCSI) or other types of hard Disk connectors. The SoC on the accelerator card, independent of the physical server, may run distributed storage services software that may manage multiple SSD data disks. The accelerator card panel may be configured with either a 10GE or 25GE portal through which it connects to the switch. Through multi-copy or erasure coding techniques, distributed storage services on multiple accelerator cards may be grouped into distributed storage clusters.
The reliability of accelerator card based distributed storage clusters is achieved by multi-copy or erasure coding techniques. For example, the minimum distributed storage cluster includes 3 storage nodes. For a distributed storage cluster with a certain number of storage nodes, when an individual accelerator card fails, the storage cluster can also provide storage service if the number of other storage nodes in the cluster can still meet the requirement by means of data balance and the like.
However, when the virtualization system is connected to the switch through the accelerator card, when the accelerator card fails, there is no connection channel between the virtualization system and the switch, which may cause the virtualization system on the server with the failed accelerator card to no longer access the distributed storage, and thus cause the service to be unable to continue.
Disclosure of Invention
In view of this, the present invention provides an accelerator card distributed storage access method, apparatus, device and storage medium, which are used to improve the continuous effectiveness of the accelerator card distributed storage service.
Based on one aspect of the embodiments of the present invention, the present invention provides an accelerator card distributed storage access method, which is applied to a first server configured with an accelerator card and a common network card, and includes:
a virtualization system running on a main board CPU of a first server preferentially accesses distributed storage through a main access path, wherein the main access path is a path to a virtual volume controller bound with a first virtual IP address on a local accelerator card;
when the first server accelerator card has a fault, switching an access path of distributed storage to a standby access path, wherein the standby access path is an access path from a local common network card to a virtual volume controller bound with the first virtual IP address on the accelerator card of a second server;
and after the fault repair of the accelerator card of the first server is on line again and the distributed storage service on the accelerator card is started, migrating the first virtual IP address back and switching the access path of the distributed storage back to the main access path.
Further, before running the distributed storage service on the accelerator card of the first server, the method further comprises:
configuring a first virtual IP address for a virtual volume controller on an accelerator card of a first server;
creating a protection group for the first virtual IP address on an accelerator card of the first server, and adding the virtual IP addresses distributed to the virtual volume controllers of other storage nodes into the created protection group;
the method comprises the steps of configuring a shared storage service on an accelerator card of a first server, and binding the shared storage service with a first virtual IP address.
Further, the method further comprises:
establishing a heartbeat detection relationship between a virtualization system of a first server and a virtual volume controller on a local acceleration card to detect whether the local acceleration card fails;
when the fault of an accelerator card of a first server is detected, triggering the switching operation from a main access path to a standby access path;
after the first virtual IP is migrated to a virtual volume controller on an accelerator card of a second server, a virtualization system of the first server hangs a storage interface to a shared storage service bound with the first virtual IP on the second server;
and the second server is a server determined by election of other storage nodes after detecting the failure of the accelerator card of the first server.
Further, the method further comprises:
establishing a heartbeat detection relationship between a virtual volume controller on an accelerator card of a first server and virtual volume controllers bound to other virtual IP addresses in the protection group to detect whether the accelerator cards on other storage nodes are in failure;
and when detecting that the accelerator cards on other storage nodes in the protection group are in fault through the heartbeat detection relationship, electing the third server based on a preset election strategy.
Further, after the accelerator card of the first server is repaired to be online again and the distributed storage service on the first server is started, the first virtual IP address is migrated back, and the method for switching the access path of the distributed storage back to the main access path includes:
after the distributed storage service on the accelerator card of the first server is started, triggering the virtual IP address allocated to the first server to be migrated back to the first server based on a local priority principle;
the virtualization system of the first server triggers the switching of the main access path and the standby access path based on a main access path priority principle;
and after the virtualization system of the first server waits for the completion of the read-write operation of all the volumes under the virtual volume controller bound to the first virtual IP address, hanging the storage interface to the shared storage service on the local accelerator card.
Based on an aspect of the embodiments of the present invention, the present invention further provides an accelerator card distributed storage access apparatus, where the apparatus is applied to a first server configured with an accelerator card and a common network card, and the apparatus includes:
the virtualization system module runs on a main board CPU of the first server and is used for preferentially accessing the distributed storage through a main access path, wherein the main access path is a path to a virtual volume controller bound with a first virtual IP address on a local accelerator card; when the first server accelerator card has a fault, accessing the distributed storage through a standby access path, wherein the standby access path is an access path from a local common network card to a virtual volume controller which is located on the accelerator card of the second server and is bound with the first virtual IP address
The distributed storage service module runs on a CPU (central processing unit) on an accelerator card of the first server and is used for providing distributed storage service;
the virtualization system module includes:
a storage access module for accessing the distributed storage through the virtual volume controller and the shared storage service bound to the first virtual IP address;
the fault detection module is used for detecting whether the local accelerator card is in fault;
the path switching module is used for switching the access path of the distributed storage from the main access path to the standby access path when the fault of the local accelerator card is detected; and after the fact that the local accelerator card is repaired to be online again and the distributed storage service on the local accelerator card is started is detected, the first virtual IP address is migrated back, and then the access path of the distributed storage is switched back to the main access path.
Further, the fault detection module establishes a heartbeat detection relationship with a virtual volume controller on the local accelerator card to detect whether the local accelerator card is in fault, and when the local accelerator card is detected to be in fault, the path switching module is triggered to execute switching operation from the main access path to the standby access path;
and after the first virtual IP is migrated to the virtual volume controller on the accelerator card of the second server, the path switching module hangs the storage interface of the storage access module to the second server to bind the shared storage service of the first virtual IP.
Further, the distributed storage service module includes:
the heartbeat detection module is used for establishing a heartbeat detection relation with the virtualization system module; the virtual volume controller is also used for establishing a heartbeat detection relation with the virtual IP address which is distributed to other storage nodes in a protection group to detect whether the accelerator cards on the other storage nodes are in fault or not, and the protection group comprises the first virtual IP address and a plurality of virtual IP addresses distributed to the other storage nodes;
and the election receiving module is used for electing the third server based on a preset election strategy when detecting that the accelerator cards on other storage nodes in the protection group are in failure.
The invention configures the main and standby access paths of distributed storage for a virtualization system, wherein the main path accesses the distributed storage through an accelerator card, and the standby path accesses the distributed storage through a common network card. And running the distributed storage service on the accelerator card and creating distributed shared storage, mounting the shared storage to the virtual IP address bound by the virtual volume controller, and establishing a virtual IP protection group. The accelerator cards detect faults of other accelerator cards in the protection group through a heartbeat detection mechanism, and the virtualization system detects faults of the local accelerator cards through the heartbeat detection mechanism. When the local accelerator card fails, the distributed storage service based on the virtual IP is migrated to a selected node in the protection group, and the virtualization system switches the access path of the distributed storage from the main access path to the standby access path. The invention can reduce the service interruption time caused by the failure of the accelerator card and improve the continuous effectiveness of the distributed storage service based on the accelerator card.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments of the present invention or the description in the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the present invention, and it is also possible for those skilled in the art to obtain other drawings according to the drawings of the embodiments of the present invention.
FIG. 1 is a schematic diagram illustrating a structural principle of a server to which the accelerator card distributed storage access method provided by the present invention is applied;
FIG. 2 is a flowchart illustrating steps of a method for accelerator card distributed storage access according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of an electronic device capable of implementing the accelerator card distributed storage access method provided by the present invention.
Detailed Description
The terminology used in the embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the embodiments of the invention. As used in this embodiment of the invention, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The term "and/or" as used herein is meant to encompass any and all possible combinations of one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used herein to describe various information in embodiments of the present invention, such information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of embodiments of the present invention. Depending on the context, moreover, the word "if" is used may be interpreted as "at … …" or "at … …" or "in response to a determination".
The invention aims to improve a distributed storage access method provided with an accelerator card and a common network card, so that a virtualization system on a server can still normally access a storage cluster when the accelerator card fails, and the continuous effectiveness of services is guaranteed.
In order to achieve the purpose of the invention, the invention provides an accelerator card distributed storage access method, and the basic idea of the invention is as follows: and configuring a main/standby access path of distributed storage for the virtualization system, wherein the main path accesses the distributed storage through an accelerator card, and the standby path accesses the distributed storage through a common network card. And allocating a virtual IP address for accessing the distributed storage for each storage node, and establishing a virtual IP protection group. The shared storage is mounted to the virtual IP address to which the virtual volume controller is bound. When the local accelerator card fails, the virtual volume controller bound with the virtual IP is migrated to a node where other virtual IPs in the protection group are located, and the virtualization system switches the access path of the distributed storage from the main access path to the standby access path. The invention can reduce the service interruption time caused by the failure of the accelerator card and improve the continuous effectiveness of the distributed storage service based on the accelerator card.
Based on the basic idea of the invention, the following describes a specific implementation process of the invention with reference to the accompanying drawings and specific embodiments.
Fig. 1 is a schematic structural diagram of a server to which the accelerator card distributed storage access method provided by the present invention is applied. The virtualization system operates on a CPU of a Server (Server) motherboard, and the Server motherboard CPU may adopt an x 86-series CPU, which is not specifically limited in the present invention. One or more Virtual Machines (VMs) may be virtualized in a virtualized system. The distributed storage service runs on an independent CPU of the accelerator card, and the CPU of the accelerator card may be in a SoC physical structure form, which is not specifically limited in the present invention. The normal network card is relative to the accelerator card, and means the network card without an independent CPU.
The acceleration card carries out two-layer interconnection through the switch, the common network card carries out two-layer interconnection through the interactive machine, and the common network card can carry out three-layer interconnection through the switch cross-network segment and the virtual volume controller of the distributed storage service in the acceleration card.
The SSD can be hung on an accelerator card through a Non-Volatile Memory host controller interface specification (NVMe) to serve as a data disk for distributed storage, distributed storage services on a Server1, a Server2 and a Server3 form a distributed storage cluster, and each Server is equivalent to one storage node in the distributed storage cluster. The distributed storage cluster provides the virtual volume for the virtualization system according to a preset storage strategy.
Fig. 2 is a flowchart of steps of a method for accelerator card distributed storage access according to an embodiment of the present invention, where the method is applied to a first server configured with an accelerator card and a generic network card, and the method includes:
step 200, a virtualization system running on a main board CPU of a first server preferentially accesses distributed storage through a main access path, wherein the main access path is a path to a virtual volume controller bound with a first virtual IP address on a local accelerator card;
the virtualization system in the embodiment of the invention can access the logical volume provided by the distributed storage cluster through two paths, one is a main access path through the accelerator card, and the other is a standby access path through the common network card. And under the condition that the main access path and the standby access path are normal, the main access path is preferentially used for accessing the distributed storage by default, and the standby access path is switched to under the condition that the local accelerator card fails. Taking the Server1 in fig. 1 as an example, the path (1) is a main access path, and the path (2) is a backup access path.
In the main access path, the virtualization system is in the same two-tier network as the virtual volume controller of the distributed storage service located on the accelerator card, for example, assigning the virtual volume controller a network segment address in the form of 192.168.1. X/24. The standby access path accesses the distributed storage service operated by the acceleration card by adopting a three-layer access mode of spanning network segments, for example, a network segment address in the shape of 192.168.2.X/24 is distributed for a common network card, a gateway address configured for a virtual machine is an interface address on a switch, for example, 192.168.2.Y/24, and two network segments of 192.168.1.0/24 and 192.168.2.0/24 are intercommunicated on the switch by configuration.
Before the distributed storage service is run on the acceleration card of the first server, the method further comprises the following steps of:
step 201, a first virtual IP address is configured for a virtual volume controller on an accelerator card of a first server.
Taking fig. 1 as an example, the distributed storage services running on the Server1 accelerator card 1, the Server2 accelerator card 2, and the Server3 accelerator card 3 form a distributed storage cluster, each distributed storage service has its own virtual volume controller, and each virtual volume controller binds its own virtual IP address.
Before the distributed storage service on the accelerator card runs, a first virtual IP address is allocated to the virtual volume controller on the accelerator card, and the virtual volume controller on the node is bound to the allocated first virtual IP address. For example, the first virtual IP address assigned to the virtual volume controller on the accelerator card 1 is 192.168.1.1. Likewise, the first virtual IP address assigned on Server2 for the virtual volume controller on the accelerator card 2 may be 192.168.1.2, and the first virtual IP address assigned on Server3 for the virtual volume controller on the accelerator card 3 may be 192.168.1.3.
Step 202, establishing a protection group for the first virtual IP address on an accelerator card of the first server, and adding the virtual IP addresses distributed to the virtual volume controllers of other storage nodes into the established protection group;
taking the Server1 in fig. 1 as an example, a protection Group1 is created for a first virtual IP address 192.168.1.1 allocated to a virtual volume controller on an accelerator card 1, and first virtual IP addresses 192.168.1.2 and 192.168.1.3 allocated to virtual volume controllers on an accelerator card 2 and an accelerator card 3 are added to the protection Group1, where the first virtual IP address of the accelerator card 1 is a primary virtual IP address, the first virtual IP addresses of the accelerator card 2 and the accelerator card 3 are standby virtual IP addresses, correspondingly, a virtual volume controller bound to the primary virtual IP address is referred to as a primary virtual volume controller, and a virtual volume controller bound to the standby virtual IP address is referred to as a standby virtual volume controller, but it should be noted that the primary and standby are relative to a protection Group on a fixed node, the virtual IP addresses in the protection Group on each Server are different, the primary virtual IP address is an IP address fixedly allocated to a designated node for accessing a storage Server, and the primary and standby virtual IP addresses indicate that the Server can undertake a distributed storage node access task under a local storage failure condition. Similarly, a Group2 is created in the Server2, and the first virtual IP addresses of the accelerator card 1 and the accelerator card 3 are added to the Group2, where the first virtual IP address of the accelerator card 2 is the primary virtual IP address, and the first virtual IP addresses of the accelerator card 1 and the accelerator card 3 are the standby virtual IP addresses. And creating a Group3 in the Server3, and adding the first virtual IP addresses of the accelerator card 1 and the accelerator card 2 into the Group3, wherein the first virtual IP address of the accelerator card 3 is a main virtual IP address, and the first virtual IP addresses of the accelerator card 1 and the accelerator card 2 are standby IP virtual addresses.
Under the condition that all local accelerator cards are normal, the virtualization system on each Server accesses the virtual volume provided by the distributed storage cluster through the virtual volume controller of the local accelerator card on the main access path, so that load balance among the nodes can be realized.
Step 203, configuring a shared storage service on the local accelerator card, and binding the shared storage service with the first virtual IP;
as an example of the Server1 in fig. 1, the shared storage service configured for the accelerator card 1 is NVMe of target and iscsi target services, and the shared storage service is bound to the first virtual IP address allocated to the virtual volume controller on the accelerator card 1.
And step 204, adding the local nonvolatile memory to the storage nodes of the distributed storage cluster.
As in the example of fig. 1, for example, the SSD solid state disk of the local NVMe interface is added to the distributed storage service located on the local accelerator card, and the distributed storage cluster constructs a virtual volume based on the SSD disk added by each storage node.
When the virtualization system is initialized, the virtualization system further comprises the following steps:
step 205, the virtualization system hooks the storage interface to the shared storage service on the local accelerator card by default.
As in the example of FIG. 1, the virtualization system uses the NVMe of initiator/iscsi initiator tool to connect the storage interface to the NVMe of target/iscsi target shared storage service on the local accelerator card.
After the accelerator card is successfully started and the virtual volume controller on the accelerator card is loaded and run, the virtualization system further executes the following steps in order to detect whether the accelerator card is in failure:
and step 206, establishing a heartbeat detection relationship between the virtualization system of the first server and the virtual volume controller on the local accelerator card, and detecting whether the accelerator card fails.
In an embodiment of the present invention, the virtualization system periodically writes a heartbeat count into a heartbeat counter heartbeat1 of the virtual volume controller on the local accelerator card, and determines whether the accelerator card fails by whether the heartbeat count is successfully written. Or the virtual volume controllers on the virtualization system and the local accelerator card mutually send and write heartbeat counts to the opposite side or send heartbeat messages, and the accelerator card timely writes the heartbeat counts to the virtualization system or sends the heartbeat messages after the fault is repaired, so that the virtualization system can timely acquire the running states of the accelerator card and the service on the accelerator card. If the virtualization system does not receive the response of the heartbeat message sent by the accelerator card within the specified time or for multiple times, the accelerator card can be judged to have a fault.
After a main virtual volume controller on a local accelerator card bound by a main virtual IP in a protection group is started, a heartbeat detection relation is established with a virtual volume controller bound by a standby virtual IP address in the protection group so as to detect whether an accelerator card of a server where the standby virtual IP is located is in fault, and the detection method comprises the following steps:
and step 207, establishing a heartbeat detection relationship between the main virtual volume controller bound with the main virtual IP address on the local accelerator card and the virtual volume controllers located in other servers and bound with the standby virtual IP address in the protection group so as to detect whether the accelerator cards on other servers have faults or not.
In an embodiment of the present invention, under the condition that the accelerator cards are all normal, a heartbeat detection relationship is established between the primary virtual volume controller in each server and the virtual volume controller on the server node corresponding to the virtual IP address in the protection group, whether the accelerator cards on other server nodes have a fault is detected by writing a heartbeat count into the heartbeat counter heartbeat0 of the opposite server, and if the heartbeat count is not written, the accelerator cards on the corresponding servers are determined to have a fault.
Taking fig. 1 as an example, when the virtual volume controller in the Server1 accelerator card 1 periodically writes heartbeat counts into heartbeat0 counters of virtual volume controllers on the Server2 and the Server3, the virtual volume controllers on the Server2 and the Server3 also write heartbeat counts into heartbeat0 counters corresponding to the virtual volume controllers in the Server1, if the accelerator card 1 fails and the virtual volume controllers on the Server2 and the Server3 determine that the accelerator card 1 fails when the heartbeat counts are written, the virtual volume controllers on the Server2 and the Server3 select a Server, i.e., a second Server, capable of assuming the distributed storage access task assumed by the virtual volume controller on the accelerator card 1 on the Server1 according to a preset consistent election rule, and create a virtual volume controller on the selected second Server and bind a first virtual IP address allocated to the Server 1.
Similarly, if the accelerator card 1 on the Server1 is normal, when the accelerator card 1 detects that the accelerator card 2 has a fault, a third Server is generated by election with the accelerator card 3, the third Server generated by election is used for bearing the load of the distributed storage service on the failed accelerator card, the third Server may be a Server3 behind the Server1, and the accelerator card of the third Server receives the migration of the 192.168.1.2 virtual IP address to provide the distributed storage service for the distributed system on the Server2.
And 210, when the first server acceleration card fails, switching the access path of the distributed storage to a standby access path, wherein the standby access path is the access path from a local common network card to a virtual volume controller which is located on a second server acceleration card and is bound with the first virtual IP address.
When the local virtualization system detects the fault of the local accelerator card through the heartbeat detection relation with the virtual volume controller on the local accelerator card, the switching from the main access path to the standby access path is executed. The accelerator card failure may be due to a software crash of the distributed storage service or due to a hardware failure of the accelerator card, and all the software and hardware failures which cause the accelerator card not to provide the distributed storage service are collectively referred to as accelerator card failures in the present invention.
The switching step from the main access path to the standby access path comprises the following steps:
step 211, when the virtual volume controller on the server corresponding to the virtual IP address in the protection group detects the failure of the accelerator card on the first server through the heartbeat detection relationship, a second server receiving the virtual volume controller on the first server is elected according to a preset election rule;
as in the example of fig. 1, when the accelerator card 1 fails, the accelerator card 2 and the accelerator card 3 may elect a second Server capable of receiving the virtual volume controller on the first Server according to a preset election rule, for example, assume that the elected second Server is Server2. The preset election rule may be a rule for selecting a node with a smaller load based on a load balancing policy, or may be a simple rule according to the size of the IP address, and the present invention is not limited specifically.
Step 212, a virtual volume controller is established on an accelerator card of the second server, a first virtual IP address is bound, and shared storage service bound by the first virtual IP address is started;
after the election is completed, server2 newly creates a virtual volume controller and binds to the first virtual IP address 192.168.1.1 allocated for Server1, and then starts the 192.168.1.1 bound shared storage service of distributed storage, such as NVMe of target and iscsi target services. The shared storage target service of the first virtualized IP address 192.168.1.1 is provided by the virtual volume controller on accelerator card 2 to the virtualization system on Server 1.
Step 213, when detecting a failure of the local accelerator card based on a heartbeat detection relationship between the virtualization system on the first server and the local accelerator card, executing access path switching, and linking the storage interface to a shared storage service bound with a first virtual IP address on the accelerator card of the second server;
as with the example of FIG. 1, a virtualization system on a first server may use an NVMe of initiator/iscsi initiator tool to connect a storage interface to an NVMe of target/iscsi target shared storage service on an accelerator card of a second server to which a first virtual IP address has been bound.
After the path switching, the virtual system on the first Server1 actually accesses the shared storage service bound with the first virtual IP address 192.168.1.1 on the acceleration card 2 of the second Server2 across three layers through the two-layer network where the common network card is located, and after the connection is successful, the VM on the Server1 can read and write the virtual volume normally.
And step 220, after the accelerator card fault recovery of the first server is on line again and the distributed storage service on the first server is started, migrating the first virtual IP address back and switching the access path of the distributed storage back to the main access path.
After the accelerator card 1 on the Server1 is repaired in fault, the accelerator card 1 is on-line again, a distributed file storage copy is reconstructed on the accelerator card 1, the virtualization system executes switching from the standby access path to the main access path after sensing that the accelerator card 1 is on-line again and waiting for the volume controller to be idle, and the switching method specifically comprises the following steps:
step 221, after the accelerator card of the first server is repaired and comes online again, starting the distributed storage service based on the previous distributed storage configuration, setting the distributed storage service to be in an inactive state, and rebuilding a data copy of the distributed storage cluster;
the failure of the accelerator card 1 on the Server1 may cause damage to the data copy of the distributed storage cluster on the Server1 or inconsistency of data, so that after the failure of the accelerator card 1 is repaired, the distributed storage service on the accelerator card 1 finds that the accelerator card 1 fails before the current start through the system state identifier, and the distributed storage service firstly reconstructs the data copy, so that the data copy in the distributed storage cluster formed by the distributed storage nodes on the Server1, the Server2 and the Server3 is synchronized and consistent.
The on-line mode refers to that the accelerator card serves as an independent system, an operating system and an application APP on the accelerator card are loaded and completed, and the loading of the storage software APP running the virtual volume controller is completed. The inactive state refers to a standby state, and the nodes in the standby state can synchronize data to reconstruct a copy from the storage nodes in the Active main state. The virtual IP address can not be bound when the device is in the inactive state, and the virtual IP address can be bound at the moment when the standby state is switched to the Active main state.
After the accelerator card 1 is repaired due to a failure, in the case that the configuration made for the distributed storage service before the failure is not lost and does not need to be modified, the distributed storage service is still started based on the previous configuration, including the configuration of the virtual IP address and the protection group.
Step 222, reconfiguring the shared storage service on the accelerator card of the first server, and adding the local nonvolatile memory into a storage node of the distributed storage cluster;
for example, on an accelerator card 1 of Server1, NVMe of target and iscsi target services are configured, and then a local solid state disk (local NVMe disk) is added to a storage node of a distributed storage cluster located in Server1
Step 223, after the distributed storage service on the accelerator card of the first server is started, triggering the virtual IP address allocated to the first server to be migrated back to the first server based on a local priority principle;
for example, after the distributed storage service on the accelerator card 1 of the Server1 is started, the first virtual IP address 192.168.1.1 is found to be bound to the virtual volume controller (for example, found by means of ping the first virtual address), and since 192.168.1.1 is allocated to the Server1, the process of migrating the virtual IP address 192.168.1.1 back to the accelerator card 1 is triggered based on the local priority principle.
And 224, reestablishing the heartbeat detection relationship with the local virtualization system, and reestablishing the heartbeat detection relationship with the virtual volume controllers on other acceleration cards in the protection group.
After the migration processing of the first virtual IP address, the distributed storage service on the accelerator card may send a notification message to the local virtualization system to notify the virtualization system that the virtual IP address has been migrated back, may reestablish the heartbeat detection relationship, and start the path switching.
And step 225, the virtualization system of the first server triggers the switching of the main access path and the standby access path based on the main access path priority principle.
After sensing that the local virtual volume controller is successfully started through a notification message or a heartbeat detection relation of a local distributed storage service, the virtualization system of the Server1 triggers a switching operation from the standby access path to the main access path based on a main access path priority principle.
Step 226, after the virtualization system of the first server waits for the completion of the read-write operation of all the volumes under the virtual volume controller bound to the first virtual IP address, the virtualization system of the first server attaches the storage interface to the shared storage service on the local accelerator card.
Since the virtualized system on the Server1 may still read and write data in the distributed shared storage through the backup access path before the virtual IP address 192.168.1.1 is migrated back from the Server2, the Server1 is required to stop a new read and write request and wait for the completion of the execution of the executing read and write request to start the migration and path switching of the virtual IP address.
After waiting for the completion of all the volume read-write operations under the 192.168.1.1 virtual volume controller, the virtualization system of the Server1 executes the migration operation of 192.168.1.1, and rebinding the virtual IP address 192.168.1.1 allocated to the Server1 to the virtual volume controller on the accelerator card 1.
After the migration of the virtual IP address is completed, the virtualization system of the Server1 uses the NVMe of initiator/iscsi initiator tool to reconnect the NVMe of target/iscsi target distributed sharing service on the accelerator card 1.
Therefore, after the fault of the accelerator card is repaired, the switch from the standby access path to the main access path is completed.
Fig. 3 is a schematic structural diagram of an electronic device capable of implementing the accelerator card distributed storage access method provided by the present invention, where the device 300 includes: a processor 310, such as a Central Processing Unit (CPU), a communication bus 320, a communication interface 340, and a storage medium 330. Wherein the processor 310 and the storage medium 330 may communicate with each other through a communication bus 320. The storage medium 330 stores therein a computer program that, when executed by the processor 210, implements the functionality of one or more steps of the accelerator card distributed storage access method provided by the present invention.
The storage medium may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. In addition, the storage medium may be at least one memory device located remotely from the processor. The Processor may be a general-purpose Processor including a Central Processing Unit (CPU), a Network Processor (NP), etc.; but also Digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.
It should be recognized that embodiments of the present invention can be realized and implemented by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory memory. The method may be implemented in a computer program using standard programming techniques, including a non-transitory storage medium configured with the computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner. Each program may be implemented in a high level procedural or object oriented programming language to communicate with a computer system. However, the program(s) can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Furthermore, the program can be run on a programmed application specific integrated circuit for this purpose. Further, operations of processes described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and/or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) collectively executed on one or more processors, by hardware, or combinations thereof. The computer program includes a plurality of instructions executable by one or more processors.
Further, the method may be implemented in any type of computing platform operatively connected to a suitable interface, including but not limited to a personal computer, mini computer, mainframe, workstation, networked or distributed computing environment, separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, and the like. Aspects of the invention may be embodied in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optically read and/or write storage medium, RAM, ROM, or the like, such that it may be read by a programmable computer, which when read by the storage medium or device, is operative to configure and operate the computer to perform the procedures described herein. Further, the machine-readable code, or portions thereof, may be transmitted over a wired or wireless network. The invention described herein includes these and other different types of non-transitory computer-readable storage media when such media include instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. The invention also includes the computer itself when programmed according to the methods and techniques described herein.
The above description is only an example of the present invention, and is not intended to limit the present invention. Various modifications and alterations to this invention will become apparent to those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (10)

1. An accelerator card distributed storage access method is applied to a first server configured with an accelerator card and a common network card, and comprises the following steps:
a virtualization system running on a main board CPU of a first server preferentially accesses distributed storage through a main access path, wherein the main access path is a path to a virtual volume controller bound with a first virtual IP address on a local accelerator card;
when a first server accelerator card has a fault, switching an access path of distributed storage to a standby access path, wherein the standby access path is an access path from a local common network card to a virtual volume controller bound with the first virtual IP address on an accelerator card positioned on a second server;
and after the accelerator card fault recovery of the first server is on line again and the distributed storage service on the first server is started, migrating the first virtual IP address back and switching the access path of the distributed storage back to the main access path.
2. The method of claim 1, wherein prior to running the distributed storage service on the accelerator card of the first server, the method further comprises:
configuring a first virtual IP address for a virtual volume controller on an accelerator card of a first server;
creating a protection group for the first virtual IP address on an accelerator card of the first server, and adding the virtual IP addresses distributed to the virtual volume controllers of other storage nodes into the created protection group;
the method comprises the steps of configuring a shared storage service on an accelerator card of a first server, and binding the shared storage service with a first virtual IP address.
3. The method of claim 2, further comprising:
establishing a heartbeat detection relationship between a virtualization system of a first server and a virtual volume controller on a local acceleration card to detect whether the local acceleration card fails;
when the fault of an accelerator card of a first server is detected, triggering the switching operation from a main access path to a standby access path;
after the first virtual IP is migrated to a virtual volume controller on an accelerator card of a second server, a virtualization system of the first server hangs a storage interface to a shared storage service bound with the first virtual IP on the second server;
and the second server is a server determined by election of other storage nodes after detecting the failure of the accelerator card of the first server.
4. The method of claim 2, further comprising:
establishing a heartbeat detection relationship between a virtual volume controller on an accelerator card of a first server and virtual volume controllers bound to other virtual IP addresses in the protection group to detect whether the accelerator cards on other storage nodes are in failure;
and when detecting that the accelerator cards on other storage nodes in the protection group are in fault through the heartbeat detection relationship, electing the third server based on a preset election strategy.
5. The method of claim 2, wherein after the accelerator card of the first server is repaired to be online again and the distributed storage service on the first server is started up, migrating the first virtual IP address back, and switching the access path of the distributed storage back to the main access path comprises:
after the distributed storage service on the accelerator card of the first server is started, triggering the virtual IP address allocated to the first server to be migrated back to the first server based on a local priority principle;
the virtualization system of the first server triggers the switching of the main access path and the standby access path based on a main access path priority principle;
and after the virtualization system of the first server waits for the completion of the read-write operation of all the volumes under the virtual volume controller bound to the first virtual IP address, the virtualization system hangs the storage interface to the shared storage service on the local accelerator card.
6. An accelerator card distributed storage access device, which is applied to a first server configured with an accelerator card and a common network card, the device comprising:
the virtualization system module runs on a main board CPU of the first server and is used for preferentially accessing the distributed storage through a main access path, wherein the main access path is a path to a virtual volume controller bound with a first virtual IP address on a local accelerator card; when the first server accelerator card has a fault, accessing the distributed storage through a standby access path, wherein the standby access path is an access path from a local common network card to a virtual volume controller which is located on the accelerator card of the second server and is bound with the first virtual IP address
The distributed storage service module runs on a CPU (central processing unit) on an accelerator card of the first server and is used for providing distributed storage service;
the virtualization system module includes:
a storage access module for accessing the distributed storage through the virtual volume controller and the shared storage service bound to the first virtual IP address;
the fault detection module is used for detecting whether the local accelerator card is in fault;
the path switching module is used for switching the access path of the distributed storage from the main access path to the standby access path when the fault of the local accelerator card is detected; and after the fact that the local accelerator card is repaired to be online again and the distributed storage service on the local accelerator card is started is detected, the first virtual IP address is migrated back, and the access path of the distributed storage is switched back to the main access path.
7. The apparatus of claim 6,
the fault detection module establishes a heartbeat detection relationship with a virtual volume controller on a local accelerator card to detect whether the local accelerator card is in fault, and when the local accelerator card is detected to be in fault, the path switching module is triggered to execute switching operation from a main access path to a standby access path;
and after the first virtual IP is migrated to the virtual volume controller on the accelerator card of the second server, the path switching module hangs the storage interface of the storage access module to the second server to bind the shared storage service of the first virtual IP.
8. The apparatus of claim 6, wherein the distributed storage services module comprises:
the heartbeat detection module is used for establishing a heartbeat detection relation with the virtualization system module; the virtual volume controller is also used for establishing a heartbeat detection relation with the virtual IP address which is distributed to other storage nodes in a protection group to detect whether the accelerator cards on the other storage nodes are in fault or not, and the protection group comprises the first virtual IP address and a plurality of virtual IP addresses distributed to the other storage nodes;
and the election receiving module is used for electing the third server based on a preset election strategy when detecting that the accelerator cards on other storage nodes in the protection group are in failure.
9. An electronic device is characterized by comprising a processor, a communication interface, a storage medium and a communication bus, wherein the processor, the communication interface and the storage medium are communicated with each other through the communication bus;
a storage medium for storing a computer program;
a processor for performing the method steps of any one of claims 1-5 when executing a computer program stored on a storage medium.
10. A storage medium on which a computer program is stored which, when being executed by a processor, carries out the method steps of any one of claims 1 to 5.
CN202210660002.8A 2022-06-13 2022-06-13 Acceleration card distributed storage access method, device, equipment and storage medium Pending CN115167757A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210660002.8A CN115167757A (en) 2022-06-13 2022-06-13 Acceleration card distributed storage access method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210660002.8A CN115167757A (en) 2022-06-13 2022-06-13 Acceleration card distributed storage access method, device, equipment and storage medium

Publications (1)

Publication Number Publication Date
CN115167757A true CN115167757A (en) 2022-10-11

Family

ID=83485866

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210660002.8A Pending CN115167757A (en) 2022-06-13 2022-06-13 Acceleration card distributed storage access method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN115167757A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117692322A (en) * 2024-01-30 2024-03-12 苏州元脑智能科技有限公司 Network card configuration method and device, electronic equipment and storage medium
CN117827544A (en) * 2024-02-29 2024-04-05 苏州元脑智能科技有限公司 Hot backup system, method, electronic device and storage medium

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117692322A (en) * 2024-01-30 2024-03-12 苏州元脑智能科技有限公司 Network card configuration method and device, electronic equipment and storage medium
CN117692322B (en) * 2024-01-30 2024-04-23 苏州元脑智能科技有限公司 Network card configuration method and device, electronic equipment and storage medium
CN117827544A (en) * 2024-02-29 2024-04-05 苏州元脑智能科技有限公司 Hot backup system, method, electronic device and storage medium
CN117827544B (en) * 2024-02-29 2024-05-07 苏州元脑智能科技有限公司 Hot backup system, method, electronic device and storage medium

Similar Documents

Publication Publication Date Title
US7725768B1 (en) System and method for handling a storage resource error condition based on priority information
US8464092B1 (en) System and method for monitoring an application or service group within a cluster as a resource of another cluster
US20190163593A1 (en) Storage system and control software deployment method
US9489274B2 (en) System and method for performing efficient failover and virtual machine (VM) migration in virtual desktop infrastructure (VDI)
US8521703B2 (en) Multiple node/virtual input/output (I/O) server (VIOS) failure recovery in clustered partition mobility
KR101471879B1 (en) Hypervisor-based server duplication system, method thereof and recording medium storing the computer program for the duplication
JP4923990B2 (en) Failover method and its computer system.
CN102355369B (en) Virtual clustered system as well as processing method and processing device thereof
CN115167757A (en) Acceleration card distributed storage access method, device, equipment and storage medium
US20110004708A1 (en) Computer apparatus and path management method
US10782898B2 (en) Data storage system, load rebalancing method thereof and access control method thereof
JP5998577B2 (en) Cluster monitoring apparatus, cluster monitoring method, and program
EP3360053B1 (en) Electronic storage system
CN111181780A (en) HA cluster-based host pool switching method, system, terminal and storage medium
CN112199240A (en) Method for switching nodes during node failure and related equipment
CN104170307A (en) Failure switching method, device and system
CN107147516B (en) Server, storage system and related method
US11392423B2 (en) Method for running a quorum-based system by dynamically managing the quorum
CN107977280B (en) Method for verifying ssd cache acceleration effectiveness during failover
CN112799602B (en) Online capacity expansion method for cloud hard disk
WO2023160378A1 (en) Storage device, storage method, computing device, and storage medium
US20240037026A1 (en) Memory pooling, provisioning, and sharing
CN112637336B (en) Method, device and storage medium for realizing high availability of virtual machine
JP5321658B2 (en) Failover method and its computer system.
CN114237961A (en) Hardware fault isolation method and device and server

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination