US20120230340A1 - Message forwarding toward a source end node in a converged network environment - Google Patents
Message forwarding toward a source end node in a converged network environment Download PDFInfo
- Publication number
- US20120230340A1 US20120230340A1 US13/457,113 US201213457113A US2012230340A1 US 20120230340 A1 US20120230340 A1 US 20120230340A1 US 201213457113 A US201213457113 A US 201213457113A US 2012230340 A1 US2012230340 A1 US 2012230340A1
- Authority
- US
- United States
- Prior art keywords
- message
- fcoe
- end node
- source end
- source
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/08—Protocols for interworking; Protocol conversion
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L49/00—Packet switching elements
- H04L49/35—Switches specially adapted for specific applications
- H04L49/356—Switches specially adapted for specific applications for storage area networks
- H04L49/357—Fibre channel switches
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/30—Definitions, standards or architectural aspects of layered protocol stacks
- H04L69/32—Architecture of open systems interconnection [OSI] 7-layer type protocol stacks, e.g. the interfaces between the data link level and the physical level
- H04L69/322—Intralayer communication protocols among peer entities or protocol data unit [PDU] definitions
- H04L69/324—Intralayer communication protocols among peer entities or protocol data unit [PDU] definitions in the data link layer [OSI layer 2], e.g. HDLC
Definitions
- the present invention relates in general to network communication, and in particular, to message forwarding toward a source end node in a converged network environment.
- a current design trend in high performance computing systems is to reduce power consumption, physical space requirements, management and maintenance by reducing the number of physical connections that must be supported by the computing system.
- One technique for reducing the number of physical connections required by a computing system is to converge multiple different types of data traffic on a common physical connection.
- conventional high performance computing systems have communicated with bulk storage (e.g., storage area networks (SAN)) utilizing a first physical connection employing a first communication protocol, such as Fibre Channel, and have additionally communicated with a local or wide area network utilizing a distinct second physical connection employing a second communication protocol, such as Ethernet.
- bulk storage e.g., storage area networks (SAN)
- first communication protocol such as Fibre Channel
- Ethernet a second communication protocol
- FCoE Fibre Channel over Ethernet
- IRCITS International Committee for Information Technology Standards
- Convergence of multiple types of traffic on a physical connection has the advantage of reducing the number of required physical connections while leveraging the prior development of standards and maintaining compatibility with installed infrastructure.
- network convergence can also lead to difficulty in retaining or implementing desired features of the converged data traffic.
- the present disclosure addresses one such difficulty by enabling communication of messages, such as congestion notification messages, back to a source end node in a communication network in which the source end node is obscured by the convergence of the data traffic.
- a network node that forwards traffic of a converged network received from a source end node receives a second message addressed to the network node, but intended for the source end node.
- the second message includes at least a portion of a first message originated by the source end node and previously forwarded by the network node.
- the network node extracts from the first message a source identifier of the source end node in a first communication protocol and determines by reference to a data structure (e.g., a table) a destination address of the second message in a second communication protocol.
- the network node modifies the second message to include the destination address and forwards the second message toward the source end node in accordance with the destination address.
- the first communication protocol is Fibre Channel
- the second communication protocol is Ethernet
- the first message is a Fibre Channel over Ethernet (FCoE) frame
- the second message is a congestion notification message (CNM).
- CCM congestion notification message
- FIG. 1 is a high level block diagram of a converged network environment in accordance with one embodiment
- FIG. 2 is a high level block diagram of an exemplary Fibre Channel over Ethernet (FCoE) forwarder (FCF) in accordance with one embodiment
- FIG. 3 depicts a conventional FCoE frame
- FIG. 4 is a high level logical flowchart of an exemplary process by which a network message is forwarded toward a source end node in a converged network environment in accordance with one embodiment.
- FCoE Fibre Channel over Ethernet
- FCF FCoE Forwarder
- FDF FCoE Data Forwarder
- MAC media access control
- Exemplary converged network environment 100 includes multiple end nodes represented by end nodes E 1 -E 3 , which can serve as sources or sinks of network traffic.
- end nodes E 1 -E 3 may be respective converged network adapters (CNAs) of host platforms 102 a - 102 c , such as client, server or peer computer systems.
- CNAs converged network adapters
- End nodes E 1 and E 2 are each connected by a respective one of physical Ethernet links 104 a , 104 b to an FDF 110 , which as noted above is an Ethernet switch that provides Fibre Channel (FC) support, such as forwarding FC fabric functions (e.g., name server requests and registered state change notifications (RSCNs)), locally enforcing zoning by mapping FC zoning to Ethernet access control lists (ACLs), managing FCoE Initialization Protocol (FIP) keep-alive messaging, and MAC address mapping for FCoE frames.
- FC Fibre Channel
- FDF 110 is in turn connected by physical Ethernet link 104 d to FCF 114 .
- FCF 114 is a FC switching element that is able to forward FCoE frames across one or more FCF-MACs, and that optionally includes one or more lossless Ethernet bridging elements and/or a FC fabric interface.
- FCF 114 does include an Ethernet bridging element 206 (see, e.g., FIG. 2 ) supporting connection via physical Ethernet link 104 f to local area network (LAN) 116 and a FC fabric interface 208 (see, e.g., FIG. 2 ) supporting connection via a physical Fibre Channel link 106 to a storage area network (SAN) 118 .
- LAN local area network
- FC fabric interface 208 see, e.g., FIG. 2
- Exemplary converged network environment 100 additionally includes a FIP snooping bridge (FSB) 112 , which is connected to FCF 114 and to end node E 3 of host platform 102 c by respective physical Ethernet links 104 e and 104 c .
- FSB 112 is an Ethernet bridge that supports Priority Flow Control (PFC) as defined in IEEE standard 802.1Qbb, Enhanced Transmission Selection (ETS) as defined in IEEE standard 802.1Qaz, Data Center Bridging Capabilities Exchange Protocol (DCBX) as defined in IEEE standard 802.1Qaz, and the dynamic ACL functionality described in FC-BB-5 Annex C.
- PFC Priority Flow Control
- ETS Enhanced Transmission Selection
- DCBX Data Center Bridging Capabilities Exchange Protocol
- FC-BB-5 Annex C FC-BB-5 Annex C
- Fibre Channel relies on physical point-to-point FC links between FC nodes.
- the physical point-to-point FC links are replaced by virtual links over physical Ethernet links, such as physical Ethernet links 104 a - 104 e .
- a virtual link 120 a is established over physical Ethernet link 104 a
- virtual link 120 b is established over physical Ethernet link 104 b
- virtual link 120 c is established over physical Ethernet link 104 d
- virtual link 120 d is established over physical Ethernet links 104 b and 104 c . It should be observed that virtual link 120 d spans multiple physical Ethernet links 104 because FCoE traffic is transparent to FSB 112 .
- some virtual or physical links can become congested, for example, because multiple source end nodes may direct traffic to the same destination end node or physical link.
- physical Ethernet link 104 c can become congested by end nodes E 1 and E 2 transmitting traffic destined for end node E 3 at a rate greater than the transmission rate of physical Ethernet link 104 c .
- FSB 112 detects the congestion on physical Ethernet link 10b4c, FSB 112 is unable to notify a source end node (e.g., end node E 1 and/or E 2 ) of the congestion due to its lack of knowledge of the actual source of the frames causing the congestion.
- a source end node e.g., end node E 1 and/or E 2
- a node having knowledge of the converged protocols e.g., FCF 114
- FCF 114 is enhanced to forward messages toward the source end node that originated the traffic.
- FCF 114 can be implemented in special-purpose and/or general-purpose processing hardware.
- the disclosed functionality may be implemented entirely in hardware (e.g., integrated circuitry).
- the special-purpose or general-purpose processing hardware may process firmware and/or software stored in a data storage device/medium to implement the described functionality.
- FCF 114 includes a FC switching element 200 , which is the functional entity that performs Fibre Channel switching among virtual and physical FC switch ports and end node ports (e.g., E_Ports, F_Ports, VE_Ports, and VF_Ports).
- FC switching element 200 has at least one (and in some cases many) associated Ethernet MAC(s) (FCF-MAC) 202 .
- FCF-MAC 202 is coupled with an FCoE controller 204 and is further coupled with a lossless Ethernet bridging element 206 .
- FC switching element 200 is additionally coupled with a FC fabric interface 208 , which provides native E_Port and F_Port connectivity to a FC switch or FC node, respectively.
- FCF 114 forwards FCoE frames addressed to one of its FCF-MACs 202 based on the destination ID (D_ID) of the encapsulated FC frames.
- D_ID destination ID
- FCoE controller 204 supports the instantiation of one or more virtual port/FCoE link end point (LEP) pairs.
- FCF-MAC 202 is a virtual F_port capable FCF-MAC 202
- FCoE controller 204 accordingly instantiates, on an as-needed basis, an FCoE_LEP 210 within an FCoE entity 212 at the FC-2M level and a VF_Port 214 at the FC-2V level 216 (additional mandatory functions of FCoE controller 204 are defined in the FC-BB-5 standard).
- FCoE_LEP 210 performs the encapsulation of FC frames into FCoE frames in transmission and the decapsulation of FCoE frames into FC frames in reception.
- FCoE_LEP 210 operates according to the MAC address of the local link end-point and the MAC address of the remote link end-point.
- FCoE_LEP 210 utilizes the MAC address of the local link end-point (i.e., the FCF-MAC address) as the source address and the MAC address of the remote link end-point as the destination address of the generated FCoE frame.
- FCoE_LEP 210 When decapsulating FC frames from FCoE frames, FCoE_LEP 210 verifies that the destination address of the received FCoE frame is equal to the MAC address of the local link end-point (i.e., the FCF-MAC address) and verifies that the source address of the received FCoE frame is equal to the MAC address of the remote link end-point. If either check fails, FCoE_LEP 210 discards the FCoE frame.
- Each FCoE_LEP 210 is coupled to a routing data structure (e.g., table, list, tree, etc.), such as routing table 220 .
- routing table 220 includes a plurality of table entries each associating identifiers of the ports of FCF 114 (i.e., port IDs) with MAC addresses and FC IDs.
- VF_Port 214 is an instance of the FC-2V sublevel of Fibre Channel that operates as a virtual F_Port. VF_Port 214 receives FC frames from FC switching element 200 and sends them to the proper FCoE_LEP 210 for encapsulation and transmission over the lossless Ethernet network. In addition, VF_Port 214 sends FC frames received from one of its FCoE_LEPs 210 to FC switching element 200 . When receiving FC frames from one of its FCoE_LEPs 210 , VF_Port 214 verifies that the source ID (S_ID) of the received FC frame is equal to the address identifier of the VN_Port associated to that FCoE_LEP 210 , and if not, discards the FC frame.
- S_ID source ID
- FCoE frame 300 begins with a destination MAC address field 302 and source MAC address field 304 utilized to specify the OSI Layer 2 (i.e., Ethernet) destination and source addresses of FCoE frame 300 .
- FCoE frame 300 includes an optional IEEE 802.1Q tag field 306 that can be utilized to specify a virtual LAN (VLAN) to which an Ethernet frame belongs and a version field 310 identifying which version of Ethernet is being employed.
- FCoE frame 300 includes a number of reserved fields 312 - 318 , which can be utilized to implement vendor-specific functionality.
- FCoE frame 300 includes the encapsulated FC frame 322 delimited by start of frame (SOF) and end of frame (EOF) fields 320 and 324 , respectively. Encapsulated FC frame 322 is followed, if necessary, by a reserved field 326 utilized to pad FCoE frame 300 to a desired byte boundary. Finally, FCoE frame 300 includes an Ethernet frame check sequence (FCS) field 328 containing a checksum for error detection and correction.
- FCS Ethernet frame check sequence
- FCoE frame 300 traverses virtual links, such as virtual links 120 a - 120 d of FIG. 1
- destination and source MAC addresses specified by fields 302 - 304 are updated by each network node terminating a virtual link, while the encapsulated FC frame 322 remains unchanged.
- FC IDs and Ethernet MAC addresses set forth in Table I, below.
- FDF 110 , FCF 114 and FSB 112 do not source or sink FCoE data plane traffic (and instead only source and sink FCoE control plane traffic), FDF 110 , FCF 114 and FSB 112 are not assigned FC IDs.
- FCoE frame 300 will exit end node E 1 specifying the source and destination MAC addresses (SMAC and DMAC) and FC source and destination IDs (FC SID and FC DID) set forth in Table II.
- SMAC and DMAC source and destination MAC addresses
- FC SID and FC DID FC source and destination IDs
- FCoE frame 300 Upon exit from FDF 110 , FCoE frame 300 will specify the source and destination MAC addresses set forth in Table III, below.
- FCoE frame 300 The source and destination MAC addresses of FCoE frame 300 are further updated by FCF 114 as indicated in Table IV, below.
- FIG. 4 there is depicted a high level logical flowchart of an exemplary process by which a network message is forwarded toward a source end node in a converged communication network environment in accordance with one embodiment.
- a congestion notification message CCM
- a source end node e.g., end node E 1
- FCoE traffic FCoE traffic
- the illustrated process begins at block 400 and then proceeds to block 402 , which depicts FCF 114 forwarding network traffic including FCoE frames by reference to routing table 220 .
- FCF 114 forwards traffic received from end nodes E 1 and E 2 on virtual link 120 c via the port connected to physical Ethernet link 104 d toward end node E 3 via the port connected to physical Ethernet link 104 e.
- FCF 114 receives a congestion notification message (CNM), for example, a CNM originated by FSB 112 in accordance with the IEEE 802.1Qau standard in response to FSB 112 detecting congestion on physical Ethernet link 104 c . Assuming that congestion is caused by end node E 1 , FSB 112 will detect the congestion in response to a triggering FCoE frame containing the addresses shown below in Table V:
- CNM congestion notification message
- FC SID FC DID FCFMAC FE3 1 3 It should be appreciated that the destination MAC address (DMAC) and the source MAC address (SMAC) are specified in destination and source MAC address fields 302 and 304 of the CNM, and the FC source ID (FC SID) and FC destination ID (FC DID) are specified in the encapsulated FC frame 322 .
- the CNM which FSB 112 directs to the source MAC address of the triggering FCoE frame (i.e., the apparent source of the traffic causing congestion) via physical Ethernet link 104 e , will contain a portion of the triggering FCoE frame and will specify the addresses shown in Table VI.
- FCF 114 determines whether or not the destination MAC of the CNM matches that of FCF 114 (i.e., FCFMAC). If not, the process proceeds to block 416 , which is described below. If, however, the destination MAC of the CNM matches that of FCF 114 , FCF 114 (e.g., FCoE_LEP 210 ) extracts the source FC ID from the triggering frame contained in the CNM (block 408 ) and determines if the source FC ID, which identifies the FC-aware node that is the true source of the triggering frame, is present in routing table 220 (block 410 ). In response to a determination at block 410 that the source FC ID is not present in routing table 220 , FCF 114 drops the CNM (block 412 ), and the process ends at block 420 .
- FCF 114 e.g., FCoE_LEP 210
- FCF 114 determines the destination MAC address associated with the source FC ID by routing table 220 (block 412 ) and updates the destination MAC address of the CNM (block 414 ). Thus, in the present example, FCF 114 produces a CNM having the address shown in Table VII.
- FCF 114 forwards the modified CNM toward the source end node causing the congestion via the physical port associated with the destination MAC address of the CNM (block 416 ), which in the present example is the physical port connected to physical Ethernet link 104 d . Thereafter, the process shown in FIG. 4 terminates at block 420 .
- FDF 110 repeats the process depicted at blocks 406 - 416 of FIG. 4 , extracting the FC SID of 1 from the returned triggering frame within the CNM, consulting its routing tables for the FC ID of 1 to obtain a destination MAC address of FE 1 , and forwarding the CNM via the physical port identified by the routing table as connected physical Ethernet link 104 a .
- the twice modified CNM will specify the addresses indicated in Table VIII, below.
- source end node E 1 the source of the congestion, will act upon the CNM appropriately to reduce congestion, for example, by implementing a backoff for the traffic.
- a network node that forwards traffic of a converged network received from a source end node receives a second message addressed to the network node, but intended for the source end node.
- the second message includes at least a portion of a first message originated by the source end node and previously forwarded by the network node.
- the network node extracts from the first message a source identifier of the source end node in a first communication protocol and determines by reference to a data structure (e.g., a table) a destination address of the second message in a second communication protocol.
- the network node modifies the second message to include the destination address and forwards the second message toward the source end node in accordance with the destination address.
- the first communication protocol is Fibre Channel
- the second communication protocol is Ethernet
- the first message is a Fibre Channel over Ethernet (FCoE) frame
- the second message is a congestion notification message (CNM).
- CCM congestion notification message
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
A network node that forwards traffic of a converged network received from a source end node receives a second message addressed to the network node, but intended for the source end node. The second message includes at least a portion of a first message originated by the source end node and previously forwarded by the network node. The network node extracts from the first message a source identifier of the source end node in a first communication protocol and determines by reference to a data structure a destination address of the second message in a second communication protocol. The network node modifies the second message to include the destination address and forwards the second message toward the source end node in accordance with the destination address.
Description
- The present application is a continuation of U.S. patent application Ser. No. 13/042,776, filed Mar. 8, 2011, entitled “MESSAGE FORWARDING TOWARD A SOURCE END NODE IN A CONVERGED NETWORK ENVIRONMENT,” the disclosure of which is hereby incorporated herein by reference in its entirety for all purposes.
- 1. Technical Field
- The present invention relates in general to network communication, and in particular, to message forwarding toward a source end node in a converged network environment.
- 2. Description of the Related Art
- A current design trend in high performance computing systems is to reduce power consumption, physical space requirements, management and maintenance by reducing the number of physical connections that must be supported by the computing system. One technique for reducing the number of physical connections required by a computing system is to converge multiple different types of data traffic on a common physical connection. For example, conventional high performance computing systems have communicated with bulk storage (e.g., storage area networks (SAN)) utilizing a first physical connection employing a first communication protocol, such as Fibre Channel, and have additionally communicated with a local or wide area network utilizing a distinct second physical connection employing a second communication protocol, such as Ethernet. In this case, a reduction in the number of physical connections can be achieved by converging storage and network traffic on a single physical connection employing Fibre Channel over Ethernet (FCoE), which encapsulates Fibre Channel frames within Ethernet frames. FCoE is defined, for example, by the FC-BB-5 standard promulgated by the T11 Committee of the International Committee for Information Technology Standards (INCITS), which standard is incorporated herein by reference.
- Convergence of multiple types of traffic on a physical connection has the advantage of reducing the number of required physical connections while leveraging the prior development of standards and maintaining compatibility with installed infrastructure. However, network convergence can also lead to difficulty in retaining or implementing desired features of the converged data traffic. The present disclosure addresses one such difficulty by enabling communication of messages, such as congestion notification messages, back to a source end node in a communication network in which the source end node is obscured by the convergence of the data traffic.
- In accordance with at least one embodiment, a network node that forwards traffic of a converged network received from a source end node receives a second message addressed to the network node, but intended for the source end node. The second message includes at least a portion of a first message originated by the source end node and previously forwarded by the network node. The network node extracts from the first message a source identifier of the source end node in a first communication protocol and determines by reference to a data structure (e.g., a table) a destination address of the second message in a second communication protocol. The network node modifies the second message to include the destination address and forwards the second message toward the source end node in accordance with the destination address. In one implementation, the first communication protocol is Fibre Channel, the second communication protocol is Ethernet, the first message is a Fibre Channel over Ethernet (FCoE) frame, and the second message is a congestion notification message (CNM).
-
FIG. 1 is a high level block diagram of a converged network environment in accordance with one embodiment; -
FIG. 2 is a high level block diagram of an exemplary Fibre Channel over Ethernet (FCoE) forwarder (FCF) in accordance with one embodiment; -
FIG. 3 depicts a conventional FCoE frame; and -
FIG. 4 is a high level logical flowchart of an exemplary process by which a network message is forwarded toward a source end node in a converged network environment in accordance with one embodiment. - Disclosed herein are techniques for communicating messages to a source end node of data traffic in a communication network in which the source end node is obscured by the convergence of the data traffic.
- One environment in which the disclosed techniques can be applied is a converged communication network that employs Fibre Channel over Ethernet (FCoE). The FC-BB-5 standard referenced above has defined a FCoE Forwarder (FCF), and the FC-BB-6 draft standard currently under development by the T11 Committee of INCITS has defined a FCoE Data Forwarder (FDF) function, which enables Ethernet switches to enforce Fibre Channel hard zoning and serve the data forwarding role of any FCF. To perform data forwarding, FCFs and FDFs translate source and destination media access control (MAC) addresses specified in Ethernet frames. This MAC address translation prevents implementation of congestion notification mechanisms defined, for example, by IEEE Standard 802.1Qau (incorporated herein by reference), by obscuring the true source end node of data traffic. As described further below, the disclosed techniques enable messages, such as congestion notification messages, to be directed to the source end node of a data traffic flow in the presence of converged data traffic.
- With reference now to the figures and, in particular, with reference to
FIG. 1 , there is illustrated a high level block diagram of exemplaryconverged network environment 100 in accordance with one embodiment. Exemplaryconverged network environment 100 includes multiple end nodes represented by end nodes E1-E3, which can serve as sources or sinks of network traffic. For example, in some implementations, end nodes E1-E3 may be respective converged network adapters (CNAs) of host platforms 102 a-102 c, such as client, server or peer computer systems. - End nodes E1 and E2 are each connected by a respective one of
physical Ethernet links - FDF 110 is in turn connected by physical Ethernet
link 104 d to FCF 114. As defined by the FC-BB-5 standard, FCF 114 is a FC switching element that is able to forward FCoE frames across one or more FCF-MACs, and that optionally includes one or more lossless Ethernet bridging elements and/or a FC fabric interface. In the present example, FCF 114 does include an Ethernet bridging element 206 (see, e.g.,FIG. 2 ) supporting connection via physical Ethernetlink 104 f to local area network (LAN) 116 and a FC fabric interface 208 (see, e.g.,FIG. 2 ) supporting connection via a physical FibreChannel link 106 to a storage area network (SAN) 118. - Exemplary converged
network environment 100 additionally includes a FIP snooping bridge (FSB) 112, which is connected to FCF 114 and to end node E3 ofhost platform 102 c by respective physical Ethernetlinks - As is known in the art, Fibre Channel relies on physical point-to-point FC links between FC nodes. When this arrangement is virtualized in an FCoE network environment, such as that illustrated in
FIG. 1 , the physical point-to-point FC links are replaced by virtual links over physical Ethernet links, such as physical Ethernet links 104 a-104 e. For example, in exemplaryconverged network environment 100, avirtual link 120 a is established over physical Ethernetlink 104 a,virtual link 120 b is established overphysical Ethernet link 104 b,virtual link 120 c is established overphysical Ethernet link 104 d, andvirtual link 120 d is established overphysical Ethernet links virtual link 120 d spans multiple physical Ethernet links 104 because FCoE traffic is transparent to FSB 112. - In converged network environments such as exemplary converged
network environment 100, some virtual or physical links can become congested, for example, because multiple source end nodes may direct traffic to the same destination end node or physical link. For example, inFIG. 1 , physical Ethernetlink 104 c can become congested by end nodes E1 and E2 transmitting traffic destined for end node E3 at a rate greater than the transmission rate of physical Ethernetlink 104 c. As described in greater detail below, if FSB 112 detects the congestion on physical Ethernet link 10b4c, FSB 112 is unable to notify a source end node (e.g., end node E1 and/or E2) of the congestion due to its lack of knowledge of the actual source of the frames causing the congestion. To support the transmission of messages such as congestion notification messages to the source end node of traffic on a converged network, such as that depicted inFIG. 1 , a node having knowledge of the converged protocols (e.g., FCF 114) is enhanced to forward messages toward the source end node that originated the traffic. - Referring now to
FIG. 2 , there is depicted a more detailed functional model of an exemplary FCF 114 in accordance with one embodiment. In various implementations, FCF 114 can be implemented in special-purpose and/or general-purpose processing hardware. In some cases, the disclosed functionality may be implemented entirely in hardware (e.g., integrated circuitry). In other cases, the special-purpose or general-purpose processing hardware may process firmware and/or software stored in a data storage device/medium to implement the described functionality. - As depicted, FCF 114 includes a
FC switching element 200, which is the functional entity that performs Fibre Channel switching among virtual and physical FC switch ports and end node ports (e.g., E_Ports, F_Ports, VE_Ports, and VF_Ports).FC switching element 200 has at least one (and in some cases many) associated Ethernet MAC(s) (FCF-MAC) 202. FCF-MAC 202 is coupled with anFCoE controller 204 and is further coupled with a losslessEthernet bridging element 206.FC switching element 200 is additionally coupled with aFC fabric interface 208, which provides native E_Port and F_Port connectivity to a FC switch or FC node, respectively. FCF 114 forwards FCoE frames addressed to one of its FCF-MACs 202 based on the destination ID (D_ID) of the encapsulated FC frames. - FCoE
controller 204 supports the instantiation of one or more virtual port/FCoE link end point (LEP) pairs. In the present example, FCF-MAC 202 is a virtual F_port capable FCF-MAC 202, and FCoEcontroller 204 accordingly instantiates, on an as-needed basis, an FCoE_LEP 210 within an FCoEentity 212 at the FC-2M level and a VF_Port 214 at the FC-2V level 216 (additional mandatory functions of FCoEcontroller 204 are defined in the FC-BB-5 standard). - FCoE_LEP 210 performs the encapsulation of FC frames into FCoE frames in transmission and the decapsulation of FCoE frames into FC frames in reception. FCoE_LEP 210 operates according to the MAC address of the local link end-point and the MAC address of the remote link end-point. Thus, when encapsulating FC frames into FCoE frames, FCoE_LEP 210 utilizes the MAC address of the local link end-point (i.e., the FCF-MAC address) as the source address and the MAC address of the remote link end-point as the destination address of the generated FCoE frame. When decapsulating FC frames from FCoE frames,
FCoE_LEP 210 verifies that the destination address of the received FCoE frame is equal to the MAC address of the local link end-point (i.e., the FCF-MAC address) and verifies that the source address of the received FCoE frame is equal to the MAC address of the remote link end-point. If either check fails,FCoE_LEP 210 discards the FCoE frame. - Each
FCoE_LEP 210 is coupled to a routing data structure (e.g., table, list, tree, etc.), such as routing table 220. As shown, routing table 220 includes a plurality of table entries each associating identifiers of the ports of FCF 114 (i.e., port IDs) with MAC addresses and FC IDs. -
VF_Port 214 is an instance of the FC-2V sublevel of Fibre Channel that operates as a virtual F_Port.VF_Port 214 receives FC frames fromFC switching element 200 and sends them to theproper FCoE_LEP 210 for encapsulation and transmission over the lossless Ethernet network. In addition,VF_Port 214 sends FC frames received from one of itsFCoE_LEPs 210 toFC switching element 200. When receiving FC frames from one of itsFCoE_LEPs 210,VF_Port 214 verifies that the source ID (S_ID) of the received FC frame is equal to the address identifier of the VN_Port associated to thatFCoE_LEP 210, and if not, discards the FC frame. - With reference now to
FIG. 3 , the format of aconventional FCoE frame 300 is illustrated. As shown,FCoE frame 300 begins with a destinationMAC address field 302 and sourceMAC address field 304 utilized to specify the OSI Layer 2 (i.e., Ethernet) destination and source addresses ofFCoE frame 300. Next,FCoE frame 300 includes an optional IEEE 802.1Q tag field 306 that can be utilized to specify a virtual LAN (VLAN) to which an Ethernet frame belongs and aversion field 310 identifying which version of Ethernet is being employed. Thereafter,FCoE frame 300 includes a number of reserved fields 312-318, which can be utilized to implement vendor-specific functionality. Following reserved fields 312-318,FCoE frame 300 includes the encapsulatedFC frame 322 delimited by start of frame (SOF) and end of frame (EOF) fields 320 and 324, respectively. EncapsulatedFC frame 322 is followed, if necessary, by areserved field 326 utilized to padFCoE frame 300 to a desired byte boundary. Finally,FCoE frame 300 includes an Ethernet frame check sequence (FCS)field 328 containing a checksum for error detection and correction. - As noted above, when a
conventional FCoE frame 300 traverses virtual links, such as virtual links 120 a-120 d ofFIG. 1 , destination and source MAC addresses specified by fields 302-304 are updated by each network node terminating a virtual link, while the encapsulatedFC frame 322 remains unchanged. For example, consider an exemplary scenario in which the components of exemplary convergednetwork environment 100 are assigned the FC IDs and Ethernet MAC addresses set forth in Table I, below. It should be noted that becauseFDF 110,FCF 114 andFSB 112 do not source or sink FCoE data plane traffic (and instead only source and sink FCoE control plane traffic),FDF 110,FCF 114 andFSB 112 are not assigned FC IDs. -
TABLE I FCID MAC E1 1 FE1 E2 2 FE2 E3 3 FE3 FDF NA FDFMAC FCF NA FCFMAC FSB NA FSBMAC - Assuming the transmission of an
FCoE frame 300 from end node E1 to end node E3, theFCoE frame 300 will exit end node E1 specifying the source and destination MAC addresses (SMAC and DMAC) and FC source and destination IDs (FC SID and FC DID) set forth in Table II. -
TABLE II SMAC DMAC FC SID FC DID FE1 FDFMAC 1 3
Upon exit fromFDF 110,FCoE frame 300 will specify the source and destination MAC addresses set forth in Table III, below. -
TABLE III SMAC DMAC FC SID FC DID FDFMAC FCFMAC 1 3
The source and destination MAC addresses ofFCoE frame 300 are further updated byFCF 114 as indicated in Table IV, below. -
TABLE IV SMAC DMAC FC SID FC DID FCFMAC FE3 1 3
The updating of sourceMAC address field 304 in this manner complicates communication by downstream nodes of messages, such as congestion notification messages, back toward end node E1 because the source MAC address of end node E1 is obscured. - Referring now to
FIG. 4 , there is depicted a high level logical flowchart of an exemplary process by which a network message is forwarded toward a source end node in a converged communication network environment in accordance with one embodiment. For clarity, the depicted process is described with reference to forwarding of a congestion notification message (CNM) toward a source end node (e.g., end node E1) causing congestion in a converged communication network carrying FCoE traffic. - The illustrated process begins at
block 400 and then proceeds to block 402, which depictsFCF 114 forwarding network traffic including FCoE frames by reference to routing table 220. For example, as depicted inFIG. 1 ,FCF 114 forwards traffic received from end nodes E1 and E2 onvirtual link 120 c via the port connected tophysical Ethernet link 104 d toward end node E3 via the port connected tophysical Ethernet link 104 e. - At
block 404,FCF 114 receives a congestion notification message (CNM), for example, a CNM originated byFSB 112 in accordance with the IEEE 802.1Qau standard in response toFSB 112 detecting congestion onphysical Ethernet link 104 c. Assuming that congestion is caused by end node E1,FSB 112 will detect the congestion in response to a triggering FCoE frame containing the addresses shown below in Table V: -
TABLE V SMAC DMAC FC SID FC DID FCFMAC FE3 1 3
It should be appreciated that the destination MAC address (DMAC) and the source MAC address (SMAC) are specified in destination and source MAC address fields 302 and 304 of the CNM, and the FC source ID (FC SID) and FC destination ID (FC DID) are specified in the encapsulatedFC frame 322. - In accordance with the IEEE 802.1Qau standard, the CNM, which
FSB 112 directs to the source MAC address of the triggering FCoE frame (i.e., the apparent source of the traffic causing congestion) viaphysical Ethernet link 104 e, will contain a portion of the triggering FCoE frame and will specify the addresses shown in Table VI. -
TABLE VI SMAC DMAC FC SID FC DID FSBMAC FCFMAC 1 3 - At
block 406,FCF 114 determines whether or not the destination MAC of the CNM matches that of FCF 114 (i.e., FCFMAC). If not, the process proceeds to block 416, which is described below. If, however, the destination MAC of the CNM matches that ofFCF 114, FCF 114 (e.g., FCoE_LEP 210) extracts the source FC ID from the triggering frame contained in the CNM (block 408) and determines if the source FC ID, which identifies the FC-aware node that is the true source of the triggering frame, is present in routing table 220 (block 410). In response to a determination atblock 410 that the source FC ID is not present in routing table 220,FCF 114 drops the CNM (block 412), and the process ends atblock 420. - Returning to block 410, in response to determining that the source FC ID of the triggering frame is present in forwarding table 220,
FCF 114 determines the destination MAC address associated with the source FC ID by routing table 220 (block 412) and updates the destination MAC address of the CNM (block 414). Thus, in the present example,FCF 114 produces a CNM having the address shown in Table VII. -
TABLE VII SMAC DMAC FC SID FC DID FCFMAC FDFMAC 1 3
Thereafter,FCF 114 forwards the modified CNM toward the source end node causing the congestion via the physical port associated with the destination MAC address of the CNM (block 416), which in the present example is the physical port connected tophysical Ethernet link 104 d. Thereafter, the process shown inFIG. 4 terminates atblock 420. - Thereafter, in response to receiving the CNM,
FDF 110 repeats the process depicted at blocks 406-416 ofFIG. 4 , extracting the FC SID of 1 from the returned triggering frame within the CNM, consulting its routing tables for the FC ID of 1 to obtain a destination MAC address of FE1, and forwarding the CNM via the physical port identified by the routing table as connectedphysical Ethernet link 104 a. Thus, the twice modified CNM will specify the addresses indicated in Table VIII, below. -
TABLE VIII SMAC DMAC FC SID FC DID FDFMAC FE1 1 3 - In response to receipt of the CNM, source end node E1, the source of the congestion, will act upon the CNM appropriately to reduce congestion, for example, by implementing a backoff for the traffic.
- As has been described, in some embodiments, a network node that forwards traffic of a converged network received from a source end node receives a second message addressed to the network node, but intended for the source end node. The second message includes at least a portion of a first message originated by the source end node and previously forwarded by the network node. The network node extracts from the first message a source identifier of the source end node in a first communication protocol and determines by reference to a data structure (e.g., a table) a destination address of the second message in a second communication protocol. The network node modifies the second message to include the destination address and forwards the second message toward the source end node in accordance with the destination address. In one implementation, the first communication protocol is Fibre Channel, the second communication protocol is Ethernet, the first message is a Fibre Channel over Ethernet (FCoE) frame, and the second message is a congestion notification message (CNM).
- While the present invention has been particularly shown as described with reference to one or more preferred embodiments, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention. For example, although aspects have been described with respect one or more machines executing program code (e.g., software, firmware or a combination thereof) that direct the functions described herein, it should be understood that embodiments may alternatively be implemented as a program product including a tangible machine-readable storage medium or storage device (e.g., an optical storage medium, memory storage medium, disk storage medium, etc.) storing program code that can be processed by a machine to cause the machine to perform one or more of the described functions. Further, although the present invention has been described with reference to the communication of congestion notification messages back to a source endpoint in a FCoE environment, it should be appreciated that the illustrated processes are more generally applicable to the communication of other messages to source endpoints in the presence of converged network traffic.
Claims (6)
1. A method of data processing in a converged network, the method comprising:
at a network node that forwards traffic of the converged network received from a source end node, receiving a second message addressed to the network node, but intended for the source end node, the second message including at least a portion of a first message originated by the source end node and previously forwarded by the network node;
the network node extracting from the first message a source identifier of the source end node in a first communication protocol;
the network node determining by reference to a data structure a destination address of the second message in a second communication protocol; and
the network node modifying the second message to include the destination address and forwarding the second message toward the source end node in accordance with the destination address.
2. The method of claim 1 , wherein:
the first communication protocol is Fibre Channel;
the second communication protocol is Ethernet.
3. The method of claim 2 , wherein the network node is one of a set including a Fibre Channel over Ethernet (FCoE) forwarder (FCF) and a FCoE Data Forwarder (FDF).
4. The method of claim 2 , wherein:
the first message is a Fibre Channel over Ethernet (FCoE) frame; and
the second message is a congestion notification message (CNM).
5. The method of claim 2 , wherein the receiving comprises receiving the second message from a Fibre Channel over Ethernet (FCoE) Initiation Protocol (FIP) snooping bridge.
6. The method of claim 1 , and further comprising:
the network node dropping the second message in response to failing to find the source identifier of the source end node in the data structure.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/457,113 US20120230340A1 (en) | 2011-03-08 | 2012-04-26 | Message forwarding toward a source end node in a converged network environment |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/042,776 US8687649B2 (en) | 2011-03-08 | 2011-03-08 | Message forwarding toward a source end node in a converged network environment |
US13/457,113 US20120230340A1 (en) | 2011-03-08 | 2012-04-26 | Message forwarding toward a source end node in a converged network environment |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/042,776 Continuation US8687649B2 (en) | 2011-03-08 | 2011-03-08 | Message forwarding toward a source end node in a converged network environment |
Publications (1)
Publication Number | Publication Date |
---|---|
US20120230340A1 true US20120230340A1 (en) | 2012-09-13 |
Family
ID=46795539
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/042,776 Active 2031-11-13 US8687649B2 (en) | 2011-03-08 | 2011-03-08 | Message forwarding toward a source end node in a converged network environment |
US13/457,113 Abandoned US20120230340A1 (en) | 2011-03-08 | 2012-04-26 | Message forwarding toward a source end node in a converged network environment |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/042,776 Active 2031-11-13 US8687649B2 (en) | 2011-03-08 | 2011-03-08 | Message forwarding toward a source end node in a converged network environment |
Country Status (6)
Country | Link |
---|---|
US (2) | US8687649B2 (en) |
CN (1) | CN103392324B (en) |
DE (1) | DE112012001141B4 (en) |
GB (1) | GB2502743B (en) |
TW (1) | TWI535252B (en) |
WO (1) | WO2012120407A1 (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140098672A1 (en) * | 2012-10-10 | 2014-04-10 | Huawei Technologies Co., Ltd. | Method for processing congestion and network device |
US20160065387A1 (en) * | 2014-06-04 | 2016-03-03 | Dell Products L.P. | Fibre channel gateway system |
US9426546B2 (en) | 2013-10-21 | 2016-08-23 | Global Foundries Inc. | Maintaining a fabric name across a distributed switch |
US10205676B2 (en) * | 2014-08-08 | 2019-02-12 | Zte Corporation | RSCN method and system, related device and computer storage medium |
Families Citing this family (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP6007595B2 (en) * | 2012-05-28 | 2016-10-12 | 富士通株式会社 | Transmission method, apparatus and program |
US8804708B1 (en) * | 2012-12-21 | 2014-08-12 | Juniper Networks, Inc. | Methods and apparatus for implementing access control at a network switch |
US10367681B2 (en) | 2013-03-15 | 2019-07-30 | Hewlett Packard Enterprise Development Lp | Maintenance of data forwarder connection state information |
CN104980359A (en) * | 2014-04-04 | 2015-10-14 | 中兴通讯股份有限公司 | Flow control method of fiber channel over Ethernet (FCoE), flow control device of FCoE and flow control system of FCoE |
CN115174139B (en) * | 2022-05-26 | 2024-04-09 | 华为技术有限公司 | Node isolation method, switching equipment and Ethernet storage system |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5920699A (en) * | 1996-11-07 | 1999-07-06 | Hewlett-Packard Company | Broadcast isolation and level 3 network switch |
US20080068990A1 (en) * | 2005-07-22 | 2008-03-20 | Huawei Technologies Co., Ltd. | Method and apparatus for implementing multicast service |
US20080075089A1 (en) * | 2006-09-26 | 2008-03-27 | Cisco Technology, Inc. | Snooping of on-path ip reservation protocols for layer 2 nodes |
US20090274153A1 (en) * | 2002-10-01 | 2009-11-05 | Andrew Tai-Chin Kuo | System and method for implementation of layer 2 redundancy protocols across multiple networks |
US20120063316A1 (en) * | 2010-09-10 | 2012-03-15 | Brocade Communications Systems, Inc. | Congestion notification across multiple layer-2 domains |
US20130007839A1 (en) * | 2001-09-28 | 2013-01-03 | Juniper Networks, Inc. | Routing a packet by a device |
Family Cites Families (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5459723A (en) | 1994-03-08 | 1995-10-17 | Advanced Micro Devices | Packet management device for fast-packet network |
GB9411357D0 (en) | 1994-06-07 | 1994-07-27 | Newbridge Networks Corp | Data transmission system |
US7573876B2 (en) | 2002-12-05 | 2009-08-11 | Intel Corporation | Interconnecting network processors with heterogeneous fabrics |
US7733770B2 (en) | 2004-11-15 | 2010-06-08 | Intel Corporation | Congestion control in a network |
US8121038B2 (en) | 2007-08-21 | 2012-02-21 | Cisco Technology, Inc. | Backward congestion notification |
CN102017542B (en) | 2008-05-08 | 2014-04-30 | 惠普开发有限公司 | A method for interfacing a fibre channel network with an Ethernet based network |
US7944812B2 (en) | 2008-10-20 | 2011-05-17 | International Business Machines Corporation | Redundant intermediary switch solution for detecting and managing fibre channel over ethernet FCoE switch failures |
CN101815040A (en) | 2009-02-23 | 2010-08-25 | 国际商业机器公司 | Method for forwarding packet in hybrid switch network and switch |
WO2010105092A1 (en) | 2009-03-12 | 2010-09-16 | James Paul Rivers | Providing fibre channel services and forwarding fibre channel over ethernet frames |
CN101631076B (en) | 2009-08-21 | 2011-06-08 | 杭州华三通信技术有限公司 | Message transmission control method and fiber channel over Ethernet protocol system |
-
2011
- 2011-03-08 US US13/042,776 patent/US8687649B2/en active Active
-
2012
- 2012-02-28 DE DE112012001141.4T patent/DE112012001141B4/en active Active
- 2012-02-28 CN CN201280009497.2A patent/CN103392324B/en active Active
- 2012-02-28 GB GB1316176.5A patent/GB2502743B/en active Active
- 2012-02-28 WO PCT/IB2012/050922 patent/WO2012120407A1/en active Application Filing
- 2012-03-05 TW TW101107296A patent/TWI535252B/en not_active IP Right Cessation
- 2012-04-26 US US13/457,113 patent/US20120230340A1/en not_active Abandoned
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5920699A (en) * | 1996-11-07 | 1999-07-06 | Hewlett-Packard Company | Broadcast isolation and level 3 network switch |
US20130007839A1 (en) * | 2001-09-28 | 2013-01-03 | Juniper Networks, Inc. | Routing a packet by a device |
US20090274153A1 (en) * | 2002-10-01 | 2009-11-05 | Andrew Tai-Chin Kuo | System and method for implementation of layer 2 redundancy protocols across multiple networks |
US20080068990A1 (en) * | 2005-07-22 | 2008-03-20 | Huawei Technologies Co., Ltd. | Method and apparatus for implementing multicast service |
US20080075089A1 (en) * | 2006-09-26 | 2008-03-27 | Cisco Technology, Inc. | Snooping of on-path ip reservation protocols for layer 2 nodes |
US20120063316A1 (en) * | 2010-09-10 | 2012-03-15 | Brocade Communications Systems, Inc. | Congestion notification across multiple layer-2 domains |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140098672A1 (en) * | 2012-10-10 | 2014-04-10 | Huawei Technologies Co., Ltd. | Method for processing congestion and network device |
US9426546B2 (en) | 2013-10-21 | 2016-08-23 | Global Foundries Inc. | Maintaining a fabric name across a distributed switch |
US9432754B2 (en) | 2013-10-21 | 2016-08-30 | Globalfoundries Inc. | Maintaining a fabric name across a distributed switch |
US20160065387A1 (en) * | 2014-06-04 | 2016-03-03 | Dell Products L.P. | Fibre channel gateway system |
US9935874B2 (en) * | 2014-06-04 | 2018-04-03 | Dell Products L.P. | Fibre channel gateway system |
US10205676B2 (en) * | 2014-08-08 | 2019-02-12 | Zte Corporation | RSCN method and system, related device and computer storage medium |
Also Published As
Publication number | Publication date |
---|---|
WO2012120407A1 (en) | 2012-09-13 |
US8687649B2 (en) | 2014-04-01 |
TWI535252B (en) | 2016-05-21 |
DE112012001141B4 (en) | 2019-12-19 |
GB2502743A (en) | 2013-12-04 |
CN103392324A (en) | 2013-11-13 |
DE112012001141T5 (en) | 2014-01-09 |
GB2502743B (en) | 2018-06-27 |
GB201316176D0 (en) | 2013-10-23 |
US20120230334A1 (en) | 2012-09-13 |
TW201251395A (en) | 2012-12-16 |
CN103392324B (en) | 2016-01-20 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8687649B2 (en) | Message forwarding toward a source end node in a converged network environment | |
US8885641B2 (en) | Efficient trill forwarding | |
US9515922B2 (en) | Distributed fibre channel forwarder | |
US9742693B2 (en) | Dynamic service insertion in a fabric switch | |
US8369347B2 (en) | Fiber channel over Ethernet and fiber channel switching based on Ethernet switch fabrics | |
US9178969B2 (en) | Methods, systems and apparatus for the servicing of fibre channel login frames | |
US10454760B2 (en) | Layer-3 overlay gateways | |
US9614759B2 (en) | Systems and methods for providing anycast MAC addressing in an information handling system | |
US9071629B2 (en) | Methods for the interconnection of fibre channel over ethernet devices using shortest path bridging | |
JP5893644B2 (en) | Method, system, and apparatus for interconnecting Fiber Channel over Ethernet devices | |
US8908704B2 (en) | Switch with dual-function management port | |
US8848575B2 (en) | High availability and multipathing for fibre channel over ethernet | |
US20080159277A1 (en) | Ethernet over fibre channel | |
US9414136B2 (en) | Methods and apparatus to route fibre channel frames using reduced forwarding state on an FCoE-to-FC gateway | |
WO2017113306A1 (en) | Method for sending virtual extensible local area network packet, computer device, and readable medium | |
US8351442B1 (en) | Method and system for network communication | |
US20080181243A1 (en) | Ethernet forwarding in high performance fabrics | |
US9504016B2 (en) | Optimized multicast routing in a Clos-like network | |
US20110044344A1 (en) | A method for interfacing a fibre channel network with an ethernet based network | |
US20090141734A1 (en) | Method and system for a converged infiniband over ethernet network | |
US9407547B2 (en) | Fibre channel over ethernet (FCoE) over virtual port channel (vPC) | |
US20140301402A1 (en) | FCoE VN_Port Virtualizer |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |