WO2020212997A1

WO2020212997A1 - Generating and utilizing topology information for an ethernet ring to support network management

Info

Publication number: WO2020212997A1
Application number: PCT/IN2019/050310
Authority: WO
Inventors: Mudarakola KARTHEEK; Satheesh Kumar KARRA
Original assignee: Telefonaktiebolaget Lm Ericsson (Publ)
Priority date: 2019-04-16
Filing date: 2019-04-16
Publication date: 2020-10-22

Abstract

A method is described for performing a management function using topology information. The method includes determining, by a first interconnecting node shared by a closed and a sub-ring of a network, topology information for the closed and sub-rings, wherein the closed ring includes a first set of nodes, including the first interconnecting node, and the sub-ring includes a second set of nodes, including the first interconnecting node, and the topology information describes links between the first set of nodes and links between the second set of nodes, wherein the topology information is based on topology information packets that indicate identifiers of (1) an originating node of a corresponding topology information packet and/or (2) a neighbor to the originating node. The method further includes generating and transmitting, by the first interconnecting node in the closed ring, a set of management packets based on the topology information for performing the management function.

Description

GENERATING AND UTILIZING TOPOLOGY INFORMATION FOR AN ETHERNET RING TO SUPPORT NETWORK MANAGEMENT

TECHNICAL FIELD

[0001] Embodiments of the invention relate to the field of ethemet rings; and more specifically, to generating and utilizing topology information for an ethernet ring to support network management.

BACKGROUND ART

[0002] A ring network topology is a network topology in which each node connects to exactly two other nodes via corresponding dedicated links, forming a single continuous pathway for signals through each node. Data travels from node-to-node, with each node along the way handling and possibly forwarding the packets.

[0003] Although ring topologies may provide certain advantages, including a reduced number of links between nodes in comparison to other network topologies, information about configuration of ring networks is limited. For example, ring networks may employ loop avoidance techniques to avoid packets from continually circulating in these networks. In some cases, loop avoidance in a ring network is achieved by guaranteeing that, at any time, traffic may flow on all but one of the links in the ring network. This particular link may be termed the ring protection link and, under normal conditions, this ring protection link is blocked (i.e., not used for servicing traffic and, in particular, not used for servicing inbound traffic to a corresponding node). When another link in the ring network fails, the ring protection link needs to be reactivated such that only one link (i.e., the failed link) remains inactive.

[0004] Accordingly, the activity/operation of links and connections between nodes in a ring network may alter over time. However, this topology information is not visible to administrators/users of the ring network such that the administrators/users can make network management decisions.

SUMMARY

[0005] A method for performing a management function using layer 2 topology information in a layer 2 network system is described. The method includes determining, by a first interconnecting node shared by a closed ring and a sub-ring of the layer 2 network system, topology information for the closed ring and the sub-ring, wherein the closed ring includes a first set of nodes, including the first interconnecting node, and the sub-ring includes a second set of nodes, including the first interconnecting node, and the topology information describes links between the first set of nodes and links between the second set of nodes, wherein the topology information is determined based on topology information packets that indicate one or more of (1) identifiers of an originating node of a corresponding topology information packet and (2) identifiers of neighbor nodes to the originating node; generating, by the first interconnecting node, a set of management packets based on the topology information for performing the management function in the layer 2 network system; and transmitting, by the first interconnecting node, the set of management packets in the closed ring of the layer 2 network system to perform the management function.

[0006] A non-transitory machine -readable storage medium is described that provides instructions that, if executed by a processor of an interconnecting node, which is shared by a closed ring and a sub-ring of a layer 2 network system, will cause said processor to perform operations. The operations include determining topology information for the closed ring and the sub-ring, wherein the closed ring includes a first set of nodes, including the first interconnecting node, and the sub-ring includes a second set of nodes, including the first interconnecting node, and the topology information describes links between the first set of nodes and links between the second set of nodes, wherein the topology information is determined based on topology information packets that indicate one or more of (1) identifiers of an originating node of a corresponding topology information packet and (2) identifiers of neighbor nodes to the originating node; generating a set of management packets based on the topology information for performing the management function in the layer 2 network system; and transmitting the set of management packets in the closed ring of the layer 2 network system to perform the management function.

[0007] A node is described for performing a management function using topology information in a layer 2 network, wherein the node is a first interconnecting node that is shared by a closed ring and a sub-ring of the layer 2 network. The node includes a memory unit that stores instructions and a processor coupled to the memory unit to execute the instructions, wherein the instructions cause the system to: determine topology information for the closed ring and the sub-ring, wherein the closed ring includes a first set of nodes, including the first interconnecting node, and the sub-ring includes a second set of nodes, including the first interconnecting node, and the topology information describes links between the first set of nodes and links between the second set of nodes, wherein the topology information is determined based on topology information packets that indicate one or more of (1) identifiers of an originating node of a corresponding topology information packet and (2) identifiers of neighbor nodes to the originating node; generate a set of management packets based on the topology information for performing the management function in the layer 2 network; and transmit the set of management packets in the closed ring of the layer 2 network to perform the management function.

[0008] As described herein, using a new set of control packets (e.g., a set of topology information packets, including introduction topology information packets and neighbor topology information packets), topology information can be derived for major/closed rings and/or sub-rings in a network system. In particular, the set of topology information packets facilitate the generation of a topology table describing the connections between nodes in major/closed rings and/or sub-rings in a network system. This topology information may thereafter be used for performing/guiding one or more network/ring management functions (e.g., an intelligent media access control (MAC) flush operation).

BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The invention may best be understood by referring to the following description and accompanying drawings that are used to illustrate embodiments of the invention. In the drawings:

[0010] Figure 1A shows a network system, including an Ethernet ring, according to one example embodiment.

[0011] Figure IB shows the network system with a blocked ring protection link in the Ethernet ring, according to one example embodiment.

[0012] Figure 1C shows the network system with a link/port failure in the Ethernet ring, according to one example embodiment.

[0013] Figure 2 shows a network system with a major/closed ring and a sub-ring, according to one example embodiment.

[0014] Figure 3 shows a method for determining topology information in relation to a major/closed ring and/or a sub-ring, according to one example embodiment.

[0015] Figure 4 shows an example of an introduction topology information packet, according to one example embodiment.

[0016] Figure 5 shows an example of a neighbor topology information packet, according to one example embodiment. [0017] Figure 6 shows a neighbor table, according to one example embodiment.

[0018] Figure 7 shows a topology table, according to one example embodiment.

[0019] Figure 8 shows a neighbor table for a node populated after receipt of an introduction topology information packet from another node, according to one example embodiment.

[0020] Figure 9 shows a topology table for a node populated after receipt of an introduction topology information packet from another node, according to one example embodiment.

[0021] Figure 10 shows the neighbor table for the node populated after receipt of another introduction topology information packet from another node, according to one example embodiment.

[0022] Figure 11 shows the topology table for the node populated after receipt of an introduction topology information packet from another node, according to one example embodiment.

[0023] Figure 12 shows a method for determining topology information in relation to a major/closed ring and/or a sub-ring, according to one example embodiment.

[0024] Figure 13 shows an example of the topology table that is updated based on a neighbor topology information packet from another node, according to one example embodiment.

[0025] Figure 14 shows an example of the topology table that is updated based on a neighbor topology information packet from another node, according to one example embodiment.

[0026] Figure 15 shows an example of the topology table that is updated based on a neighbor topology information packet from another node, according to one example embodiment.

[0027] Figure 16 shows a method for determining topology information in relation to a major/closed ring and/or a sub-ring, according to one example embodiment.

[0028] Figure 17 shows a method for performing an efficient media access control (MAC) flush in a network system with a major/closed ring and a sub-ring, according to one example embodiment.

[0029] Figure 18 shows a network system with a major/closed ring and a sub-ring, according to one example embodiment. [0030] Figure 19 shows transmission of signal fail (SF) packets in the sub-ring of the network system, according to one example embodiment.

[0031] Figure 20 shows transmission of topology change notification (TCN) packets in the major/closed ring of the network system, according to one example embodiment.

[0032] Figure 21 shows a method for performing a management function using layer 2 topology information in a layer 2 network system, according to one example embodiment.

[0033] Figure 22A illustrates connectivity between network devices (NDs) within an exemplary network, as well as three exemplary implementations of the NDs, according to some embodiments of the invention.

[0034] Figure 22B illustrates an exemplary way to implement a special-purpose network device according to some embodiments of the invention.

[0035] Figure 22C illustrates various exemplary ways in which virtual network elements (VNEs) may be coupled according to some embodiments of the invention.

[0036] Figure 22D illustrates a network with a single network element (NE) on each of the NDs, and within this straight forward approach contrasts a traditional distributed approach (commonly used by traditional routers) with a centralized approach for maintaining reachability and forwarding information (also called network control), according to some embodiments of the invention.

[0037] Figure 22E illustrates the simple case of where each of the NDs implements a single NE, but a centralized control plane has abstracted multiple of the NEs in different NDs into (to represent) a single NE in one of the virtual network(s), according to some embodiments of the invention.

[0038] Figure 22F illustrates a case where multiple VNEs are implemented on different NDs and are coupled to each other, and where a centralized control plane has abstracted these multiple VNEs such that they appear as a single VNE within one of the virtual networks, according to some embodiments of the invention.

[0039] Figure 23 illustrates a general purpose control plane device with centralized control plane (CCP), according to some embodiments of the invention.

DETAILED DESCRIPTION

[0040] The following description describes methods and apparatus for generating and utilizing topology information for an ethemet ring to support network management. In the following description, numerous specific details such as logic implementations, opcodes, means to specify operands, resource partitioning/sharing/duplication implementations, types and interrelationships of system components, and logic partitioning/integration choices are set forth in order to provide a more thorough understanding of the present invention. It will be appreciated, however, by one skilled in the art that the invention may be practiced without such specific details. In other instances, control structures, gate level circuits and full software instruction sequences have not been shown in detail in order not to obscure the invention. Those of ordinary skill in the art, with the included descriptions, will be able to implement appropriate functionality without undue experimentation.

[0041] References in the specification to “one embodiment,” “an embodiment,” “an example embodiment,” etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.

[0042] Bracketed text and blocks with dashed borders (e.g., large dashes, small dashes, dot-dash, and dots) may be used herein to illustrate optional operations that add additional features to embodiments of the invention. However, such notation should not be taken to mean that these are the only options or optional operations, and/or that blocks with solid borders are not optional in certain embodiments of the invention.

[0043] In the following description and claims, the terms“coupled” and“connected,” along with their derivatives, may be used. It should be understood that these terms are not intended as synonyms for each other. “Coupled” is used to indicate that two or more elements, which may or may not be in direct physical or electrical contact with each other, co-operate or interact with each other. “Connected” is used to indicate the establishment of communication between two or more elements that are coupled with each other.

[0044] Figure 1A shows a network system 100, according to one example embodiment. As shown in Figure 1A, the network system 100 includes a set of nodes 102A-102D that are each communicatively coupled via respective links 104₁-104₄ to a pair of other nodes 102 in the network system 100. In particular, each node 102A-102D includes a set of ports (i.e., each node 102 includes a port 0 and a port 1) for coupling each node 102 with a pair of other nodes 102 via the links 104. For example, the node 102A is coupled to (1) the node 102B via port 0 and link 104i and (2) the node 102C via port 1 and link 104₂. Similarly, the node 102B is coupled to (1) the node 102 A via port 0 and link 104i and (2) the node 102D via port 1 and link 104₃. Moreover, the node 102C is coupled to (1) the node 102A via port 0 and link 104₂ and (2) the node 102D via port 1 and link 104₄. Lastly, the node 102D is coupled to (1) the node 102B via port 0 and link 104₃ and (2) the node 102C via port 1 and link 104₄.

[0045] Each of the nodes 102A-102D may be a network component in the network system 100. For example, each of the nodes 102A-102D may be a network switch that connects host devices together in the network system 100 by using packet switching to receive, process, and forward data to a destination host device. In particular, a set of host devices can be communicatively coupled to each of the nodes 102A-102D via a set of corresponding links and ports. When acting as a switch or a similar device, the nodes 102A-102D can establish a set of media access control (MAC) forwarding tables based on their discovery of the reachability of various host devices in the network system 100 through corresponding nodes 102. Using these MAC forwarding tables, the nodes 102A-102D receive, process, and forward data/packets to a destination host device.

[0046] As shown in Figure 1A and described above in relation to the links 104 between the nodes 102A-102D, the nodes 102A-102D form a ring 108, which may be termed a major ring 108 or a closed ring 108 as each node 102 in the ring 108 is connected to exactly two other nodes 102 in the ring 108. The ring 108 can provide wide-area multipoint connectivity more economically due to the reduced number of links 104 that are required for connecting the nodes 102 in comparison to network systems that rely on more than two links to couple corresponding nodes in these network systems. In particular, as described above, each node 102A-102D (sometimes referred to as an ethemet ring node 102 or a ring node 102) in the ring 108 is connected to adjacent nodes 102 participating in the ring 108, using two independent links 104 (sometimes referred to as ring links 104) and a corresponding set of ports (sometimes referred to as ring ports) of these nodes 102.

[0047] Although the ring 108 may provide certain advantages, including a reduced number of links 104 in comparison to other network system configurations, the ring 108 is susceptible to loops that may cause packets to continually circulate in the network system 100. Accordingly, the network system 100 may employ a ring protection switching architecture to avoid loops. In some embodiments, an Ethernet Ring Protection Switching (ERPS) instance 106A-106D may be instantiated/established on each of the nodes 102 A- 102D, respectively. The ERPS instances 106A-106D may be used for (1) establishing the ring 108, including control message passing to establish the links 104 and (2) performing loop avoidance routines in the ring 108.

[0048] In one embodiment, loop avoidance in the ring 108 is achieved by guaranteeing that, at any time, traffic may flow on all but one of the links 104 in the ring 108. This particular link 104 may be termed the ring protection link 104 and, under normal conditions, this ring protection link 104 is blocked (i.e., not used for servicing traffic and, in particular, not used for servicing inbound traffic to a corresponding node 102). For example, a ring protection link owner node 102 controls the ring protection link 104, including blocking traffic that is received via a corresponding port of the ring protection link 104. For instance, Figure IB shows the network system 100 from Figure 1A, in which an administrator of the network system 100 has set the link 104i as the ring protection link 104 and the node 102A as the ring protection link owner. Accordingly, the node 102 A (in particular, the ERPS instance 106 A) may block traffic on port 0 of the node 102 A, which corresponds to the ring protection link 104i .

[0049] Although the ring protection link 104 is normally blocked to avoid a loop in the ring 108, under certain conditions the ring protection link 104 can be unblocked by the ring protection link owner node 102 (in particular, the ERPS instance 106). For example, in response to a failure to another link 104 or port in the ring 108, the ring protection link owner node 102 may unblock the ring protection link 104 since the failed link 104 is effectively blocked and will thus result in the avoidance of a loop in the ring 108. Using the example above in which the node 102 A is designated as the ring protection link owner node 102 and the link 104i is designated as the ring protection link 104, port 1 of the node 102C may fail, which causes the link 104₄ to consequently fail (i.e., become blocked). In response to detection of this failure, the nodes 104C and 104D, which rely on the link 104₄, may transmit a set of packets in the ring 108 (e.g., a signal failure (SF) packet or topology change notification (TCN) packet according to an automatic protection switching (APS) protocol). In response to these set of packets (e.g., in response to a SF packet from the node 102C), the ring protection link owner node 102 A unblocks port 0 such that the ring protection link 104i is no longer blocked, as shown in Figure 1C. When port 1 of the node 102C has recovered and is consequently no longer in a failing state, the ring protection link owner node 102 A can again block port 0 such that the ring protection link 1041 is blocked and thus avoid a loop in the ring 108. [0050] Although the above techniques assist in avoiding loops in rings quickly, there is not enough topology information that can be used by the customers/users to identify if there are any failures in the network system 100. In particular, each node 102 must be examined to determine port status information (e.g., failure of ports). For example, as described above, if there is a single failure in the topology of the network system 100, a port of the ring protection link 104 is unblocked to ensure less traffic loss. However, the user/administrator does not have sufficient information to easily determine which port is blocked and which port is unblocked. In another example, if multiple ports and/or links 104 fail, it is difficult to immediately identify which ports and links 104 are blocked.

[0051] In addition to lack of information on port and link 104 status, there is insufficient information on how nodes 102 are connected in the network system 100 such that additional functions can be intelligently performed. For instance, in a software defined network (SDN) scenario, topology information at the controller level can be useful to perform various management functions. Although some protocols can be used to obtain node 102 connection information, this does not help in getting topology information at virtual local area network (VLAN)/broadcast domain level.

[0052] A solution is proposed for the above-mentioned problems by defining a new set of control packets, which may be referred to as a set of topology information packets. As will be described in additional detail below, a first type of topology information packet may be transmitted by each node 102 to the immediate/direct neighbor nodes 102 of the transmitting/originating node 102 in the ring 108. These immediate/direct neighbor nodes 102 are connected by a single link 104. The first topology information packet may include an identifier of the transmitting node 102 and may be referred to as an introduction topology information packet that can be transmitted to one or more neighbor nodes 102 of the transmitting/originating node 102 via corresponding ports.

[0053] Introduction topology information packets may be consumed by receiving nodes 102 and be used to construct a second type of topology information packet, which may be referred to as a neighbor topology information packet. The neighbor topology information packet includes identifiers of neighbor nodes 102 corresponding to the transmitting/originating node 102 in addition to an identifier of the transmitting/originating node 102. In contrast to the introduction topology information packet, the neighbor topology information packet is forwarded to each neighbor of the receiving node 102 until the neighbor topology information packet reaches the originating node 102 or is otherwise blocked (e.g., blocked based on a blocked ring protection link 104).

[0054] The nodes 102 use the introduction topology information packets and the neighbor topology information packets to populate neighbor and/or a topology tables kept by each node 102, which define the topology of the ring 108 (e.g., the ports and corresponding links 104 that couple the nodes 102 in the ring 108 together along with identification of a ring protection link 104 and corresponding blocked port). Once the topology of the ring 108 has converged (i.e., the neighbor table and/or the topology table is completed for each node 102 in the ring 108), topology information describing the ring 108 can be obtained from any node 102 in the ring 108. This topology information, as described by the topology tables, can be used for performing various management functions for the network system 100, including the ring 108 (e.g., design and debugging operations for the ring 108). For example, as will be described in greater detail below, media access control (MAC) flush operations can be performed on a subset of nodes 102 in the ring 108 instead of for all nodes 102 in the ring 108 such that recovery from a fault can be realized with minimal overhead.

[0055] Although described above in relation to a major/closed ring (e.g., the ring 108), the techniques described herein may be similarly applied to sub-rings, which are defined by a pair of nodes 102 that are each connected to only one other node 102. For example, Figure 2 shows a network system 200 with the major/closed ring 108 and the sub-ring 202. As shown, the sub-ring 202 is defined by the nodes 102C-102F. In particular, in the sub-ring 202, the node 102C is coupled to the node 102E via port 0 and link 104_s . Similarly, the node 102D is coupled to the node 102F via port 0 and link 104₆. Moreover, the node 102E is coupled to (1) the node 102C via port 0 and link 104s and (2) the node 102F via port 1 and link 104g. Lastly, the node 102F is coupled to (1) the node 102D via port 0 and link 104 , and (2) the node 102E via port 1 and link 104₇. Although the node 102C and the node 102D are connected via the link 104₄, this link 104₄ is part of the major/closed ring 108 and is not part of the sub-ring 202. Accordingly, the sub-ring 202 forms an open structure, while the major/closed ring 108 forms a closed structure. As noted above and as will be described in greater detail below, the techniques described herein for determining and utilizing topology information may be utilized for both major/closed rings (e.g., the ring 108) and sub-rings (e.g., the ring 202) using corresponding ERPS instances 106A-106F.

[0056] Turning now to Figure 3, a method 300 will be described for determining topology information in relation to a major/closed ring (e.g., the ring 108) and/or a sub-ring (e.g., the ring 202), according to one example embodiment. The operations in the flow diagrams will be described with reference to the exemplary embodiments of the other figures. However, it should be understood that the operations of the flow diagrams can be performed by embodiments of the invention other than those discussed with reference to the other figures, and the embodiments of the invention discussed with reference to these other figures can perform operations different than those discussed with reference to the flow diagrams.

[0057] Each of the operations of the method 300 may be performed by one or more nodes 102 in a network system (e.g., the network system 100 or the network system 200). For example, the method 300 may be separately performed in entirely or partially overlapping time periods by each of the ERPS instances 106 in each of the nodes 102. The method 300 will be described in relation to the major/closed ring 108 and, in particular, from the point- of-view of the node 102 A but can be similarly applied to the sub-ring 202.

[0058] The method 300 may commence at operation 302 with the node 102A (i.e., a first node 102) receiving an introduction topology information packet from another node 102 (i.e., a second node 102) in the ring 108 via a port of the node 102A (i.e., port 0 or port 1). As described above, an introduction topology information packet is a type of topology information packet and includes an identifier of the transmitting/originating node 102. In contrast, a neighbor topology information packet, which is also a type of topology information packet, includes the identifier of the transmitting/originating node 102 in addition to the identifiers of the immediately adjacent/neighboring nodes 102 to the transmitting/originating node 102. In one embodiment, the topology information packets (e.g., introduction topology information packets and neighbor topology information packets) may be ring automatic protection switching (R-APS) packets. For instance, Figure 4 shows an example of an introduction topology information packet 400 and Figure 5 shows an example of a neighbor topology information packet 500, according to one example embodiment. As shown in Figure 4, the introduction topology information packet 400 includes several fields, including a request/state identifier 402 (four bits), a sub-code 404 (four bits), a status identifier 406 (eight bits/one octet), a node identifier 408A-408B (six octets) that corresponds to the transmitting/originating node 102, and a set of reserved fields 410A-410F (twenty-four octets). As shown in Figure 5, the neighbor topology information packet 500 may include several fields, including a request/state identifier 402 (four bits), a sub-code 404 (four bits), a status identifier 406 (eight bits/one octet), a node identifier 408A-408B (six octets) that corresponds to the transmitting/originating node 102, a port 0 node identifier 502A-502B that corresponds to a direct neighbor node 102 relative to the transmitting/originating node 102 that is reachable via port 0 of the transmitting/originating node 102 (six octets), a port 1 node identifier 504A-504B that corresponds to a direct neighbor node 102 relative to the transmitting/originating node 102 that is reachable via port 1 of the transmitting/originating node 102 (six octets), and a set of reserved fields 506A- 506C (twelve-octets).

[0059] In the example embodiments shown in Figure 4 and Figure 5, the request/status identifier 402 and the sub-code 404 may be selected to uniquely represent the topology information packets (i.e., the introduction topology information packet 400 and the neighbor topology information packet 500) relative to other R-APS packets and themselves. For example, while the value“1101” may be used for the request/state identifier 402 to represent a forced switch R-APS packet, the value“1110” may be used for the request/state identifier 402 to represent an event packet, the value“1011” may be used for the request/state identifier 402 to represent a signal-fail (SF) R-APS packet, the value“0111” may be used for the request/state identifier 402 to represent a manual switch (MS) R-APS packet, and the value“0000” may be used for the request/state identifier 402 to represent a no request (NR) R-APS packet, the value“1100” may be used for the request/state identifier 402 to represent topology information packets (e.g., the introduction topology information packet 400 and a neighbor topology information packet 500). To distinguish between the introduction topology information packet 400 and a neighbor topology information packet 500, a value for the sub-code 404 may be assigned to each of these packets 400 and 500. For example, the value “0000” may be used for the sub-code 404 for the introduction topology information packet 400 and the value“0001” may be used for the sub-code 404 for the neighbor topology information packet 500. Although particular values have been provided for the request/state identifier 402 and the sub-code 404 to distinguish topology information packets (i.e., the introduction topology information packet 400 and the neighbor topology information packet 500) relative to other R-APS packets and themselves, other values may be used in other embodiments.

[0060] Upon receipt by the node 102 A of an introduction topology information packet 400 from another node 102, the method 300 may move to operation 304. At operation 304, the node 102A (i.e., the first node 102) determines what port (i.e., a port number) the introduction topology information packet 400 was received on. In particular, the node 102A determines whether the introduction topology information packet 400 from operation 302 was received on port 0 (corresponding to link 1040 or port 1 (corresponding to link 104₂) of the node 102A.

[0061] At operation 306, the node 102A determines an identifier of the node 102 (i.e., the second node 102) that transmitted the introduction topology information packet 400 from operation 302 based on the introduction topology information packet 400. For example, the identifier of the node 102 that transmitted the introduction topology information packet 400 may be determined by examining the node identifier 408A-408B of the introduction topology information packet 400.

[0062] At operation 308, the node 102A (i.e., the first node) updates one or more of a neighbor table and a topology table. The neighbor table indicates the direct neighbor nodes 102 corresponding to the node 102 A and their reachability on corresponding ports of the node 102A, while the topology table indicates the reachability of all nodes 102 in the ring 108 in relation to each port of all nodes 102. For example, Figure 6 shows a neighbor table 600 while Figure 7 shows a topology table 700, according to one example embodiment. In some embodiments, each of the nodes 102 stores (or has access to) and updates (or directs updates to) its own neighbor table 600 and topology table 700.

[0063] As shown in Figure 6, the neighbor table 600 includes a port 0 node identifier 602 and a port 1 node identifier 604. Upon receipt of an introduction topology information packet 400 from each of the neighbor nodes 102 of the node 102A (e.g., the nodes 102B and 102C), the node 102A populates the port 0 node identifier 602 and the port 1 node identifier 604, respectively. For example, Figure 8 shows the neighbor table 600 for the node 102A populated after receipt of an introduction topology information packet 400 from the node 102B. As shown, the identifier of the node 102B (i.e.,“B”) is added to the port 0 node identifier 602 for the neighbor table 600, since the introduction topology information packet 400 was received on port 0 of the node 102 A and the node 102B is a neighbor to the node 102A via port 0 of the node 102A.

[0064] In the above examples, the identifiers of the nodes 102A and 102B are assumed to be the values“A” and“B”, respectively; however, the nodes 102A and 102B may have more extensive identifiers (e.g., identifiers represented by six octets/forty-eight bits). This convention of using shorter identifiers will be used for later examples as well for sake of simplicity.

[0065] As shown in Figure 7, the topology table 700 includes a node identifier 702, a port 0 node identifier 704, and a port 1 node identifier 706. Further, the topology table 700 may include entries 708I-708_N corresponding to the number of nodes 102 in the ring 108. In the example ring 108, the number of nodes 102A-102D is equal to four. Thus, the topology table 700 will grow to four entries 708 (i.e., N = 4), where each entry 708 corresponds to a separate node 102 in the ring 108. In particular, as the node 102 A becomes aware of a new node 102 in the ring via topology information packets, the node 102A will add a new entry 708 to the topology table 700.

[0066] For each entry 708 in the topology table 700, the node 102 A populates with corresponding values for the node identifier 702, the port 0 node identifier 704, and the port 1 node identifier 706. For example, based on an introduction topology information packet 400 received from the neighbor node 102B, which is represented in the neighbor table 600 shown in Figure 8, the node 102 A can populate the topology table 700, as shown in Figure 9. Namely, the node 102 A can populate the port 0 node identifier 704 for the entry 708 corresponding to topology information for the node 102A in a similar fashion as the neighbor table 600 for the node 102A. Since an introduction topology information packet 400 has not yet been received from a node on port 1 of the node 102 A, port 1 node identifier 706 for the entry 708 cannot yet be populated. However, entry 708₂ can be established with a corresponding node identifier for the node 102B since this node’s existence was referenced in the introduction topology information packet 400 received from the neighbor node 102B. Nevertheless, the neighboring nodes 102 to the node 102B are not yet known such that the port 0 node identifier 704 and the port 1 node identifier 706 for entry 708₂ cannot yet be populated.

[0067] At operation 310, the node 102A determines if topology for the ring 108 for the node 102A has converged. In one embodiment, determining whether topology for the ring 108 for the node 102A has converged is based on the completion of the topology table 700. In particular, the node 102 A determines that the topology for the ring 108 has converged when the topology table 700 is complete (i.e., all fields for all identified nodes 102 have been populated). Upon determining that the topology for the ring 108 has converged for the node 102A, the method 300 moves to operation 312.

[0068] At operation 312, the node 102A (i.e., the first node 102) transmits an introduction topology information packet 400 on each/both ports of the node 102A. Providing the introduction topology information packet 400 on both ports will cause both neighbor nodes 102 of the node 102 A to provide corresponding introduction topology information packets 400 in the hope that these introduction topology information packets 400 will allow the topology of the ring 108 to converge for the other nodes 102.

[0069] Returning to operation 310, upon determining that the topology for the ring 108 has not converged for the node 102A, the method 300 moves to operation 314. At operation 314, the node 102A (i.e., the first node 102) transmits an introduction topology information packet 400 on the port from which the current introduction topology information packet 400 from operation 302 was received. Providing the introduction topology information packet 400 on this receiving port supports a corresponding node 102 to complete the corresponding topology table 700.

[0070] The method 300 may continually be performed on receipt of new introduction topology information packets 400 such that the neighbor table 600 and/or the topology table 700 can be completed (i.e., the topology of the ring 108 can converge for all nodes 102). For example, the node 102A may also receive an introduction topology information packet 400 from the node 102C via port 1 of the node 102A. This causes the node 102A to perform the method 300, including populating the neighbor table 600 and the topology table 700. Namely, (1) the neighbor table 600 may be populated by adding an identifier of the node 102C to the port 1 node identifier 604 (i.e.,“C”), as shown in Figure 10, and (2) the topology table 700 may be populated by adding an identifier of the node 102C to the port 1 node identifier 706 (i.e.,“C”) for the entry 708i and adding a new entry 7O83 for the node 102C, as shown in Figure 11. However, the topology table 700 of the node 102 A will not be able to converge/complete with only introduction topology information packets 400, as these packets can only be received from neighboring nodes 102 of the node 102 A and the node 102 A has received introduction topology information packets 400 from each neighboring node 102 (i.e., the nodes 102B and 102C). Accordingly, additional packets must be utilized to allow the topology of the ring 108 to converge for the node 102A and other nodes 102. Further, completion of the topology table 700 can be supported by one or more other methods, as will be described below.

[0071] Turning now to Figure 12, a method 1200 will be described for determining topology information in relation to a major/closed ring (e.g., the ring 108) and/or a sub-ring (e.g., the ring 202), according to one example embodiment. Each of the operations of the method 1200 may be performed by one or more nodes 102 in a network system (e.g., the network system 100 or the network system 200). For example, the method 1200 may be separately performed in entirely or partially overlapping time periods by each of the ERPS instances 106 in each of the nodes 102. The method 1200 will be described in relation to the major/closed ring 108 and, in particular, from the point-of-view of the node 102A but can also be performed in relation to the sub-ring 202.

[0072] The method 1200 may commence at operation 1202 with the node 102A (i.e., a first node 102) receiving a neighbor topology information packet 500 from another node 102 (i.e., a second node 102) in the ring 108 via a first port of the node 102A (i.e., port 0 or port 1). As described above, the neighbor topology information packet 500 may include the fields/data shown in Figure 5. In particular, the neighbor topology information packet 500 may include (1) an identifier of the transmitting/originating node 102 in the node identifier 408A-408B, (2) an identifier of a neighbor node 102 of the transmitting/originating node 102 that is directly reachable via a single link 104 on port 0 in the port 0 neighbor node identifier 502A-502B, and (3) an identifier of a neighbor node 102 of the transmitting/originating node 102 that is directly reachable via a single link 104 on port 1 in the port 1 neighbor node identifier 504A-504B. For example, the node 102A may receive a neighbor topology information packet 500 from the node 102D. In this case, because the node 102D is not directly coupled to the node 102 A, the neighbor topology information packet 500 from the node 102D is received via the node 102B (from port 0 of the node 102 A) or the node 102C (from port 1 of the node 102A). The neighbor topology information packet 500 from the node 102D includes (1) an address of the node 102D in the node identifier 408A-408B (i.e.,“D”), (2) an address of the node 102B in the port 0 node identifier 502A-502B (i.e.,“B”), as the node 102B is a direct neighbor to the node 102D via port 0 of the node 102D, and (3) an address of the node 102C in the port 1 node identifier 504A-504B (i.e.,“C”), as the node 102C is a direct neighbor to the node 102D via port 1 of the node 102D.

[0073] At operation 1204, the node 102A forwards the neighbor topology information packet 500 received at operation 502 out of a second port of the node 102A such that it can be received by another node 102 (i.e., a third node 102). In particular, if the neighbor topology information packet 500 was received on port 0 of the node 102A, the neighbor topology information packet 500 is forwarded at operation 1204 on port 1 of the node 102A. Conversely, if the neighbor topology information packet 500 was received on port 1 of the node 102A, the neighbor topology information packet 500 is forwarded at operation 1204 on port 0 of the node 102A. Forwarding the neighbor topology information packet 500 on the opposite port allows the neighbor topology information packet 500 to be circulated in the ring 108 and, consequently, allows other nodes 102 to utilize information in the neighbor topology information packet 500 from the node 102D to complete corresponding topology tables 700. The neighbor topology information packet 500 will continue to be forwarded in the ring 108 until the packet 500 is received by the transmitting/originating node 102 (e.g., the node 102D).

[0074] At operation 1206, the node 102A determines a node identifier of the transmitting/originating node 102 (i.e., the second node) and a set of neighbor node identifiers from the neighbor topology information packet 500. In particular, as described above, the neighbor topology information packet 500 from the node 102D includes (1) an address of the node 102D in the node identifier 408A-408B (i.e.,“D”), (2) an address of the node 102B in the port 0 node identifier 502A-502B (i.e.,“B”), as the node 102B is a direct neighbor to the node 102D via port 0 of the node 102D, and (3) an address of the node 102C in the port 1 node identifier 504A-504B (i.e.,“C”), as the node 102C is a direct neighbor to the node 102D via port 1 of the node 102D. Accordingly, in this example, the identifier“D” is determined at operation 1206 as the node identifier of the transmitting/originating node 102 and the identifiers“B” and“C” are determined at operation 1206 as the set of neighbor node identifiers.

[0075] At operation 1208, the node 102A updates the topology table 700 based on the node identifier of the transmitting/originating node 102 and a set of neighbor node identifiers. In particular, the node 102A examines the topology table 700 and determines if there is an existing entry 708 for the node identifier of the transmitting/originating node 102. If there is, the node 102A completes the port 0 node identifier 704 and the port 1 node identifier 706 based on the determined set of neighbor node identifiers. If there is not an existing entry 708 for the node identifier of the transmitting/originating node 102, the node 102 A adds an entry 708 and completes the port 0 node identifier 704 and the port 1 node identifier 706 for this new entry 708 based on the determined set of neighbor node identifiers. Further, if any of the identifiers in the set of neighbor node identifiers is not represented by an entry 708 in the topology table 700, an entry 708 is added. Figure 13 shows an example of the topology table 700 from Figure 9 that is updated based on a neighbor topology information packet 500 from the node 102D. As shown, an entry 7O83 is added to the topology table 700 with the node identifier 702 having the value“D”, which corresponds to the node identifier of the transmitting/originating node 102D for the received neighbor topology information packet 500. Further, the node 102A can populate (1) the port 0 node identifier 704 with the value“B” for the entry 7O8₃ based on the received neighbor topology information packet 500 and (2) the port 1 node identifier 706 with the value“C” for the entry 7O8₃ based on the received neighbor topology information packet 500. Moreover, since there is not an entry 708 in the topology table 700 corresponding to the node 102C, the node 102 A can added an entry 7O8₄ to the topology table 700 for the node 102C, which was referenced in the neighbor topology information packet 500. Although the topology tables 700 in Figures 11 and 13 were derived from the topology table 700 of Figure 9, the topology tables 700 in Figures 11 and 13 include different orders for entries 708 (e.g., the node 102C is represented by the entry 7O8₃ in Figure 11 while the node 102C represented by the entry 7O8₄ in Figure 13) since these tables 700 represent the node 102A receiving information packets (e.g., neighbor topology information packets 500) from nodes 102 in different orders. Thus, while each of the topology tables 700 in Figures 11 and 13 will represent consistent information, the order of entries 708 is different. In other examples that are not shown here, the entries 708 could be in a different order based on the order information packets are received by the node 102A.

[0076] At operation 1210, the node 102 A determines if topology for the ring 108 has converged for the node 102A. As noted above, in one embodiment, determining whether topology for the ring 108 has converged for the node 102 A is based on the completion of the topology table 700. In particular, the node 102A determines that the topology for the ring 108 has converged when the topology table 700 is complete for the node 102A (i.e., all fields for all identified nodes 102 have been populated). Upon determining that the topology for the ring 108 has converged for the node 102A, the method 1200 moves to operation 1212.

[0077] At operation 1212, the node 102A transmits a neighbor topology information packet 500 on each/both ports of the node 102A. In this case, the neighbor topology information packet 500 is populated with values corresponding to the node 102A. Providing the neighbor topology information packet 500 on both ports will cause both neighbor nodes 102 of the node 102A to provide corresponding neighbor topology information packets 500 in the hope that these neighbor topology information packets 500 will allow the topology of the ring 108 to converge for other nodes 102B-102D.

[0078] Returning to operation 1210, upon determining that the topology for the ring 108 has not converged for the node 102A, the method 1200 moves to operation 1214. At operation 1214, the node 102A transmits a neighbor topology information packet 500 on the port from which the current neighbor topology information packet 500 from operation 1202 was received. In this case, the neighbor topology information packet 500 is populated with values corresponding to the node 102A. Providing the neighbor topology information packet 500 on this receiving port supports a corresponding node 102 to complete their corresponding topology table 700.

[0079] The method 1200 may continually be performed on receipt of new neighbor topology information packets 500 such that the topology table 700 can be completed (i.e., the topology of the ring 108 can converge for the node 102 A and other nodes 102). For example, the node 102A may also receive a neighbor topology information packet 500 from the node 102C via port 1 of the node 102A. This causes the node 102A to perform the method 1200, including populating the topology table 700. Namely, the topology table 700 may be populated by adding an identifier of the node 102 A to the port 0 node identifier 704 (i.e.,“A”) for the entry 708₄ and an identifier of the node 102D to the port 1 node identifier 706 (i.e.,“D”) for the entry 7O8₄, as shown in Figure 14. Additionally, the topology table 700 may be populated by adding an identifier of the node 102C to the port 1 node identifier 706 (i.e.,“C”) for the entry 708i, as shown in Figure 15, since the neighbor topology information packet 500 was received from node 102C on port 1. Further, the node 102A may also receive a neighbor topology information packet 500 from the node 102B via port 0 of the node 102A. This causes the node 102A to perform the method 1200, including populating the topology table 700. Namely, the topology table 700 may be populated by adding an identifier of the node 102A to the port 0 node identifier 704 (i.e.,“A”) for the entry 7O82 and an identifier of the node 102D to the port 1 node identifier 706 (i.e.,“D”) for the entry 708₂ as shown in Figure 15. Accordingly, as shown in Figure 15, the topology table 700 is completed and, consequently, the topology for the ring 108 is considered to have converged for the node 102A. The topology for the ring 108 may have converged for the other nodes 102B-102D or the methods 300 and 1200 may be performed for these nodes 102B-102D to assist in convergence. Although the topology table 700 is completed using the methods 300 and 1200, the completion of the topology table 700 can be supported by one or more additional methods, as will be described further below.

[0080] In some embodiments, additional information may be added to the topology table 700. For example, an indication of the ring protection link owner node 102, an indication of the ring protection link 104 and corresponding port, and/or an indication of links 104 or ports that are disabled (e.g., the ring protection link 104) or otherwise not functioning (e.g., links 104 that have failed). In one embodiment, an indication of the ring protection link 104 may be populated in the topology table 700 based on a No Request - Ring Protection Link (RPL) Blocked (NR-RB) message that is sent by the ring protection link owner node 102. This NR-RB message contains the identifier of the ring protection link 104 owner node and port number of the blocked port. In one embodiment, the indication of non functioning/failed links may be populated in the topology table 700 based on signal-fail packets that are sent by the corresponding failing node 102.

[0081] Although described above in relation to the major/closed ring 108, updating the topology table 700 can be performed for a sub-ring (e.g., the sub-ring 202), as well using similar techniques. In particular, since a sub-ring includes two nodes 102 that are only connected to a single neighbor node 102, these nodes 102 will not have information for both the port 0 node identifier 502A-502B and the port 1 node identifier 504A-504B of the neighbor topology information packet 500, the port 0 node identifier 602 and the port 1 node identifier 604 of the neighbor table 600, or the port 0 node identifier 704 and the port 1 node identifier 706 of the topology table 700. Accordingly, in such a case, the identifier of the node 102 is used in the place of the missing information. For example, in the case of the sub-ring 202 with the interconnecting nodes 102C and 102D, when a corresponding neighbor node 102 is not present, these nodes 102C and 102D will use their own identifiers (i.e.,“C” and“D”, respectively) for the port 0 node identifier 502A-502B and the port 1 node identifier 504A-504B of the neighbor topology information packet 500, the port 0 node identifier 602 and the port 1 node identifier 604 of the neighbor table 600, or the port 0 node identifier 704 and the port 1 node identifier 706 of the topology table 700. Using the identifier of the interconnecting nodes 102C and 102D in these fields will identify these nodes as interconnecting nodes 102 and, consequently, the presence of a sub-ring in the network system 200.

[0082] Turning now to Figure 16, a method 1600 will be described for determining topology information in relation to a major/closed ring (e.g., the ring 108) and/or a sub-ring (e.g., the ring 202), according to one example embodiment. Each of the operations of the method 1600 may be performed by one or more nodes 102 in a network system (e.g., the network system 100 or the network system 200). For example, the method 1600 may be separately performed in entirely or partially overlapping time periods by each of the ERPS instances 106 in each of the nodes 102. The method 1600 will be described in relation to the major/closed ring 108 and in particular from the point-of-view of the node 102A. [0083] The method 1600 may commence at operation 1602 with the completion of configuration of the ring 108. This configuration completion of the ring 108 may include one or more of (1) instantiating the ERPS instances 106 on each node 102, (2) establishing all links 104 between nodes 102 in the ring 108, and (3) establishing/configuring a ring protection link 104 and a ring protection link owner node 102 in the ring 108.

[0084] At operation 1604, the node 102A (i.e., a first node 102) transmits an introduction topology information packet 400 on each/both ports of the node 102A. Providing the introduction topology information packet 400 on both ports will cause both neighbor nodes 102 of the node 102 A to provide corresponding introduction topology information packets in the hope that these introduction topology information packets 400 will allow the topology of the ring 108 to converge for the node 102A and the other nodes 102B-102D.

[0085] At operation 1606, the node 102A determines if topology for the ring 108 has converged for the node 102A. In one embodiment, determining whether topology for the ring 108 has converged for the node 102 A is based on the completion of the topology table 700. In particular, the node 102A determines that the topology for the ring 102A has converged for the node 102A when the topology table 700 is complete (i.e., all fields for all identified nodes 102 have been populated). Upon determining that the topology for the ring 108 has converged for the node 102 A, the method 1600 moves to operation 1608.

[0086] At operation 1608, the node 102A stops transmitting all topology information packets (e.g., introduction topology information packets 400 and neighbor topology information packets 500).

[0087] Returning to operation 1606, upon determining that the topology for the ring 108 has not converged for the node 102A, the method 1600 moves to operation 1610. At operation 1610, the node 102A determines if topology information packets 500 have been received on both ports of the node 102A that identify the node 102A (i.e., include an identifier of the node 102A). In particular, the node 102A determines if a neighbor topology information packet 500 was received on port 0 and port 1 that includes an identifier of the node 102A (e.g.,“A”) as the value for the port 0 neighbor node identifier 502A-502B or the port 1 neighbor node identifier 504A-504B. In response to determining at operation 1610 that topology information packets have not been received on both ports of the node 102A that identify the node 102 A, the method 1600 moves to operation 1612.

[0088] At operation 1612, similar to operation 1604, the node 102A transmits an introduction topology information packet 400 on each/both ports of the node 102A. Providing the introduction topology information packet 400 on both ports will cause both neighbor nodes 102 of the node 102A to provide corresponding introduction topology information packets 400 in the hope that these introduction topology information packets 400 will allow the topology of the ring 108 to converge for the node 102A.

[0089] Returning to operation 1610, in response to determining that topology information packets 500 have been received on both ports of the node 102A that identify the node 102A, the method 1600 moves to operation 1614. At operation 1614, the node 102A determines if introduction topology information packets 400 are currently being transmitted by the node 102A. In particular, while the node 102A may have begun transmitting an introduction topology information packet 400 (e.g., begun generating the introduction topology information packet 400), transmission of this packet 400 may have not been completed. Accordingly, the node 102 A determines at operation 1614 if introduction topology information packets 400 have begun to transmit but have not yet transmitted.

[0090] In response to determining at operation 1614 that introduction topology information packets 400 are currently being transmitted by the node 102A, the method 1600 moves to operation 1616. At operation 1616, the node 102A stops transmission of introduction topology information packets 400. In particular, since neighbor topology information packets 500 have been received on both ports of the node 102A (as determined at operation 1610), transmission of additional introduction topology information packets 400 is not necessary to evoke additional neighbor topology information packets 500 from neighbor nodes 102.

[0091] Following operation 1612, following operation 1616, or upon determining at operation 1614 that introduction topology information packets 400 are not currently being transmitted by the node 102 A, the method 1600 moves to operation 1618. At operation 1618, the node 102A transmits a neighbor topology information packet 500 on each/both ports of the node 102A. In this case, the neighbor topology information packet 500 is populated with values corresponding to the node 102A. Providing the neighbor topology information packet 500 on both ports will cause both neighbor nodes 102 of the node 102A to provide corresponding neighbor topology information packets 500 in the hope that these neighbor topology information packets 500 will allow the topology of the ring 108 to converge for the node 102A. [0092] At operation 1620, the node 102A waits/pauses for a predefined period of time (e.g., one millisecond, one second, etc.) before returning to operation 1606 to determine if the topology of the ring 108 has converged for the node 102A.

[0093] As described above, each of the methods 300, 1200, and 1600 may be performed simultaneously or during overlapping time periods by each of the nodes 102 in the ring 108 and/or the sub-ring 202 such that the topology of the ring 108 and/or the sub-ring 202 can converge for each of the nodes (i.e., the topology table 700 can be completed for each of the nodes). Based on the topology information provided by the topology table 700, one or more network/ring management functions may be performed by nodes 102 or other components of the network system 100.

[0094] For example, Figure 17 shows a method 1700 for performing an efficient MAC flush in a network system with a major/closed ring and a sub-ring, according to one example embodiment. In particular, as will be described in greater detail below, the method 1700 reduces MAC flush operations in the major/closed ring in response to a failure in the sub ring. For illustrative purposes, the method 1700 will be described in relation to the network system 1800 of Figure 18, including the major/closed ring 1802 and the sub-ring 1804, but the method 1700 can be performed in relation to any network system with a major/closed ring and a sub-ring.

[0095] The method 1700 may commence at operation 1702 with an interconnecting node 102 in the network system 1800 (sometimes referred to as a layer 2 network system 1800) detecting a failure of a port/link 104 in the sub-ring 1804. For example, Figure 18 shows the set of nodes 102A-102F that form the major/closed ring 1802 using the links 104I-104₆ and the set of nodes 102D and 102F-102I that form the sub-ring 1804 using the links 104₇-104I_O. In this network system 1800, link 104(, may be designated as the ring protection link 104 and the nodes 102D and 102F are the interconnecting nodes 102, as these nodes 102D and 102F interconnect the major/closed ring 1802 and the sub-ring 1804. At operation 1702, the interconnecting node 102D may detect a failure of port 1 of node 102H, which consequently causes the link 104io to fail and the ring protection link 104(, to be unblocked. In one embodiment, interconnecting node 102D detects a failure of the link 104io based on a signal-fail (SF) R-APS packet originating from node 102H and forwarded to node 102D by the node 102G, as shown in Figure 19. In particular, as shown in Figure 19, signal-fail (SF) packets are generated by the nodes 102H and 1021 in response to detecting the link 104 io failure and forwarded throughout the sub-ring 1804. [0096] In response to detecting a failure of a port/link 104 in the sub-ring 1804 (i.e., in response to receipt of a signal-fail packet), the interconnecting node 102D retrieves topology information for one or more of the major/closed ring 1802 and the sub-ring 1804 at operation 1704. For example, in response to a signal fail R-APS packet, the interconnecting node 102D may retrieve a topology table 700 corresponding to the major/closed ring 1802 and/or the sub-ring 1804. These topology tables 700 may be generated based on the techniques described herein, including the methods 300, 1200, and/or 1600.

[0097] At operation 1706, the interconnecting node 102D may generate one or more packets for causing a MAC flush operation in other nodes 102 in the major/closed ring 1802. For example, the interconnecting node 102D may generate a topology change notification (TCN) packet according to an APS protocol that causes nodes 102 to perform a MAC flush operation. A MAC flush operation causes a corresponding node 102 to flush all MAC addresses in a corresponding forwarding table. In particular, the nodes 102 can establish a set of MAC forwarding tables based on their discovery of the reachability of various host devices in the network system 1800 through corresponding nodes 102. Using these MAC forwarding tables, the nodes 102 receive, process, and forward data/packets to a destination host device. However, when there is a failure in the network system 1800, the forwarding information in these MAC forwarding tables may now be incorrect as the topology of the network system 1800 has been altered. Accordingly, the MAC flush operation clears these possible inaccurate tables. However, the MAC flush operations are costly in terms of consuming time and resources to perform. Accordingly, unnecessary or repetitive performance should be avoided.

[0098] At operation 1708, the interconnecting node 102D may transmit the one or more packets to neighbor nodes in the major/closed ring 1802. In one embodiment, the interconnecting node 102D generates the one or more packets for causing the MAC flush operation by addressing a single TCN packet along a path of nodes 102 that connect the interconnecting nodes 102D and 102F. In particular, since the interconnecting nodes 102D and 102F are connected in the major ring 1802 via the nodes 102A-102C, the TCN packet is transmitted from the node 102D through port 0. Further, the TCN packet is addressed to the node 102 just before the interconnecting node 102F (i.e., the node 102C), as shown in Figure 20, such that the interconnecting node 102F does not receive the TCN packet and is not triggered to perform a MAC flush operation based on this TCN packet. In particular, the interconnecting node 102F will be triggered to perform a MAC flush based on receipt of a signal-fail packet from the node 1021 (as shown in Figure 19). Thus, another MAC flush operation at interconnecting node 102F caused by a TCN packet is unnecessary.

[0099] As described above, using a new set of control packets (e.g., the set of topology information packets, including introduction topology information packets 400 and neighbor topology information packets 500), topology information can be derived for major/closed rings and/or sub-rings in a network system. In particular, the set of topology information packets facilitate the generation of a topology table describing the connections between nodes 102 in major/closed and/or sub-rings in a network system using one or more of the methods 300, 1200, and/or 1600. This topology information may thereafter be used for performing/guiding one or more network/ring management functions.

[00100] Turning now to Figure 21, a method 2100 will be described for performing a management function using layer 2 topology information in a layer 2 network system 1800, according to one example embodiment. The operations in the flow diagrams will be described with reference to the exemplary embodiments of the other figures. However, it should be understood that the operations of the flow diagrams can be performed by embodiments of the invention other than those discussed with reference to the other figures, and the embodiments of the invention discussed with reference to these other figures can perform operations different than those discussed with reference to the flow diagrams.

[00101] Each of the operations of the method 2100 may be performed by one or more nodes 102 in a network system (e.g., the network system 1800). For example, the method 2100 may be separately performed in entirely or partially overlapping time periods by each of the ERPS instances 106 in each of the nodes 102.

[00102] The method 2100 may commence at operation 2102 with a first interconnecting node 102D, shared by a closed ring 1802 and a sub-ring 1804 of the network system, determining topology information for the closed ring 1802 and the sub-ring 1804. In one embodiment, the closed ring 1802 includes a first set of nodes 102A-102F, including the first interconnecting node 102D, and the sub-ring 1804 includes a second set of nodes 102D and 102F-102I, including the first interconnecting node 102D, and the topology information describes links 104_I- 104₆ between the first set of nodes 102A-102F and links 104₇-104_IO between the second set of nodes 102D and 102F-102I. In some embodiments, the topology information is determined based on topology information packets (e.g., packets 400 and 500) that indicate one or more of (1) identifiers 408A-408B of an originating node 102 of a corresponding topology information packet and (2) identifiers 502A-502B and 504A-504B of neighbor nodes 102 to the originating node 102. As shown in Figure 18, the first set of nodes 102A-102F and the second set of nodes 102D and 102F-102I include a second interconnecting node 102F. The first interconnecting node 102D is connected to the second interconnecting node 102F in the closed ring 1802 via a set of links 104, and the first interconnecting node 102D is not connected to the second interconnecting node 102F in the sub-ring 1804.

[00103] In one embodiment, the topology information packets include (1) an introduction topology information packet 400 that includes an identifier 408A-408B of the originating node 102 and (2) a neighbor topology information packet 500 that includes an identifier 408A-408B of the originating node 102 and identifiers 502A-502B and 504A-504B of neighbor nodes 102 of the originating node 102. In this embodiment, determining the network topology information includes populating a first neighbor table 600 based on received topology information packets, wherein the first neighbor table 600 indicates neighbor nodes 102 from the first set of nodes 102A-102F relative to the first interconnecting node 102D. Determining the network topology information in this embodiment further includes populating a first network topology table 700 based on received topology information packets for the first set of nodes 102A-102F in the closed ring 1802, wherein the first network topology table 700 includes an entry 708 for each node 102 in the first set of nodes 102A-102F and each entry 708 includes a first identifier field 704 corresponding to a first neighbor node 102 and a second identifier field 706 corresponding to a second neighbor node 102 of an associated node 102.

[00104] In one embodiment, determining network topology information further includes populating a second neighbor table 600 based on received topology information packets, wherein the second neighbor table 600 indicates neighbor nodes 102 from the second set of nodes 102D and 102F-102I relative to the first interconnecting node 102D. Determining the network topology information in this embodiment further includes populating a second network topology table 700 based on received topology information packets for the second set of nodes 102D and 102F-102I in the sub-ring 1804, wherein the second topology table 700 includes an entry 708 for each node 102 in the second set of nodes 102D and 102F-102I and each entry 708 includes a first identifier field 704 corresponding to a first neighbor node 102 and a second identifier field 706 corresponding to a second neighbor node 102 of an associated node 102. [00105] At operation 2104, the first interconnecting node 102D generates a set of management packets based on the topology information for performing the management function in the network system 1800. In one embodiment, the set of management packets includes a topology change notification (TCN) packet that is addressed to a node 102C immediately preceding the second interconnecting node 102F along the set of links 104.

[00106] At operation 2106, the first interconnecting node 102D transmits the set of management packets in the closed ring 1802 of the network system 1800 to perform the management function. In one embodiment, the management function controls media access control (MAC) flush operations by the first set of nodes 102A-102F.

[00107] In some embodiments, the introduction topology information packet 400 and the neighbor topology information packet 500 are automatic protection switching (APS) packets with the same request/state identifier 402 but different sub-codes 404.

[00108] An electronic device stores and transmits (internally and/or with other electronic devices over a network) code (which is composed of software instructions and which is sometimes referred to as computer program code or a computer program) and/or data using machine-readable media (also called computer-readable media), such as machine -readable storage media (e.g., magnetic disks, optical disks, solid state drives, read only memory (ROM), flash memory devices, phase change memory) and machine-readable transmission media (also called a carrier) (e.g., electrical, optical, radio, acoustical or other form of propagated signals - such as carrier waves, infrared signals). Thus, an electronic device (e.g., a computer) includes hardware and software, such as a set of one or more processors (e.g., wherein a processor is a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application specific integrated circuit, field programmable gate array, other electronic circuitry, a combination of one or more of the preceding) coupled to one or more machine-readable storage media to store code for execution on the set of processors and/or to store data. For instance, an electronic device may include non-volatile memory containing the code since the non-volatile memory can persist code/data even when the electronic device is turned off (when power is removed), and while the electronic device is turned on that part of the code that is to be executed by the processor(s) of that electronic device is typically copied from the slower non-volatile memory into volatile memory (e.g., dynamic random access memory (DRAM), static random access memory (SRAM)) of that electronic device. Typical electronic devices also include a set or one or more physical network interface(s) (NI(s)) to establish network connections (to transmit and/or receive code and/or data using propagating signals) with other electronic devices. For example, the set of physical NIs (or the set of physical NI(s) in combination with the set of processors executing code) may perform any formatting, coding, or translating to allow the electronic device to send and receive data whether over a wired and/or a wireless connection. In some embodiments, a physical NI may comprise radio circuitry capable of receiving data from other electronic devices over a wireless connection and/or sending data out to other devices via a wireless connection. This radio circuitry may include transmitter(s), receiver(s), and/or transceiver(s) suitable for radiofrequency communication. The radio circuitry may convert digital data into a radio signal having the appropriate parameters (e.g., frequency, timing, channel, bandwidth, etc.). The radio signal may then be transmitted via antennas to the appropriate recipient(s). In some embodiments, the set of physical NI(s) may comprise network interface controller(s) (NICs), also known as a network interface card, network adapter, or local area network (LAN) adapter. The NIC(s) may facilitate in connecting the electronic device to other electronic devices allowing them to communicate via wire through plugging in a cable to a physical port connected to a NIC. One or more parts of an embodiment of the invention may be implemented using different combinations of software, firmware, and/or hardware.

[00109] A network device (ND) is an electronic device that communicatively interconnects other electronic devices on the network (e.g., other network devices, end-user devices). Some network devices are“multiple services network devices” that provide support for multiple networking functions (e.g., routing, bridging, switching, Layer 2 aggregation, session border control, Quality of Service, and/or subscriber management), and/or provide support for multiple application services (e.g., data, voice, and video).

[00110] Figure 22A illustrates connectivity between network devices (NDs) within an exemplary network, as well as three exemplary implementations of the NDs, according to some embodiments of the invention. Figure 22A shows NDs 2200A-H, and their connectivity by way of lines between 2200A-2200B, 2200B-2200C, 2200C-2200D, 2200D- 2200E, 2200E-2200F, 2200F-2200G, and 2200A-2200G, as well as between 2200H and each of 2200A, 2200C, 2200D, and 2200G. These NDs are physical devices, and the connectivity between these NDs can be wireless or wired (often referred to as a link). An additional line extending from NDs 2200A, 2200E, and 2200F illustrates that these NDs act as ingress and egress points for the network (and thus, these NDs are sometimes referred to as edge NDs; while the other NDs may be called core NDs). [00111] Two of the exemplary ND implementations in Figure 22A are: 1) a special- purpose network device 2202 that uses custom application-specific integrated-circuits (ASICs) and a special-purpose operating system (OS); and 2) a general purpose network device 2204 that uses common off-the-shelf (COTS) processors and a standard OS.

[00112] The special-purpose network device 2202 includes networking hardware 2210 comprising a set of one or more processor(s) 2212, forwarding resource(s) 2214 (which typically include one or more ASICs and/or network processors), and physical network interfaces (NIs) 2216 (through which network connections are made, such as those shown by the connectivity between NDs 2200A-H), as well as non-transitory machine readable storage media 2218 having stored therein networking software 2220. During operation, the networking software 2220 may be executed by the networking hardware 2210 to instantiate a set of one or more networking software instance(s) 2222. Each of the networking software instance(s) 2222, and that part of the networking hardware 2210 that executes that network software instance (be it hardware dedicated to that networking software instance and/or time slices of hardware temporally shared by that networking software instance with others of the networking software instance(s) 2222), form a separate virtual network element 2230A-R. Each of the virtual network element(s) (VNEs) 2230A-R includes a control communication and configuration module 2232A-R (sometimes referred to as a local control module or control communication module) and forwarding table(s) 2234A-R, such that a given virtual network element (e.g., 2230A) includes the control communication and configuration module (e.g., 2232A), a set of one or more forwarding table(s) (e.g., 2234A), and that portion of the networking hardware 2210 that executes the virtual network element (e.g., 2230A).

[00113] The special-purpose network device 2202 is often physically and/or logically considered to include: 1) a ND control plane 2224 (sometimes referred to as a control plane) comprising the processor(s) 2212 that execute the control communication and configuration module(s) 2232A-R; and 2) a ND forwarding plane 2226 (sometimes referred to as a forwarding plane, a data plane, or a media plane) comprising the forwarding resource(s) 2214 that utilize the forwarding table(s) 2234A-R and the physical NIs 2216. By way of example, where the ND is a router (or is implementing routing functionality), the ND control plane 2224 (the processor(s) 2212 executing the control communication and configuration module(s) 2232A-R) is typically responsible for participating in controlling how data (e.g., packets) is to be routed (e.g., the next hop for the data and the outgoing physical NI for that data) and storing that routing information in the forwarding table(s) 2234A-R, and the ND forwarding plane 2226 is responsible for receiving that data on the physical NIs 2216 and forwarding that data out the appropriate ones of the physical NIs 2216 based on the forwarding table(s) 2234A-R.

[00114] Figure 22B illustrates an exemplary way to implement the special-purpose network device 2202 according to some embodiments of the invention. Figure 22B shows a special-purpose network device including cards 2238 (typically hot pluggable). While in some embodiments the cards 2238 are of two types (one or more that operate as the ND forwarding plane 2226 (sometimes called line cards), and one or more that operate to implement the ND control plane 2224 (sometimes called control cards)), alternative embodiments may combine functionality onto a single card and/or include additional card types (e.g., one additional type of card is called a service card, resource card, or multi application card). A service card can provide specialized processing (e.g., Layer 4 to Layer 7 services (e.g., firewall, Internet Protocol Security (IPsec), Secure Sockets Layer (SSL) / Transport Layer Security (TLS), Intrusion Detection System (IDS), peer-to-peer (P2P), Voice over IP (VoIP) Session Border Controller, Mobile Wireless Gateways (Gateway General Packet Radio Service (GPRS) Support Node (GGSN), Evolved Packet Core (EPC) Gateway)). By way of example, a service card may be used to terminate IPsec tunnels and execute the attendant authentication and encryption algorithms. These cards are coupled together through one or more interconnect mechanisms illustrated as backplane 2236 (e.g., a first full mesh coupling the line cards and a second full mesh coupling all of the cards).

[00115] Returning to Figure 22A, the general purpose network device 2204 includes hardware 2240 comprising a set of one or more processor(s) 2242 (which are often COTS processors) and physical NIs 2246, as well as non-transitory machine readable storage media 2248 having stored therein software 2250 and/or an ERPS instance 106. During operation, the processor(s) 2242 execute the software 2250 and/or an ERPS instance 106 to instantiate one or more sets of one or more applications 2264A-R. While one embodiment does not implement virtualization, alternative embodiments may use different forms of virtualization. For example, in one such alternative embodiment the virtualization layer 2254 represents the kernel of an operating system (or a shim executing on a base operating system) that allows for the creation of multiple instances 2262A-R called software containers that may each be used to execute one (or more) of the sets of applications 2264A- R; where the multiple software containers (also called virtualization engines, virtual private servers, or jails) are user spaces (typically a virtual memory space) that are separate from each other and separate from the kernel space in which the operating system is run; and where the set of applications running in a given user space, unless explicitly allowed, cannot access the memory of the other processes. In another such alternative embodiment the virtualization layer 2254 represents a hypervisor (sometimes referred to as a virtual machine monitor (VMM)) or a hypervisor executing on top of a host operating system, and each of the sets of applications 2264A-R is run on top of a guest operating system within an instance 2262A-R called a virtual machine (which may in some cases be considered a tightly isolated form of software container) that is run on top of the hypervisor - the guest operating system and application may not know they are running on a virtual machine as opposed to running on a“bare metal” host electronic device, or through para-virtualization the operating system and/or application may be aware of the presence of virtualization for optimization purposes. In yet other alternative embodiments, one, some or all of the applications are implemented as unikernel(s), which can be generated by compiling directly with an application only a limited set of libraries (e.g., from a library operating system (LibOS) including drivers/libraries of OS services) that provide the particular 08 services needed by the application. As a unikernel can be implemented to run directly on hardware 2240, directly on a hypervisor (in which case the unikemel is sometimes described as running within a LibOS virtual machine), or in a software container, embodiments can be implemented fully with unikemels running directly on a hypervisor represented by virtualization layer 2254, unikernels running within software containers represented by instances 2262A-R, or as a combination of unikemels and the above-described techniques (e.g., unikemels and virtual machines both run directly on a hypervisor, unikemels and sets of applications that are mn in different software containers).

[00116] The instantiation of the one or more sets of one or more applications 2264 A-R, as well as virtualization if implemented, are collectively referred to as software instance(s) 2252. Each set of applications 2264A-R, corresponding virtualization constmct (e.g., instance 2262A-R) if implemented, and that part of the hardware 2240 that executes them (be it hardware dedicated to that execution and/or time slices of hardware temporally shared), forms a separate virtual network element(s) 2260A-R.

[00117] The virtual network element(s) 2260A-R perform similar functionality to the virtual network element(s) 2230A-R - e.g., similar to the control communication and configuration module(s) 2232A and forwarding table(s) 2234A (this virtualization of the hardware 2240 is sometimes referred to as network function virtualization (NFV)). Thus, NFV may be used to consolidate many network equipment types onto industry standard high volume server hardware, physical switches, and physical storage, which could be located in Data centers, NDs, and customer premise equipment (CPE). While embodiments of the invention are illustrated with each instance 2262A-R corresponding to one VNE 2260A-R, alternative embodiments may implement this correspondence at a finer level granularity (e.g., line card virtual machines virtualize line cards, control card virtual machine virtualize control cards, etc.); it should be understood that the techniques described herein with reference to a correspondence of instances 2262A-R to VNEs also apply to embodiments where such a finer level of granularity and/or unikernels are used.

[00118] In certain embodiments, the virtualization layer 2254 includes a virtual switch that provides similar forwarding services as a physical Ethernet switch. Specifically, this virtual switch forwards traffic between instances 2262A-R and the physical NI(s) 2246, as well as optionally between the instances 2262A-R; in addition, this virtual switch may enforce network isolation between the VNEs 2260A-R that by policy are not permitted to communicate with each other (e.g., by honoring virtual local area networks (VLANs)).

[00119] The third exemplary ND implementation in Figure 22 A is a hybrid network device 2206, which includes both custom ASICs/special-purpose OS and COTS processors/standard OS in a single ND or a single card within an ND. In certain embodiments of such a hybrid network device, a platform VM (i.e., a VM that that implements the functionality of the special-purpose network device 2202) could provide for para-virtualization to the networking hardware present in the hybrid network device 2206.

[00120] Regardless of the above exemplary implementations of an ND, when a single one of multiple VNEs implemented by an ND is being considered (e.g., only one of the VNEs is part of a given virtual network) or where only a single VNE is currently being implemented by an ND, the shortened term network element (NE) is sometimes used to refer to that VNE. Also in all of the above exemplary implementations, each of the VNEs (e.g., VNE(s) 2230A-R, VNEs 2260A-R, and those in the hybrid network device 2206) receives data on the physical NIs (e.g., 2216, 2246) and forwards that data out the appropriate ones of the physical NIs (e.g., 2216, 2246). For example, a VNE implementing IP router functionality forwards IP packets on the basis of some of the IP header information in the IP packet; where IP header information includes source IP address, destination IP address, source port, destination port (where“source port” and“destination port” refer herein to protocol ports, as opposed to physical ports of a ND), transport protocol (e.g., user datagram protocol (UDP), Transmission Control Protocol (TCP), and differentiated services code point (DSCP) values.

[00121] Figure 22C illustrates various exemplary ways in which VNEs may be coupled according to some embodiments of the invention. Figure 22C shows VNEs 2270A.1- 2270A.P (and optionally VNEs 2270A.Q-2270A.R) implemented in ND 2200A and VNE 2270H.1 in ND 2200H. In Figure 22C, VNEs 2270A.1-P are separate from each other in the sense that they can receive packets from outside ND 2200A and forward packets outside of ND 2200A; VNE 2270A.1 is coupled with VNE 2270H.1, and thus they communicate packets between their respective NDs; VNE 2270A.2-2270A.3 may optionally forward packets between themselves without forwarding them outside of the ND 2200A; and VNE 2270A.P may optionally be the first in a chain of VNEs that includes VNE 2270A.Q followed by VNE 2270A.R (this is sometimes referred to as dynamic service chaining, where each of the VNEs in the series of VNEs provides a different service - e.g., one or more layer 4-7 network services). While Figure 22C illustrates various exemplary relationships between the VNEs, alternative embodiments may support other relationships (e.g., more/fewer VNEs, more/fewer dynamic service chains, multiple different dynamic service chains with some common VNEs and some different VNEs).

[00122] The NDs of Figure 22A, for example, may form part of the Internet or a private network; and other electronic devices (not shown; such as end user devices including workstations, laptops, netbooks, tablets, palm tops, mobile phones, smartphones, phablets, multimedia phones, Voice Over Internet Protocol (VOIP) phones, terminals, portable media players, GPS units, wearable devices, gaming systems, set-top boxes, Internet enabled household appliances) may be coupled to the network (directly or through other networks such as access networks) to communicate over the network (e.g., the Internet or virtual private networks (VPNs) overlaid on (e.g., tunneled through) the Internet) with each other (directly or through servers) and/or access content and/or services. Such content and/or services are typically provided by one or more servers (not shown) belonging to a service/content provider or one or more end user devices (not shown) participating in a peer- to-peer (P2P) service, and may include, for example, public webpages (e.g., free content, store fronts, search services), private webpages (e.g., username/password accessed webpages providing email services), and/or corporate networks over VPNs. For instance, end user devices may be coupled (e.g., through customer premise equipment coupled to an access network (wired or wirelessly)) to edge NDs, which are coupled (e.g., through one or more core NDs) to other edge NDs, which are coupled to electronic devices acting as servers. However, through compute and storage virtualization, one or more of the electronic devices operating as the NDs in Figure 22A may also host one or more such servers (e.g., in the case of the general purpose network device 2204, one or more of the software instances 2262A-R may operate as servers; the same would be true for the hybrid network device 2206; in the case of the special-purpose network device 2202, one or more such servers could also be run on a virtualization layer executed by the processor(s) 2212); in which case the servers are said to be co-located with the VNEs of that ND.

[00123] A virtual network is a logical abstraction of a physical network (such as that in Figure 22A) that provides network services (e.g., L2 and/or L3 services). A virtual network can be implemented as an overlay network (sometimes referred to as a network virtualization overlay) that provides network services (e.g., layer 2 (L2, data link layer) and/or layer 3 (L3, network layer) services) over an underlay network (e.g., an L3 network, such as an Internet Protocol (IP) network that uses tunnels (e.g., generic routing encapsulation (GRE), layer 2 tunneling protocol (L2TP), IPSec) to create the overlay network).

[00124] A network virtualization edge (NVE) sits at the edge of the underlay network and participates in implementing the network virtualization; the network-facing side of the NVE uses the underlay network to tunnel frames to and from other NVEs; the outward-facing side of the NVE sends and receives data to and from systems outside the network. A virtual network instance (VNI) is a specific instance of a virtual network on a NVE (e.g., a NE/VNE on an ND, a part of a NE/VNE on a ND where that NE/VNE is divided into multiple VNEs through emulation); one or more VNIs can be instantiated on an NVE (e.g., as different VNEs on an ND). A virtual access point (VAP) is a logical connection point on the NVE for connecting external systems to a virtual network; a VAP can be physical or virtual ports identified through logical interface identifiers (e.g., a VLAN ID).

[00125] Examples of network services include: 1) an Ethernet LAN emulation service (an Ethernet-based multipoint service similar to an Internet Engineering Task Force (IETF) Multiprotocol Label Switching (MPLS) or Ethernet VPN (EVPN) service) in which external systems are interconnected across the network by a LAN environment over the underlay network (e.g., an NVE provides separate L2 VNIs (virtual switching instances) for different such virtual networks, and L3 (e.g., IP/MPLS) tunneling encapsulation across the underlay network); and 2) a virtualized IP forwarding service (similar to IETF IP VPN (e.g., Border Gateway Protocol (BGP)/MPLS IP VPN) from a service definition perspective) in which external systems are interconnected across the network by an L3 environment over the underlay network (e.g., an NVE provides separate L3 VNIs (forwarding and routing instances) for different such virtual networks, and L3 (e.g., IP/MPLS) tunneling encapsulation across the underlay network)). Network services may also include quality of service capabilities (e.g., traffic classification marking, traffic conditioning and scheduling), security capabilities (e.g., filters to protect customer premises from network - originated attacks, to avoid malformed route announcements), and management capabilities (e.g., full detection and processing).

[00126] Fig. 22D illustrates a network with a single network element on each of the NDs of Figure 22A, and within this straight forward approach contrasts a traditional distributed approach (commonly used by traditional routers) with a centralized approach for maintaining reachability and forwarding information (also called network control), according to some embodiments of the invention. Specifically, Figure 22D illustrates network elements (NEs) 2270A-H with the same connectivity as the NDs 2200A-H of Figure 22A.

[00127] Figure 22D illustrates that the distributed approach 2272 distributes responsibility for generating the reachability and forwarding information across the NEs 2270A-H; in other words, the process of neighbor discovery and topology discovery is distributed.

[00128] For example, where the special-purpose network device 2202 is used, the control communication and configuration module(s) 2232A-R of the ND control plane 2224 typically include a reachability and forwarding information module to implement one or more routing protocols (e.g., an exterior gateway protocol such as Border Gateway Protocol (BGP), Interior Gateway Protocol(s) (IGP) (e.g., Open Shortest Path First (OSPF), Intermediate System to Intermediate System (IS -IS), Routing Information Protocol (RIP), Fabel Distribution Protocol (FDP), Resource Reservation Protocol (RSVP) (including RS VP-Traffic Engineering (TE): Extensions to RSVP for ESP Tunnels and Generalized Multi-Protocol Fabel Switching (GMPFS) Signaling RSVP-TE)) that communicate with other NEs to exchange routes, and then selects those routes based on one or more routing metrics. Thus, the NEs 2270A-H (e.g., the processor(s) 2212 executing the control communication and configuration module(s) 2232A-R) perform their responsibility for participating in controlling how data (e.g., packets) is to be routed (e.g., the next hop for the data and the outgoing physical NI for that data) by distributively determining the reachability within the network and calculating their respective forwarding information. Routes and adjacencies are stored in one or more routing structures (e.g., Routing Information Base (RIB), Label Information Base (LIB), one or more adjacency structures) on the ND control plane 2224. The ND control plane 2224 programs the ND forwarding plane 2226 with information (e.g., adjacency and route information) based on the routing stmcture(s). For example, the ND control plane 2224 programs the adjacency and route information into one or more forwarding table(s) 2234A-R (e.g., Forwarding Information Base (FIB), Label Forwarding Information Base (LFIB), and one or more adjacency structures) on the ND forwarding plane 2226. For layer 2 forwarding, the ND can store one or more bridging tables that are used to forward data based on the layer 2 information in that data. While the above example uses the special-purpose network device 2202, the same distributed approach 2272 can be implemented on the general purpose network device 2204 and the hybrid network device 2206.

[00129] Figure 22D illustrates that a centralized approach 2274 (also known as software defined networking (SDN)) that decouples the system that makes decisions about where traffic is sent from the underlying systems that forwards traffic to the selected destination. The illustrated centralized approach 2274 has the responsibility for the generation of reachability and forwarding information in a centralized control plane 2276 (sometimes referred to as a SDN control module, controller, network controller, OpenFlow controller, SDN controller, control plane node, network virtualization authority, or management control entity), and thus the process of neighbor discovery and topology discovery is centralized. The centralized control plane 2276 has a south bound interface 2282 with a data plane 2280 (sometime referred to the infrastructure layer, network forwarding plane, or forwarding plane (which should not be confused with a ND forwarding plane)) that includes the NEs 2270A-H (sometimes referred to as switches, forwarding elements, data plane elements, or nodes). The centralized control plane 2276 includes a network controller 2278, which includes a centralized reachability and forwarding information module 2279 that determines the reachability within the network and distributes the forwarding information to the NEs 2270A-H of the data plane 2280 over the south bound interface 2282 (which may use the OpenFlow protocol). Thus, the network intelligence is centralized in the centralized control plane 2276 executing on electronic devices that are typically separate from the NDs.

[00130] For example, where the special-purpose network device 2202 is used in the data plane 2280, each of the control communication and configuration module(s) 2232A-R of the ND control plane 2224 typically include a control agent that provides the VNE side of the south bound interface 2282. In this case, the ND control plane 2224 (the processor(s) 2212 executing the control communication and configuration module(s) 2232A-R) performs its responsibility for participating in controlling how data (e.g., packets) is to be routed (e.g., the next hop for the data and the outgoing physical NI for that data) through the control agent communicating with the centralized control plane 2276 to receive the forwarding information (and in some cases, the reachability information) from the centralized reachability and forwarding information module 2279 (it should be understood that in some embodiments of the invention, the control communication and configuration module(s) 2232A-R, in addition to communicating with the centralized control plane 2276, may also play some role in determining reachability and/or calculating forwarding information - albeit less so than in the case of a distributed approach; such embodiments are generally considered to fall under the centralized approach 2274, but may also be considered a hybrid approach).

[00131] While the above example uses the special-purpose network device 2202, the same centralized approach 2274 can be implemented with the general purpose network device 2204 (e.g., each of the VNE 2260A-R performs its responsibility for controlling how data (e.g., packets) is to be routed (e.g., the next hop for the data and the outgoing physical NI for that data) by communicating with the centralized control plane 2276 to receive the forwarding information (and in some cases, the reachability information) from the centralized reachability and forwarding information module 2279; it should be understood that in some embodiments of the invention, the VNEs 2260A-R, in addition to communicating with the centralized control plane 2276, may also play some role in determining reachability and/or calculating forwarding information - albeit less so than in the case of a distributed approach) and the hybrid network device 2206. In fact, the use of SDN techniques can enhance the NFV techniques typically used in the general purpose network device 2204 or hybrid network device 2206 implementations as NFV is able to support SDN by providing an infrastructure upon which the SDN software can be run, and NFV and SDN both aim to make use of commodity server hardware and physical switches.

[00132] Figure 22D also shows that the centralized control plane 2276 has a north bound interface 2284 to an application layer 2286, in which resides application(s) 2288 and/or an ERPS instance 106. The centralized control plane 2276 has the ability to form virtual networks 2292 (sometimes referred to as a logical forwarding plane, network services, or overlay networks (with the NEs 2270A-H of the data plane 2280 being the underlay network)) for the application(s) 2288 and/or an ERPS instance 106. Thus, the centralized control plane 2276 maintains a global view of all NDs and configured NEs/VNEs, and it maps the virtual networks to the underlying NDs efficiently (including maintaining these mappings as the physical network changes either through hardware (ND, link, or ND component) failure, addition, or removal).

[00133] While Figure 22D shows the distributed approach 2272 separate from the centralized approach 2274, the effort of network control may be distributed differently or the two combined in certain embodiments of the invention. For example: 1) embodiments may generally use the centralized approach (SDN) 2274, but have certain functions delegated to the NEs (e.g., the distributed approach may be used to implement one or more of fault monitoring, performance monitoring, protection switching, and primitives for neighbor and/or topology discovery); or 2) embodiments of the invention may perform neighbor discovery and topology discovery via both the centralized control plane and the distributed protocols, and the results compared to raise exceptions where they do not agree. Such embodiments are generally considered to fall under the centralized approach 2274, but may also be considered a hybrid approach.

[00134] While Figure 22D illustrates the simple case where each of the NDs 2200A-H implements a single NE 2270A-H, it should be understood that the network control approaches described with reference to Figure 22D also work for networks where one or more of the NDs 2200A-H implement multiple VNEs (e.g., VNEs 2230A-R, VNEs 2260A- R, those in the hybrid network device 2206). Alternatively or in addition, the network controller 2278 may also emulate the implementation of multiple VNEs in a single ND. Specifically, instead of (or in addition to) implementing multiple VNEs in a single ND, the network controller 2278 may present the implementation of a VNE/NE in a single ND as multiple VNEs in the virtual networks 2292 (all in the same one of the virtual network(s) 2292, each in different ones of the virtual network(s) 2292, or some combination). For example, the network controller 2278 may cause an ND to implement a single VNE (a NE) in the underlay network, and then logically divide up the resources of that NE within the centralized control plane 2276 to present different VNEs in the virtual network(s) 2292 (where these different VNEs in the overlay networks are sharing the resources of the single VNE/NE implementation on the ND in the underlay network). [00135] On the other hand, Figures 22E and 22F respectively illustrate exemplary abstractions of NEs and VNEs that the network controller 2278 may present as part of different ones of the virtual networks 2292. Figure 22E illustrates the simple case of where each of the NDs 2200A-H implements a single NE 2270A-H (see Figure 22D), but the centralized control plane 2276 has abstracted multiple of the NEs in different NDs (the NEs 2270A-C and G-H) into (to represent) a single NE 22701 in one of the virtual network(s) 2292 of Figure 22D, according to some embodiments of the invention. Figure 22E shows that in this virtual network, the NE 22701 is coupled to NE 2270D and 2270F, which are both still coupled to NE 2270E.

[00136] Figure 22F illustrates a case where multiple VNEs (VNE 2270A.1 and VNE 2270H.1) are implemented on different NDs (ND 2200 A and ND 2200H) and are coupled to each other, and where the centralized control plane 2276 has abstracted these multiple VNEs such that they appear as a single VNE 2270T within one of the virtual networks 2292 of Figure 22D, according to some embodiments of the invention. Thus, the abstraction of a NE or VNE can span multiple NDs.

[00137] While some embodiments of the invention implement the centralized control plane 2276 as a single entity (e.g., a single instance of software running on a single electronic device), alternative embodiments may spread the functionality across multiple entities for redundancy and/or scalability purposes (e.g., multiple instances of software running on different electronic devices).

[00138] Similar to the network device implementations, the electronic device(s) running the centralized control plane 2276, and thus the network controller 2278 including the centralized reachability and forwarding information module 2279, may be implemented a variety of ways (e.g., a special purpose device, a general-purpose (e.g., COTS) device, or hybrid device). These electronic device(s) would similarly include processor(s), a set or one or more physical NIs, and a non-transitory machine -readable storage medium having stored thereon the centralized control plane software. For instance, Figure 23 illustrates, a general purpose control plane device 2304 including hardware 2340 comprising a set of one or more processor(s) 2342 (which are often COTS processors) and physical NIs 2346, as well as non-transitory machine readable storage media 2348 having stored therein centralized control plane (CCP) software 2350 and/or an ERPS instance 106.

[00139] In embodiments that use compute virtualization, the processor(s) 2342 typically execute software to instantiate a virtualization layer 2354 (e.g., in one embodiment the virtualization layer 2354 represents the kernel of an operating system (or a shim executing on a base operating system) that allows for the creation of multiple instances 2362A-R called software containers (representing separate user spaces and also called virtualization engines, virtual private servers, or jails) that may each be used to execute a set of one or more applications; in another embodiment the virtualization layer 2354 represents a hypervisor (sometimes referred to as a virtual machine monitor (VMM)) or a hypervisor executing on top of a host operating system, and an application is run on top of a guest operating system within an instance 2362A-R called a virtual machine (which in some cases may be considered a tightly isolated form of software container) that is run by the hypervisor ; in another embodiment, an application is implemented as a unikernel, which can be generated by compiling directly with an application only a limited set of libraries (e.g., from a library operating system (LibOS) including drivers/libraries of OS services) that provide the particular OS services needed by the application, and the unikemel can run directly on hardware 2340, directly on a hypervisor represented by virtualization layer 2354 (in which case the unikernel is sometimes described as running within a LibOS virtual machine), or in a software container represented by one of instances 2362A-R). Again, in embodiments where compute virtualization is used, during operation an instance of the CCP software 2350 (illustrated as CCP instance 2376A) is executed (e.g., within the instance 2362A) on the virtualization layer 2354. In embodiments where compute virtualization is not used, the CCP instance 2376A is executed, as a unikernel or on top of a host operating system, on the“bare metal” general purpose control plane device 2304. The instantiation of the CCP instance 2376A, as well as the virtualization layer 2354 and instances 2362A-R if implemented, are collectively referred to as software instance(s) 2352.

[00140] In some embodiments, the CCP instance 2376A includes a network controller instance 2378. The network controller instance 2378 includes a centralized reachability and forwarding information module instance 2379 (which is a middleware layer providing the context of the network controller 2278 to the operating system and communicating with the various NEs and for processing an ERPS instance 106), and an CCP application layer 2380 (sometimes referred to as an application layer) over the middleware layer (providing the intelligence required for various network operations such as protocols, network situational awareness, and user - interfaces). At a more abstract level, this CCP application layer 2380 within the centralized control plane 2276 works with virtual network view(s) (logical view(s) of the network) and the middleware layer provides the conversion from the virtual networks to the physical view.

[00141] The centralized control plane 2276 transmits relevant messages to the data plane 2280 based on CCP application layer 2380 calculations and middleware layer mapping for each flow. A flow may be defined as a set of packets whose headers match a given pattern of bits; in this sense, traditional IP forwarding is also flow-based forwarding where the flows are defined by the destination IP address for example; however, in other implementations, the given pattern of bits used for a flow definition may include more fields (e.g., 10 or more) in the packet headers. Different NDs/NEs/VNEs of the data plane 2280 may receive different messages, and thus different forwarding information. The data plane 2280 processes these messages and programs the appropriate flow information and corresponding actions in the forwarding tables (sometime referred to as flow tables) of the appropriate NE/VNEs, and then the NEs/VNEs map incoming packets to flows represented in the forwarding tables and forward packets based on the matches in the forwarding tables.

[00142] Standards such as OpenFlow define the protocols used for the messages, as well as a model for processing the packets. The model for processing packets includes header parsing, packet classification, and making forwarding decisions. Header parsing describes how to interpret a packet based upon a well-known set of protocols. Some protocol fields are used to build a match structure (or key) that will be used in packet classification (e.g., a first key field could be a source media access control (MAC) address, and a second key field could be a destination MAC address).

[00143] Packet classification involves executing a lookup in memory to classify the packet by determining which entry (also referred to as a forwarding table entry or flow entry) in the forwarding tables best matches the packet based upon the match structure, or key, of the forwarding table entries. It is possible that many flows represented in the forwarding table entries can correspond/match to a packet; in this case the system is typically configured to determine one forwarding table entry from the many according to a defined scheme (e.g., selecting a first forwarding table entry that is matched). Forwarding table entries include both a specific set of match criteria (a set of values or wildcards, or an indication of what portions of a packet should be compared to a particular value/values/wildcards, as defined by the matching capabilities - for specific fields in the packet header, or for some other packet content), and a set of one or more actions for the data plane to take on receiving a matching packet. For example, an action may be to push a header onto the packet, for the packet using a particular port, flood the packet, or simply drop the packet. Thus, a forwarding table entry for IPv4/IPv6 packets with a particular transmission control protocol (TCP) destination port could contain an action specifying that these packets should be dropped.

[00144] Making forwarding decisions and performing actions occurs, based upon the forwarding table entry identified during packet classification, by executing the set of actions identified in the matched forwarding table entry on the packet.

[00145] However, when an unknown packet (for example, a“missed packet” or a“match- miss” as used in OpenFlow parlance) arrives at the data plane 2280, the packet (or a subset of the packet header and content) is typically forwarded to the centralized control plane 2276. The centralized control plane 2276 will then program forwarding table entries into the data plane 2280 to accommodate packets belonging to the flow of the unknown packet. Once a specific forwarding table entry has been programmed into the data plane 2280 by the centralized control plane 2276, the next packet with matching credentials will match that forwarding table entry and take the set of actions associated with that matched entry.

[00146] A network interface (NI) may be physical or virtual; and in the context of IP, an interface address is an IP address assigned to a NI, be it a physical NI or virtual NI. A virtual NI may be associated with a physical NI, with another virtual interface, or stand on its own (e.g., a loopback interface, a point-to-point protocol interface). A NI (physical or virtual) may be numbered (a NI with an IP address) or unnumbered (a NI without an IP address). A loopback interface (and its loopback address) is a specific type of virtual NI (and IP address) of a NE/VNE (physical or virtual) often used for management purposes; where such an IP address is referred to as the nodal loopback address. The IP address(es) assigned to the NI(s) of a ND are referred to as IP addresses of that ND; at a more granular level, the IP address(es) assigned to NI(s) assigned to a NE/VNE implemented on a ND can be referred to as IP addresses of that NE/VNE.

[00147] Next hop selection by the routing system for a given destination may resolve to one path (that is, a routing protocol may generate one next hop on a shortest path); but if the routing system determines there are multiple viable next hops (that is, the routing protocol generated forwarding solution offers more than one next hop on a shortest path - multiple equal cost next hops), some additional criteria is used - for instance, in a connectionless network, Equal Cost Multi Path (ECMP) (also known as Equal Cost Multi Pathing, multipath forwarding and IP multipath) may be used (e.g., typical implementations use as the criteria particular header fields to ensure that the packets of a particular packet flow are always forwarded on the same next hop to preserve packet flow ordering). For purposes of multipath forwarding, a packet flow is defined as a set of packets that share an ordering constraint. As an example, the set of packets in a particular TCP transfer sequence need to arrive in order, else the TCP logic will interpret the out of order delivery as congestion and slow the TCP transfer rate down.

[00148] A Layer 3 (L3) Link Aggregation (LAG) link is a link directly connecting two NDs with multiple IP-addressed link paths (each link path is assigned a different IP address), and a load distribution decision across these different link paths is performed at the ND forwarding plane; in which case, a load distribution decision is made between the link paths.

[00149] Some NDs include functionality for authentication, authorization, and accounting (AAA) protocols (e.g., RADIUS (Remote Authentication Dial-In User Service), Diameter, and/or TACACS+ (Terminal Access Controller Access Control System Plus). AAA can be provided through a client/server model, where the AAA client is implemented on a ND and the AAA server can be implemented either locally on the ND or on a remote electronic device coupled with the ND. Authentication is the process of identifying and verifying a subscriber. For instance, a subscriber might be identified by a combination of a username and a password or through a unique key. Authorization determines what a subscriber can do after being authenticated, such as gaining access to certain electronic device information resources (e.g., through the use of access control policies). Accounting is recording user activity. By way of a summary example, end user devices may be coupled (e.g., through an access network) through an edge ND (supporting AAA processing) coupled to core NDs coupled to electronic devices implementing servers of service/content providers. AAA processing is performed to identify for a subscriber the subscriber record stored in the AAA server for that subscriber. A subscriber record includes a set of attributes (e.g., subscriber name, password, authentication information, access control information, rate-limiting information, policing information) used during processing of that subscriber’s traffic.

[00150] Certain NDs (e.g., certain edge NDs) internally represent end user devices (or sometimes customer premise equipment (CPE) such as a residential gateway (e.g., a router, modem)) using subscriber circuits. A subscriber circuit uniquely identifies within the ND a subscriber session and typically exists for the lifetime of the session. Thus, a ND typically allocates a subscriber circuit when the subscriber connects to that ND, and correspondingly de-allocates that subscriber circuit when that subscriber disconnects. Each subscriber session represents a distinguishable flow of packets communicated between the ND and an end user device (or sometimes CPE such as a residential gateway or modem) using a protocol, such as the point-to-point protocol over another protocol (PPPoX) (e.g., where X is Ethernet or Asynchronous Transfer Mode (ATM)), Ethernet, 802. IQ Virtual LAN (VLAN), Internet Protocol, or ATM). A subscriber session can be initiated using a variety of mechanisms (e.g., manual provisioning a dynamic host configuration protocol (DHCP), DHCP/client-less internet protocol service (CLIPS) or Media Access Control (MAC) address tracking). For example, the point-to-point protocol (PPP) is commonly used for digital subscriber line (DSL) services and requires installation of a PPP client that enables the subscriber to enter a username and a password, which in turn may be used to select a subscriber record. When DHCP is used (e.g., for cable modem services), a username typically is not provided; but in such situations other information (e.g., information that includes the MAC address of the hardware in the end user device (or CPE)) is provided. The use of DHCP and CLIPS on the ND captures the MAC addresses and uses these addresses to distinguish subscribers and access their subscriber records.

[00151] A virtual circuit (VC), synonymous with virtual connection and virtual channel, is a connection oriented communication service that is delivered by means of packet mode communication. Virtual circuit communication resembles circuit switching, since both are connection oriented, meaning that in both cases data is delivered in correct order, and signaling overhead is required during a connection establishment phase. Virtual circuits may exist at different layers. For example, at layer 4, a connection oriented transport layer datalink protocol such as Transmission Control Protocol (TCP) may rely on a connectionless packet switching network layer protocol such as IP, where different packets may be routed over different paths, and thus be delivered out of order. Where a reliable virtual circuit is established with TCP on top of the underlying unreliable and connectionless IP protocol, the virtual circuit is identified by the source and destination network socket address pair, i.e. the sender and receiver IP address and port number. However, a virtual circuit is possible since TCP includes segment numbering and reordering on the receiver side to prevent out-of-order delivery. Virtual circuits are also possible at Layer 3 (network layer) and Layer 2 (datalink layer); such virtual circuit protocols are based on connection oriented packet switching, meaning that data is always delivered along the same network path, i.e. through the same NEs/VNEs. In such protocols, the packets are not routed individually and complete addressing information is not provided in the header of each data packet; only a small virtual channel identifier (VCI) is required in each packet; and routing information is transferred to the NEs/VNEs during the connection establishment phase; switching only involves looking up the virtual channel identifier in a table rather than analyzing a complete address. Examples of network layer and datalink layer virtual circuit protocols, where data always is delivered over the same path: X.25, where the VC is identified by a virtual channel identifier (VCI); Frame relay, where the VC is identified by a VCI; Asynchronous Transfer Mode (ATM), where the circuit is identified by a virtual path identifier (VPI) and virtual channel identifier (VCI) pair; General Packet Radio Service (GPRS); and Multiprotocol label switching (MPLS), which can be used for IP over virtual circuits (Each circuit is identified by a label).

[00152] Certain NDs (e.g., certain edge NDs) use a hierarchy of circuits. The leaf nodes of the hierarchy of circuits are subscriber circuits. The subscriber circuits have parent circuits in the hierarchy that typically represent aggregations of multiple subscriber circuits, and thus the network segments and elements used to provide access network connectivity of those end user devices to the ND. These parent circuits may represent physical or logical aggregations of subscriber circuits (e.g., a virtual local area network (VLAN), a permanent virtual circuit (PVC) (e.g., for Asynchronous Transfer Mode (ATM)), a circuit-group, a channel, a pseudo-wire, a physical NI of the ND, and a link aggregation group). A circuit- group is a virtual construct that allows various sets of circuits to be grouped together for configuration purposes, for example aggregate rate control. A pseudo-wire is an emulation of a layer 2 point-to-point connection-oriented service. A link aggregation group is a virtual construct that merges multiple physical NIs for purposes of bandwidth aggregation and redundancy. Thus, the parent circuits physically or logically encapsulate the subscriber circuits.

[00153] Each VNE (e.g., a virtual router, a virtual bridge (which may act as a virtual switch instance in a Virtual Private LAN Service (VPLS) is typically independently administrable. For example, in the case of multiple virtual routers, each of the virtual routers may share system resources but is separate from the other virtual routers regarding its management domain, AAA (authentication, authorization, and accounting) name space, IP address, and routing database(s). Multiple VNEs may be employed in an edge ND to provide direct network access and/or different classes of services for subscribers of service and/or content providers. [00154] Within certain NDs,“interfaces” that are independent of physical NIs may be configured as part of the VNEs to provide higher-layer protocol and service information (e.g., Layer 3 addressing). The subscriber records in the AAA server identify, in addition to the other subscriber configuration requirements, to which context (e.g., which of the VNEs/NEs) the corresponding subscribers should be bound within the ND. As used herein, a binding forms an association between a physical entity (e.g., physical NI, channel) or a logical entity (e.g., circuit such as a subscriber circuit or logical circuit (a set of one or more subscriber circuits)) and a context’s interface over which network protocols (e.g., routing protocols, bridging protocols) are configured for that context. Subscriber data flows on the physical entity when some higher-layer protocol interface is configured and associated with that physical entity.

[00155] Some NDs provide support for implementing VPNs (Virtual Private Networks) (e.g., Layer 2 VPNs and/or Layer 3 VPNs). For example, the ND where a provider’s network and a customer’s network are coupled are respectively referred to as PEs (Provider Edge) and CEs (Customer Edge). In a Layer 2 VPN, forwarding typically is performed on the CE(s) on either end of the VPN and traffic is sent across the network (e.g., through one or more PEs coupled by other NDs). Layer 2 circuits are configured between the CEs and PEs (e.g., an Ethernet port, an ATM permanent virtual circuit (PVC), a Frame Relay PVC). In a Layer 3 VPN, routing typically is performed by the PEs. By way of example, an edge ND that supports multiple VNEs may be deployed as a PE; and a VNE may be configured with a VPN protocol, and thus that VNE is referred as a VPN VNE.

[00156] Some NDs provide support for VPLS (Virtual Private LAN Service). For example, in a VPLS network, end user devices access content/services provided through the VPLS network by coupling to CEs, which are coupled through PEs coupled by other NDs. VPLS networks can be used for implementing triple play network applications (e.g., data applications (e.g., high-speed Internet access), video applications (e.g., television service such as IPTV (Internet Protocol Television), VoD (Video-on-Demand) service), and voice applications (e.g., VoIP (Voice over Internet Protocol) service)), VPN services, etc. VPLS is a type of layer 2 VPN that can be used for multi-point connectivity. VPLS networks also allow end use devices that are coupled with CEs at separate geographical locations to communicate with each other across a Wide Area Network (WAN) as if they were directly attached to each other in a Local Area Network (LAN) (referred to as an emulated LAN). [00157] In VPLS networks, each CE typically attaches, possibly through an access network (wired and/or wireless), to a bridge module of a PE via an attachment circuit (e.g., a virtual link or connection between the CE and the PE). The bridge module of the PE attaches to an emulated LAN through an emulated LAN interface. Each bridge module acts as a“Virtual Switch Instance” (VSI) by maintaining a forwarding table that maps MAC addresses to pseudowires and attachment circuits. PEs forward frames (received from CEs) to destinations (e.g., other CEs, other PEs) based on the MAC destination address field included in those frames.

[00158] While the invention has been described in terms of several embodiments, those skilled in the art will recognize that the invention is not limited to the embodiments described, can be practiced with modification and alteration within the spirit and scope of the appended claims. The description is thus to be regarded as illustrative instead of limiting.

Claims

CLAIMS:

1. A method (2100) for performing a management function using layer 2 topology information in a layer 2 network system (1800), the method comprising:

determining (2102), by a first interconnecting node (102D) shared by a closed ring (1802) and a sub-ring (1804) of the layer 2 network system, topology information for the closed ring and the sub-ring, wherein the closed ring includes a first set of nodes (102A-102F), including the first interconnecting node, and the sub-ring includes a second set of nodes (102D and 102F-102I), including the first interconnecting node, and the topology information describes links (104_I-104₆) between the first set of nodes and links ( 104₇- 104io) between the second set of nodes, wherein the topology information is determined based on topology information packets (400, 500) that indicate one or more of (1) identifiers (408A-408B) of an originating node of a corresponding topology information packet and (2) identifiers (502A-502B and 504A-504B) of neighbor nodes to the originating node;

generating (2104), by the first interconnecting node, a set of management packets based on the topology information for performing the management function in the layer 2 network system; and

transmitting (2106), by the first interconnecting node, the set of management packets in the closed ring of the layer 2 network system to perform the management function.

2. The method of claim 1, wherein the management function controls media access control (MAC) flush operations by the first set of nodes.

3. The method of claim 2, wherein the first set of nodes and the second set of nodes include a second interconnecting node (102F);

wherein the first interconnecting node is connected to the second interconnecting node in the closed ring via a set of links, and the first interconnecting node is not connected to the second interconnecting node in the sub-ring; and wherein the set of management packets includes a topology change notification (TCN) packet that is addressed to a node (102C) immediately preceding the second interconnecting node along the set of links.

4. The method of claim 3, wherein the topology information packets include (1) an introduction topology information packet (400) that includes an identifier of the originating node and (2) a neighbor topology information packet (500) that includes an identifier of the originating node and identifiers of neighbor nodes of the originating node.

5. The method of claim 4, wherein determining the topology information includes: populating a first neighbor table (600) based on received topology information packets, wherein the first neighbor table indicates neighbor nodes from the first set of nodes relative to the first interconnecting node; and populating a first network topology table (700) based on received topology information packets for the first set of nodes in the closed ring, wherein the first network topology table includes an entry (708) for each node in the first set of nodes and each entry includes a first identifier field (704) corresponding to a first neighbor node and a second identifier field (706) corresponding to a second neighbor node of an associated node.

6. The method of claim 5, wherein determining the topology information further includes:

populating a second neighbor table based on received topology information packets, wherein the second neighbor table indicates neighbor nodes from the second set of nodes relative to the first interconnecting node; and

populating a second network topology table based on received topology information packets for the second set of nodes in the sub-ring, wherein the second network topology table includes an entry for each node in the second set of nodes and each entry includes a first identifier field corresponding to a first neighbor node and a second identifier field corresponding to a second neighbor node of an associated node.

7. The method of claim 4, wherein the introduction topology information packet and the neighbor topology information packet are automatic protection switching (APS) packets with the same request/state identifier but different sub-codes.

8. A non-transitory machine-readable storage medium (2348) that provides instructions that, if executed by a processor (2342) of an interconnecting node (102), which is shared by a closed ring (1802) and a sub-ring (1804) of a layer 2 network system (1800), will cause said processor to perform operations comprising:

determining (2102) topology information for the closed ring and the sub-ring, wherein the closed ring includes a first set of nodes (102A-102F), including the first interconnecting node, and the sub-ring includes a second set of nodes (102D and 102F-102I), including the first interconnecting node, and the topology information describes links (104_I-104₆) between the first set of nodes and links (104₇-104IO) between the second set of nodes, wherein the topology information is determined based on topology information packets (400, 500) that indicate one or more of (1) identifiers (408A-408B) of an originating node of a corresponding topology information packet and (2) identifiers (502A-502B and 504A-504B) of neighbor nodes to the originating node;

generating (2104) a set of management packets based on the topology information for performing the management function in the layer 2 network system; and transmitting (2106) the set of management packets in the closed ring of the layer 2 network system to perform the management function.

9. The non-transitory machine -readable storage medium of claim 8, wherein the management function controls media access control (MAC) flush operations by the first set of nodes.

10. The non-transitory machine-readable storage medium of claim 9, wherein the first set of nodes and the second set of nodes include a second interconnecting node (102F); wherein the first interconnecting node is connected to the second interconnecting node in the closed ring via a set of links, and the first interconnecting node is not connected to the second interconnecting node in the sub-ring; and wherein the set of management packets includes a topology change notification (TCN) packet that is addressed to a node (102C) immediately preceding the second interconnecting node along the set of links.

11. The non-transitory machine-readable storage medium of claim 10, wherein the topology information packets include (1) an introduction topology information packet (400) that includes an identifier of the originating node and (2) a neighbor topology information packet (500) that includes an identifier of the originating node and identifiers of neighbor nodes of the originating node.

12. The non-transitory machine -readable storage medium of claim 11, wherein determining the topology information includes:

populating a first neighbor table (600) based on received topology information packets, wherein the first neighbor table indicates neighbor nodes from the first set of nodes relative to the first interconnecting node; and populating a first network topology table (700) based on received topology information packets for the first set of nodes in the closed ring, wherein the first network topology table includes an entry (708) for each node in the first set of nodes and each entry includes a first identifier field (704) corresponding to a first neighbor node and a second identifier field (706) corresponding to a second neighbor node of an associated node.

13. The non-transitory machine -readable storage medium of claim 12, wherein determining the topology information further includes:

14. The non-transitory machine-readable storage medium of claim 11, wherein the introduction topology information packet and the neighbor topology information packet are automatic protection switching (APS) packets with the same request/state identifier but different sub-codes.

15. A node (102D) for performing a management function using topology information in a layer 2 network (1800), wherein the node is a first interconnecting node that is shared by a closed ring (1802) and a sub-ring (1804) of the layer 2 network, the node comprising:

a memory unit (2348) that stores instructions; and

a processor (2342) coupled to the memory unit to execute the instructions, wherein the instructions cause the node to:

determine (2102) topology information for the closed ring and the sub-ring, wherein the closed ring includes a first set of nodes (102A-102F), including the first interconnecting node, and the sub-ring includes a second set of nodes (102D and 102F-102I), including the first interconnecting node, and the topology information describes links (1041-1046) between the first set of nodes and links (1047-10410) between the second set of nodes, wherein the topology information is determined based on topology information packets (400, 500) that indicate one or more of (1) identifiers (408A-408B) of an originating node of a corresponding topology information packet and (2) identifiers (502A-502B and 504A-504B) of neighbor nodes to the originating node;

generate (2104) a set of management packets based on the topology information for performing the management function in the layer 2 network; and

transmit (2106) the set of management packets in the closed ring of the layer 2 network to perform the management function.

16. The node of claim 15, wherein the management function controls media access control (MAC) flush operations by the first set of nodes.

17. The node of claim 16, wherein the first set of nodes and the second set of nodes include a second interconnecting node (102F);

18. The node of claim 17, wherein the topology information packets include (1) an introduction topology information packet (400) that includes an identifier of the originating node and (2) a neighbor topology information packet (500) that includes an identifier of the originating node and identifiers of neighbor nodes of the originating node.

19. The node of claim 18, wherein determining the topology information includes:

20. The node of claim 19, wherein determining the topology information further includes: