CN117061425A - Containerized router using virtual networking - Google Patents

Abstract

Embodiments of the present disclosure relate to a containerized router utilizing virtual networking. In general, this disclosure describes techniques for a containerized router operating within a cloud native orchestration framework. In one example, a virtualized cellular site router includes a computing device configured with a containerized router, the computing device comprising: a containerized virtual router configured to execute on the processing circuit and configured to implement a data plane for the containerized router; a containerized routing protocol process configured to execute on the processing circuit and configured to implement a control plane for the containerized router; and a pod comprising a containerized distributed unit, wherein the containerized routing protocol process is configured to advertise routing information comprising reachability information for the containerized distributed unit.

Description

Containerized router using virtual networking

The application is a divisional application of Chinese patent application with international application date of 2022, 2 and 28, national application number of 202280016939.X and the name of "utilizing a virtual networking containerized router".

Cross Reference to Related Applications

This application claims the following benefits:

U.S. patent application Ser. No. 17/649,632, filed on 1/2/2022;

U.S. patent application Ser. No. 17/649,640, filed on 1/2/2022;

U.S. patent application Ser. No. 17/649,643, filed on 1/2/2022;

U.S. provisional application No. 63/242,434, filed on 9/9 at 2021;

indian application No. 202141008548, filed on 1/3/2021;

the entire contents of each application are incorporated herein by reference.

Technical Field

The present disclosure relates to virtualized computing infrastructure, and more particularly to containerized routers.

Background

In a traditional cloud data center environment, there are a large number of interconnected servers that provide computing and/or storage capabilities to run various applications. For example, a data center may include facilities that host applications and services for subscribers (i.e., customers of the data center). For example, a data center may host all infrastructure equipment, such as network and storage systems, redundant power supplies, and environmental controls. In a conventional data center, clusters of storage systems and application servers are interconnected via a high-speed switching fabric provided by one or more layers of physical network switches and routers. More complex data centers provide infrastructure throughout the world, as well as subscriber support equipment located in various physical hosting facilities.

Virtualized data centers are becoming the core foundation of modern Information Technology (IT) infrastructure. In particular, modern data centers have widely utilized virtualized environments in which virtual hosts (also referred to herein as virtual execution elements, such as virtual machines or containers) are deployed and executed on the underlying computing platform of a physical computing device.

Virtualization within a data center or any environment that includes one or more servers may provide several advantages. One advantage is that virtualization can provide a significant improvement in efficiency. With the advent of multi-core microprocessor architectures with a large number of cores per physical CPU, the underlying physical computing devices (i.e., servers) have become more powerful and virtualization has become easier and more efficient. A second advantage is that virtualization provides important control over the computing infrastructure. As physical computing resources become alternative resources (such as in a cloud-based computing environment), provisioning and management of computing infrastructure becomes easier. Thus, enterprise IT personnel often prefer virtualized computing clusters in a data center because of management advantages in addition to the efficiency and higher Return On Investment (ROI) provided by virtualization.

Containerization is a virtualization scheme based on operating system level virtualization. The containers are lightweight and portable execution elements for applications that are isolated from each other and from the host. Because the container is not tightly coupled with the host hardware computing environment, the application may be bound to the container image and executed as a single lightweight package on any host or virtual host that supports the underlying container architecture. Thus, the container solves the problem of how to make software work in different computing environments. Containers provide promises to run consistently (virtually or physically) from one computing environment to another.

Because of the inherently lightweight nature of containers, a single host may typically support more container instances than a traditional Virtual Machine (VM). Containers are typically ephemeral, can be created and moved more efficiently than VMs, and they can also be managed as logically related element groups (for some orchestration platforms, such as Kubernetes, sometimes referred to as "pods"). These container characteristics affect the requirements of the container networking solution: the network should be flexible and scalable. The VM, container, and bare metal server may need to coexist in the same computing environment where communication is enabled between different application deployments. The container network should also be unaware of working with the various types of orchestration platforms used to deploy the containerized applications.

The computing infrastructure that manages deployment and infrastructure for application execution may involve two main roles: (1) Orchestration-automatic deployment, extension, and application operations for across host clusters, and provision of computing infrastructure, which may include container-centric computing infrastructure; and (2) network management-creating a virtual network in a network infrastructure to enable packetized communications between applications running on a virtual execution environment (such as a container or VM) and between applications running on a traditional (e.g., physical) environment. The software-defined network facilitates network management.

Disclosure of Invention

In general, this disclosure describes techniques for a containerized router operating within a cloud native orchestration framework. In at least some aspects, the techniques may involve deploying a logically related set of one or more containers ("pods") that support a Data Plane Development Kit (DPDK) to support fast path packet communications on a data channel between a virtual router and the pod. A container networking interface plug-in (CNI) is a networking solution to an application container and is a runtime executable that can help configure the interface between the container and other components of a computing device ("host") hosting the container, which can be a component of the pod. The computing device may alternatively be referred to as a "computing node" or "server. CNIs typically assign a network address (e.g., IP address) to a network interface and may also add routes related to the interface, such as routes for a default gateway and one or more name servers.

In one aspect of the disclosure, techniques are described for a virtualized cell site router having a containerized application for implementing Distributed Units (DUs) on computing nodes. In at least some cases, the compute nodes include DPDK based virtual routers for the data plane.

In one aspect of the disclosure, a containerized routing protocol daemon (cRPD) is a routing protocol process that is packaged as a container to run in a Linux-based environment. cRPD may be executed as a containerization process in the user space of the host. Accordingly, cRPD provides a rich family of physical router routing software on Linux-based computing nodes. cRPD provides control plane functionality. Existing implementations of cRPD (running on a host) use forwarding provided by the Linux kernel. Thus, the control plane is containerized.

A virtual router is a software entity that provides data plane functionality on a compute node. The compute nodes may host VMs or containers that are centrally orchestrated and provisioned. The virtual router may work with the SDN controller to create an overlay network by exchanging routes, configurations, and other data. The virtual router may operate as a Linux kernel module or a DPDK based process. DPDK allows virtual routers to process more packets per second than when running as a kernel module. The virtual router data plane may be containerized. In combination, the containerized cRPD and containerized DPDK-based virtual router may thus be a fully functional containerized router.

The computing nodes may be used to implement portions of a (5 th generation) cellular network using a cloud-originated, open radio access network ("O-RAN" or "open RAN") architecture. The cloud can be built using containers and Kubernetes. The cellular site router function may be implemented on a computing node hosting a Distributed Unit (DU) 5G function as a containerized application. That is, the DU function may be implemented as a Kubernetes pod on these computing nodes. At a very high level, the DU function will consume RAN traffic, process it and then tunnel it to the data center hosted control unit function (CU).

To meet the rich routing function and forwarding performance requirements for this 5G network use case, the compute node may be configured to use an integration scheme in which cRPD running on the compute node operates as a control plane and configures a DPDK-based virtual router as a corresponding fast path forwarding plane for mobile network traffic to be handled by the containerized DU.

In an aspect of the disclosure, the generic data plane model is decoupled from a network controller of the virtualized computing infrastructure. For example, a data plane in accordance with this aspect may expose an Application Programming Interface (API) that may be implemented by any control plane service. In some examples, the data plane can also be used with multiple types of CNIs. The data plane may be implemented using a DPDK-based virtual router and exposes a remote procedure call (e.g., gRPC) interface for exchanging control data. For example, a virtual router agent for a virtual router data plane may operate as a gRPC server exposing gRPC APIs for programming the virtual router data plane. These techniques include a workflow for configuring a virtual network interface for a pod, where a virtual router agent obtains information from a container routing protocol daemon (cRPD) in response to a port request issued by a CNI.

In one aspect of the disclosure, a containerized routing protocol daemon (cRPD) interfaces with two disjoint data planes: a kernel networking stack for the compute nodes and a DPDK based virtual router. The cRPD may leverage the networking stack of the kernel to exclusively route the DPDK fast path. The routing information received by cRPD may include underlying routing information and overlay routing information. The cRPD may run routing protocols on a vpest interface visible in the kernel, and may install Forwarding Information Base (FIB) updates in the kernel FIB corresponding to Interior Gateway Protocol (IGP) learned routes (underlayers) (e.g., to support establishment of multi-hop Interior Border Gateway Protocol (iBGP) sessions to these destinations). Meanwhile, the DPDK-based virtual router may notify the cRPD of the application pod interface created by the CNI for the compute node. Such pod interfaces cannot publish or otherwise inform the kernel. cRPD may advertise reachability to these pod interfaces to the rest of the network, e.g., L3VPN Network Layer Reachability Information (NLRI). Corresponding multiprotocol label switching (MPLS) routes are programmable on virtual routers but cannot be programmed to the core, because the next hops of these labels are "POP and forward" operations to pod interfaces, and these interfaces are only visible in the virtual router. Similarly, reachability information received over BGP L3VPN can only be programmed to virtual routers, as only pods need such reachability information for forwarding. That is, the kernel may not have any use or application that requires such reachability information.

In an aspect of the disclosure, to provide high availability of network connectivity, the CNI may also add a second backup interface to the application pod when adding the DPDK interface to the application pod instantiated on the compute node. The backup interface may be configured on a backup data plane within the computing node that is different from the active data plane on which the active interface is configured. For example, the active data plane may be a DPDK-based virtual router, while the backup data plane may be a kernel-based virtual router.

In an aspect of the disclosure, the set of software components provides CNI functionality that addresses networking requirements specific to the cloud native 5G network environment. The software components include a containerized routing protocol daemon (cRPD) to support network services grid (NSM) architecture. The set of software components support the NSM architecture and may provide additional functionality such as hybrid networking (between physical and virtual infrastructure), reachability from outside the computing node cluster to pods to dynamically establish tunnels (e.g., advertising over a protocol such as BGP) using various technologies such as MPLS, SRv6, IP-IP/VxLAN/GRE, IPsec, etc. In use cases of this aspect, a 5G O-RAN network may be deployed using cloud-native technology and follow a 5G split, where DUs (distributed units) and CSRs (cell site routers) are virtualized and run on compute nodes. The set of software components may operate as a cell site router to provide L3 reachability for a mid-roll for a 5G network.

The software component uses cRPD to distribute layer three (L3) network reachability information for pods not only inside the cluster but also outside the cluster. cRPD also programs the data plane on each compute node. For better network packet I/O performance, DU applications may run in the application pod to bypass the kernel networking stack and abstraction, and thus use, for example, a zero copy mechanism to send/receive packets directly from the physical NIC. A Data Plane Development Kit (DPDK) is one such framework and DPDK-based virtual routers can be used as a user space data plane that leverages the DPDK for high forwarding performance for this purpose.

The software components may include virtual routers based on DPDK to support DPDK applications. The CNI plug-in manages DPDK configuration for the application and programs the virtual router. This may include setting up a vhost control channel and assigning IP (e.g., both IPv4 and IPv 6) and MAC addresses, advertising pod IP addresses, and detecting and withdrawing routes when a pod is considered closed or deleted.

The various aspects described in this disclosure may be used together in any combination of these aspects. "DAY ONE CONTRAIL DPDK vROUTER",2021, kiran KN et al, juniper Networks, inc. "DAY ONE CLOUD NATIVE ROUTING WITH cRPD",2021, hitesh Mali et al, juniper Networks, inc., incorporated herein by reference in its entirety.

In one example, a computing device is configured with a containerized router, the computing device comprising: a processing circuit; a containerized virtual router configured to execute on the processing circuit and configured to implement a data plane for the containerized router; and a containerized routing protocol process configured to execute on the processing circuit and configured to implement a control plane for the containerized router.

In one example, a 5G wireless access network includes: a medium-pass network connecting the distributed units of the gNodeB to the centralized units of the gNodeB; and a computing device, the computing device comprising: a processing circuit; a containerized routing protocol process configured to execute on the processing circuit and configured to implement a control plane for the containerized router; a pod comprising one of a containerized distributed unit or a containerized centralized unit, wherein the containerized routing protocol process is configured to advertise routing information via the intermediate transmission network, the routing information comprising reachability information of the one of the containerized distributed unit or the containerized centralized unit; and a containerized virtual router configured to execute on the processing circuit and configured to implement a data plane for the containerized router to forward traffic received at the containerized virtual router to one of the containerized distributed unit or the containerized centralized unit.

In one example, a virtualized cellular site router includes a computing device configured with a containerized router, the computing device comprising: a containerized virtual router configured to execute on the processing circuit and configured to implement a data plane for the containerized router; a containerized routing protocol process configured to execute on the processing circuit and configured to implement a control plane for the containerized router; and a pod comprising a containerized distributed unit, wherein the containerized routing protocol process is configured to advertise routing information comprising reachability information for the containerized distributed unit.

The details of one or more examples of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims.

Drawings

Fig. 1 is a block diagram illustrating an example mobile network system in accordance with the techniques described in this disclosure.

Fig. 2 is a block diagram illustrating an example implementation of a portion of the mobile network system of fig. 1 in greater detail in accordance with the techniques of this disclosure.

Fig. 3A-3B are block diagrams illustrating example examples of servers implementing virtualized cell site routers in accordance with the techniques of this disclosure.

Fig. 4 is a block diagram illustrating an example server in accordance with the techniques of this disclosure.

Fig. 5 is a block diagram illustrating an example virtual router agent in accordance with the techniques of this disclosure.

Fig. 6 is a block diagram illustrating an example server having example control and data traffic flows within the server in accordance with the techniques of this disclosure.

Fig. 7 is a conceptual diagram depicting a sequence of operations added to a port that results in route programming in a virtual router, according to example aspects of the present disclosure.

Fig. 8 is a block diagram of an example computing device (e.g., host) in accordance with the techniques described in this disclosure.

FIG. 9 is a block diagram of an example computing device operating as an instance of an orchestrator master node of a cluster of virtualized computing infrastructure according to the techniques described in this disclosure.

FIG. 10 is a block diagram illustrating an example implementation of an example containerized routing protocol daemon that may be deployed by an orchestrator using pods according to the techniques described in this disclosure.

Fig. 11 is a block diagram illustrating example components of a virtual router agent and an example sequence of operations and messages for creating and advertising new ports for pods in accordance with the techniques of this disclosure.

Fig. 12 illustrates an example system and packet forwarding in accordance with the techniques described in this disclosure.

Fig. 13 illustrates an example system and packet forwarding in accordance with the techniques described in this disclosure.

Fig. 14 is a conceptual diagram illustrating example operations for programming virtual router forwarding information in accordance with the techniques of this disclosure.

Fig. 15 is a conceptual diagram illustrating example operations for configuring and advertising virtual network interfaces in a server with cloud native routers, in accordance with the techniques of this disclosure.

Like reference numerals refer to like elements throughout the description and drawings.

Detailed Description

5G uses a cloud-native approach, where functional blocks are broken down into micro-services. Micro services are deployed as containers on an x86 platform, orchestrated by Kubernetes (abbreviated "K8 s"). This includes 5G core control plane functions such as access and mobility management functions (AMF) and Session Management Functions (SMF), RAN control plane functions such as CU-CP, service Management and Orchestration (SMO), near real-time and non-real-time Radio Intelligent Controllers (RIC), and even some data plane functions such as CU-DP and DUs.

The K8s network between pods is implemented via a plug-in called a Container Networking Interface (CNI) (also called a container networking interface plug-in). However, when the containerized network functions served by the CNI play a critical role within the telecommunications network, the network capabilities of conventional CNIs are quite primitive and unsuitable. As described herein, cloud native routers are better suited for these situations. Cloud native routers are a containerized router that allows x86 or ARM based hosts to be a member of a network routing system, participating in protocols such as intermediate system-to-intermediate system (IS-IS) and Border Gateway Protocol (BGP) and providing multiprotocol label switching/segment routing (MPLS/SR) based transport and multi-tenant. In other words, the platform is not an adjunct to the network (e.g., a Customer Edge (CE) router), but rather may operate as a Provider Edge (PE) router.

Cloud native routers may have one or more advantages over conventional routers. The router has a control plane and a forwarding plane. The control plane participates in the dynamic routing protocol and exchanges routing information with other routers in the network. It downloads the result into the forwarding plane in the form of a prefix, a next hop and an associated SR/MPLS label. The specific implementations described herein are modular in the sense that the control plane does not know the exact details of how the forwarding plane is implemented. In a hardware router, the forwarding plane may be based on a custom ASIC. In contrast, cloud native routers are virtualized routers. However, the routing protocol software in both cases is functionally similar. This means that cloud-native routers benefit from the same highly comprehensive and robust protocol implementation as hardware-based routers supporting some of the largest networks of the world.

Yun Yuansheng routers use a containerized routing protocol daemon (cRPD) control plane and a virtual router (vruter) forwarding plane to deliver high performance networks in small space consuming software packages that function similarly to non-virtual routers, physical Network Functions (PNFs). The forwarding plane may be implemented via selection of a DPDK, linux kernel, or smart NIC. The complete integration delivers packets that conform to the K8s CNI, which can be deployed in the K8s environment (e.g., support Multus).

The cloud native router may be incorporated into the host in which it is located and integrated with the K8 s. By way of example, this disclosure describes how DUs and cloud native routers can coexist on an x86 or ARM based host or other computing device of the same 1U size. This is particularly attractive for those cellular sites where power and space are limited, as it avoids the need for a two-box solution in the form of separate DUs and routers. Multiple O-DUs or other workloads may be attached to the same cloud-native router.

The cell site server may be a K8s operational node (or "slave"). The O-DU pod is connected into the cloud native router. The O-DUs may require multiple network interfaces, in some cases facilitated by Multus meta CNI. Each of these interfaces may be mapped to a different third layer VPN on the cloud native router to support multiple network slices. The CNI described herein dynamically adds or deletes interfaces between pods and virtual router containers when triggered by a K8s pod event. It also dynamically updates the cRPD control plane container in the form of route specifiers and route targets with host routes and corresponding third layer VPN mappings for each pod interface. The third layer VPN may be implemented using virtual routing and forwarding instances (VRFs). Further, the cRPD control plane programs the virtual router forwarding plane accordingly via the gRPC interface. In this way, the cloud native router is introduced into the data path, supporting the F1 interface with the CU running in the edge or regional DC site. Although primarily described with respect to O-RAN applications (such as distributed units), cloud native router technology is applicable to configuring host-based virtual routers for other containerized applications.

Since CNR itself is a cloud native application, it supports installation using K8 smanfest or Helm hart. These include initial configuration of routers, including routing protocols and third layer VPNs to support slicing. CNRs can be orchestrated and configured within a few seconds, with all routing protocols contiguous with the rest of the network startup and operation. During the life cycle of the CNR, a persistent configuration change (e.g., adding or deleting a network slice) may be implemented via the selection of CLI, K8s manifest, netConf, or Terraform.

By employing the K8s CNI framework, cloud native routers can mitigate the traditional operational overhead that arises when using containerized devices rather than their physical devices. By disclosing an appropriate device interface, the cloud native router may normalize the operational model of the virtual device to a physical device, thereby eliminating the obstruction of adoption within the carrier network operating environment. The cloud native router may present a familiar routing device look and feel to any trained operating team. The cloud native router has similar characteristics and functions as a hardware-based platform and a similar operational model. Likewise, the domain controller may use protocols used with any other Junos router to communicate with and control cloud native routers, such as Netconf/OpenConfig, gRPC, path Computation Element Protocol (PCEP), and programmable routing daemon (pRPD) APIs.

Nodes executing cloud-native routers may participate in IS-IS, open Shortest Path First (OSPF), BGP, and/or other internal or external routing protocols. In addition, MPLS may be generally used based on Segment Routing (SR). There are two reasons for doing so: allow traffic engineering when needed, and support multi-tenancy by using MPLS-based third layer VPNs. As an alternative SRv6 could alternatively be used to meet these requirements. Having full routing capability is also necessary to implement network slicing. Each slice tenant is placed in its own third layer VPN. From the third layer VPN perspective, the cloud native router acts as a Provider Edge (PE) router. Thus, the cloud native router exchanges layer three VPN prefixes with other PE routers in the network via BGP, whether these other PEs are physical routers or cloud native routers residing on other hosts. Each tenant may be placed in a separate VRF table on each PE, providing the correct degree of isolation and security between tenants, just as with conventional third-tier VPN services. This skillfully solves the problem that K8s itself does not provide such isolation. The third layer VPN is a validated method for implementing multi-tenants in a network and is trusted by many large companies that purchase this service from its network service provider globally.

Typically, a transmission network provides multiple paths, each path being tuned for a particular cost function (such as minimum delay or high bandwidth). These are implemented using segment-routing resiliency algorithms or RSVP or segment-routing based traffic engineering. When traffic engineering is used, the path may be computed by the controller and transmitted to the cloud primary router via the PCEP protocol. When the controller detects network congestion via streaming telemetry, it automatically recalculates the affected paths to relieve the congestion. PE routers (including cloud-primary routers) apply labels (BGP color communities) to prefixes in a given VRF according to the type of path required for the corresponding slice. For example, the first slice may require the lowest latency transmission possible and is therefore mapped to a low latency path in order to reach an O-CU in an Edge Data Center (EDC). The second slice requires a high bandwidth with a fairly low delay. So its O-CU is also located in the EDC and the traffic is mapped to a high bandwidth path to the EDC. The third slice requires high bandwidth transmission but is not delay sensitive, so its O-CU can be placed in a Regional Data Center (RDC). The traffic of the third slice is mapped to a high bandwidth path to the RDC. In an actual deployment where there will be more slices, the mapping of slices to transmission paths will typically be many-to-one. For example, all slices that require low-delay transmissions between a given endpoint pair share the same low-delay traffic engineering or elastic algorithm path that connects the two endpoints.

In summary, cloud-native routers can bring about an omnidirectional routing function to a computing platform hosting a containerized network function. This may allow the platform to fully participate in the operator's network routing system and facilitate multi-tenant and network slicing. It may provide the same familiar lookup and release, operational experience, and control plane interfaces as a hardware-based router.

Fig. 1 is a block diagram illustrating an example mobile network system in accordance with the techniques described in this disclosure. The mobile network system 100 may be a 5G network implementing 5G standards promulgated by, for example, the 3 rd generation partnership project (3 GPP), the open radio access network ("O-RAN" or "ora") alliance, the European Telecommunications Standards Institute (ETSI), the Internet Engineering Task Force (IETF), and the International Telecommunications Union (ITU).

The 5G network decomposes mobile fronthaul and mid-haul networks by building around cloud primitive principles. Thus, the service provider may avoid being locked to a particular device provider and may build and provision the mobile network system in conjunction with efficient solutions from different providers at different layers and locations. This may improve Radio Access Networks (RANs), in particular making them more open, resilient and scalable.

The O-RAN based network breaks down the baseband unit (BBU) found in conventional telecommunications networks into three functional units: a Radio Unit (RU), a Distributed Unit (DU) and a Centralized Unit (CU). The different functions of RU, DU and CU may be implemented by software executed by an x 86-based or ARM-based host server. The CU may be further separated into different control plane (CU-CP) and user plane (CU-UP) functions to further implement control and user plane separation (cpu). This decoupling helps to provide flexibility for deployment-different combinations of RUs, DUs, and CUs may be deployed at the same location, or may be deployed at different locations. For example, RU, DU and CU may be placed together at the edge where delay is critical. The O-RAN compliant DUs and CUs are commonly referred to as O-DUs and O-CUs, respectively. An additional data plane element, called User Plane Function (UPF), operates in the mobile core network 7 to forward traffic between the CUs and the data network 15. The additional control plane elements operate in the mobile core network 7. These control plane elements include Network Slice Selection Functions (NSSF), policy Control Functions (PCF), authentication server functions (ASUF), access and mobility management functions (AMF), network Exposure Functions (NEF), network function repository functions (NRF), application Functions (AF), unified Data Management (UDM), and Session Management Functions (SMF).

The mobile network system 100 comprises a radio access network 9 and a mobile core network 7. The radio access network 9 includes RUs 14 located at respective cellular network sites ("cell sites"). Each RU 14 consists of an LO PHY and an RF transmitter. The LO PHY component may be implemented using dedicated hardware for high performance packet processing.

RU 14 is connected to DUs 22A-22X (collectively, "DUs 22") via a forwarding network. The forwarding network connects LO PHY and hipery and is used by RU 14 and DU 22 to implement the F2 interface of 5G. The DU 22 manages the packet transmission of the radio by the RU 14. In some cases, such packet transmissions conform to the Common Packet Radio Interface (CPRI) and/or enhanced CPRI (eCPRI) standards, or IEEE 1914.3. The DU 22 may implement Radio Link Control (RLC), medium Access Control (MAC), and HIPHY layers. DU 22 is controlled, at least in part, by CUs 13A-13B (collectively, "CUs 13").

The DU 22 is connected to the CU 13 via an intermediate network, which may be used by the DU 22 and the CU 13 to implement F1 of 5G. CU 13 may implement a Radio Resource Control (RRC) layer and a Packet Data Convergence Protocol (PDCP) layer. The CU 13 is connected to the mobile core network 7 via a backhaul network. The intermediate and backhaul networks may each be a Wide Area Network (WAN).

In the radio access network 9 of the mobile network system 100, the gNodeB includes one of the CUs 13 and one of the DUs 22. A CU may support multiple DUs to implement multiple gnodebs. And a single DU may support one or more RUs. Thus, for example, with respect to fig. 1, one RU 14 of CUs 13A and DU22A and RU 14 may form one eNodeB, while another RU 14 of CUs 13A and DU 22B and RU 14 (of server 12B) may form another eNodeB.

As shown in fig. 1, any of DUs 22 may or may not be located at a cellular site that includes RUs 14 supported by the DUs. DU 22X is located at a cellular site, while DUs 22A-22N are located at a local data center and collectively support multiple RUs 14. The mobile network system 100 may have a radio access network 9 that includes thousands of cell sites, each having one or more RUs 14 and optionally one or more DUs 22. Whether located at a cellular site or off-site, DUs are typically located within 20km of the RU being supported. CU 13 is shown in fig. 1 as being located at a regional data center, typically within 40km of the supported DU 22.

The radio access network 9 is connected to the mobile core network 7 for exchanging packets with the data network 15. The mobile core network 7 may be a 5G core network and the Data Network (DN) 15 may represent, for example, one or more service provider networks and services, the internet, a 3 rd party service, an IP multimedia subsystem or other networks.

The mobile network system 100 includes a plurality of servers 12A-12X to execute DUs 22. Each of the servers 12 may be real or virtual servers hosting/executing software implementing the DUs 22. Such software may include one or more applications deployed to the server 12 as, for example, virtual machines or containers. Although not shown in fig. 1, CU 13 may also be executed by a server.

The combination of DU 22, the intermediate network, CU 13 and the backhaul network effectively implements an IP-based transport network between the radio unit 14 and the mobile core network 7.

In accordance with techniques of one or more aspects of the present disclosure, virtualized cell site routers 24A-24X ("vCSR 20A-20X" and collectively "vCSR 20") provide a third layer routing function between DU 22 and CU 13. These vCSR 24 may execute on the same server 12 as one or more DUs 22 to provide provider edge router functionality to such DUs 22. Although each vCSR 20 is referred to as a "cell site" router, any vCSR 20 may be deployed to a local data center along with one or more DUs 22 for which the vCSR provides IP services, as shown with respect to the vcsrs 20A-20N, i.e., where the local data center includes a server 12 executing DUs 22 for one or more cell sites.

Each of the vcsrs 20 is implemented using one of the containerized routing protocol daemons 20A-20X ("cRPD 24A-24X" and collectively "cRPD 24"). More specifically, each of vCSR 20 uses a corresponding one of cRPD 24 as a control plane for implementing the third tier routers. cRPD provides control plane routing functions. For example, cRPD may execute IP (IPv 4/IPv 6) underlying routing protocols such as intermediate system-intermediate system (IS-IS) and Border Gateway Protocol (BGP); advertising reachability of DUs 22 inside and outside the group, e.g. to CU 13; implementing a network namespace (supported using L3VPN and EVPN type 5 advertising); implementing Access Control Lists (ACLs) and network policies for security, network isolation, and quality of service (QoS); supporting tunnels and tunneling protocols (e.g., MPLS, SR-MPLS, SRv6, SR-MPLS IPv4, vxLAN, IP-in-IP, GRE); support dynamic tunneling signaled using BGP; supporting encryption of IPSec tunnels; and programming the forwarding plane of the server's vCSR with learned and/or configured routing information to provide third-layer packet forwarding, encapsulation, packet filtering, and/or QoS between one or more of the DUs 22 and one of the CUs 13.

For example, the vCSR 20A executed by the server 12A includes cRPD 24A and a forwarding plane of the server 12A (e.g., smartNIC, kernel-based forwarding plane, or Data Plane Development Kit (DPDK) -based forwarding plane). The cRPD 24A provides one or more of the above-described routing functions to program the forwarding plane of the vCSR 20A to, among other tasks, advertise layer three routes for DUs 22A outside the cluster, including traversing the network to the CU 13A, and forward layer three packets between the DUs 22A and the CU 13A. In this way, the technique implements a cloud native containerized cell site router 20 executing on the same server 12 as the containerized DU 22, thereby significantly reducing the intermediate propagation delay between the DU 22 and CU 13.

As a containerized router, vCSR 20 allows x 86-based or ARM-based hosts to become a member of a stream of network routing systems, thereby participating in protocols such as IS-IS and BGP and providing MPLS/SR-based transport and multi-tenant. Thus, the vCSR 20 may operate as a Provider Edge (PE) router for a network that transmits layer three packets between the DUs 22, CUs 13, and the mobile core network 7, rather than as an adjunct to the network (similar to a Customer Edge (CE) router).

Furthermore, in some examples, integration of cRPD 24 and a host-based forwarding plane may also deliver Kubernetes CNI-compliant packets that may be deployed within a Kubernetes environment. The single server of the DU 22 and the co-execution of the vCSR 20 may avoid a two-box solution with separate DUs and routers, potentially reducing cost, power and space requirements, which is particularly attractive for cellular sites. The application workload may be a Containerized Network Function (CNF), such as a DU.

The orchestrator 50 represents a container orchestration platform. In the context of virtualized computing infrastructure, "orchestration" generally refers to provisioning, scheduling, and managing virtual execution elements and/or applications and services executing on such virtual execution elements by a host server available to an orchestration platform. In particular, container orchestration allows containers to coordinate and refers to the deployment, management, expansion, and configuration of containers to host servers, for example, by a container orchestration platform. Examples of orchestration platforms include Kubernetes, docker swarm, meso/Marathon, openShift, openStack, VMware, and Amazon ECS. Scheduler 50 schedules at least the containerized RPD 24 of DUs 22 and vCSR 20. In some examples, the data plane of vCSR 20 is also containerized and orchestrated by orchestrator 50. For example, the data plane may be a virtual router based on DPDK.

The container (including the container implementing the containerized routing protocol daemon 24) may be deployed to the virtualized environment using a cluster-based framework in which a cluster master node of a cluster manages the deployment and operation of the container to one or more cluster slave nodes of the cluster. The terms "master node" and "slave node" as used herein encompass different orchestration platform terms of similar devices that distinguish between primary management elements of a cluster and primary virtual execution element hosting devices of the cluster. For example, the Kubernetes platform uses the terms "cluster master node" and "slave node", while the Docker switch platform refers to a cluster manager and cluster nodes. The server 12 or virtual machine thereon may represent a cluster node.

Orchestrator 50 and Software Defined Network (SDN) controller 70 may execute on separate computing devices or on the same computing device. Each of orchestrator 50 and SDN controller 70 may be a distributed application executing on one or more computing devices. Orchestrator 50 and SDN controller 70 may implement respective master nodes of one or more clusters, each cluster having one or more slave nodes implemented by respective servers 12. In general, SDN controller 70 controls the network configuration of radio access network 9 to facilitate packetized communications between DUs 22, CUs 13 and mobile core network 7. SDN controller 70 may distribute routing and configuration information to control plane elements of radio access network 9, particularly to cRPD 24. For example, SDN controller 70 may program a segment routing header, configure an L3VPN, configure a VRF in a router of radio access network 9 (including virtualized cell site router 20). SDN controller 70 may implement one or more southbound protocols for configuring routers, switches, and other network devices of the mid-pass and backhaul networks, as well as for configuring vCSR 20. Example southbound protocols may include Path Computation Element Protocol (PCEP), BGP, netconf, openConfig, another protocol for configuring cRPD 24, and so on. Additional information about L3 VPNs can be found in "BGP/MPLS IP Virtual Private Networks (VPNs)", solicitation opinion 4364, network work group of internet engineering task force, 2006, incorporated by reference in its entirety.

SDN controller 70 may provide a logically and in some cases physically centralized controller. In some examples, SDN controller 70 may operate in response to configuration inputs received from orchestrator 50 and/or an administrator/operator. SDN controller 70 may program NFV infrastructure (NFVI) such as server 12, network switches/routers, and/or other network infrastructure. With NFVI programming, SDN controller 70 may configure aspects of the operating system kernel to configure L3 IP routes, linux bridges, iptables, network namespaces, and/or virtual switches.

Additional information for the example SDN controller 70, virtual router and virtual router agent may be found in: international application number PCT/US2013/044378, filed on 5 th 6 th 2013, and entitled "PHYSICAL PATH DETERMINATION FOR VIRTUAL NETWORK PACKET FLOWS"; U.S. patent application Ser. No. 14/226,509, filed on day 26 of 3.2014, and entitled "Tunneled Packet Aggregation for Virtual Network"; and U.S. patent application Ser. No. 17/305,110, filed on day 30 of 6 at 2021, and entitled "Network Controller Horizontal Scaling for Network Device Configurations Session Management"; each incorporated by reference as if fully set forth herein.

In general, orchestrator 50 controls the deployment, expansion, and operation of containers across the cluster of servers 12, as well as the provision of computing infrastructure, which may include container-centric computing infrastructure. Scheduler 50 and in some cases network controller 70 may implement respective cluster masters for one or more Kubernetes clusters. For example, kubernetes is a container management platform that provides portability across public and private clouds, each of which may provide a virtualization infrastructure to the container management platform.

Virtualized cellular site router 20 may provide one or more technical advantages to implement at least one practical application. Existing mobile networks use physical cell site routers located on or near each BBU. Physical routers typically have specialized physical dimensions, are relatively difficult to update and configure, and are relatively difficult to replace due to vendor locking effects. While these effects are tolerable in relatively few cell sites (e.g., 3G and 4G/LTE mobile networks), the relatively large number of cell sites required by the RAN of a 5G mobile network exacerbates the capital and operating costs associated with these effects. Although 5G network providers are turning to split RAN architectures (e.g., O-RANs), such networks still rely on physical cell site routers or virtual machine-based routers to manage routing and data traffic between DUs and CUs through the intermediate transmission network.

The virtualized cell site router 20 with the containerized routing protocol daemon 24 mitigates many of the negative effects of deploying physical or VM-based routers at the cell site. For example, the containerized RPD 24 is lighter in computing resources (CPU, memory) and may be more efficient in space and power utilization than VM-based and physical routers, as compared to VM-based routers. Virtualizing the CSR 20 may achieve these advantages while achieving comparable performance, wherein a DPDK-based virtual router is used as a data plane to provide the vCSR 20 with an efficient and high packet I/O rate to communicate with the DU 22. That is, having the vCSR 20A and the DU 22A (e.g., O-DU) on a single physical server 12A with a DPDK-based data plane may provide comparable packet forwarding performance to a physical cell site router. As a further example of technical advantages, the vCSR 20 eliminates the need for physical cell site routers and may reduce space, reduce power consumption, and also reduce capital/operating expenditure. Additionally, in some examples and as described in further detail below, vCSR 20 may be integrated into the Kubernetes infrastructure by presenting vCSR 20 as a Container Networking Interface (CNI) with an orchestration platform, which may be used to configure a network for application workloads. Thus, by deploying containerized vCSR 20/RPD 24 to serve as a CNI, integrated into mobile network system 100, and integrated into Kubernetes, techniques may facilitate a cloud-native experience of vCSR 20 deployment and configuration. Integration in Kubernetes allows the use of its existing mechanisms to monitor the health of the containerized RPD 24 and restart if necessary, while managing the lifecycle of the vCSR 20 (and in particular the containerized RPD 24).

Fig. 2 is a block diagram illustrating an example implementation of a portion of the mobile network system of fig. 1 in greater detail in accordance with the techniques of this disclosure. System 200 includes CUs 213A-213K, each of CUs 213A-213K may represent any of CUs 13. In this example, multiple network slices (e.g., 5G network slices) are implemented using L3 VPNs and tunnels 231A-231K to connect DU 22A to different CUs 213A-213K for the respective network slices.

One of the major technical challenges facing service providers today is the ability to provide various network performance features that will be required for future services. Bandwidth, delay, packet loss, security, and reliability will vary from service to service, to name a few. Emerging applications such as robotic teleoperation, large-scale internet of things (IOT) and autopilot require connectivity, but the characteristics are quite different. The combination of architecture flexibility, software programmability, and the need for different business fields (medical, factory, military, public safety, etc.) and applications has led to the creation of network slice concepts. Network slicing provides a method of fully segmenting a mobile network to support a specific type of service or traffic or even to support a host service provider (multi-tenant) that does not own a physical network. Furthermore, each slice may be optimized for capacity, coverage, connectivity, security, and performance characteristics. Since the slices can be isolated from each other as if they were physically separated on the control plane and user plane, the user experience of a network slice will be the same as if it were a separate network. Network slices may span all domains of the network, including software applications (memory and processing) running on the network nodes, specific configuration of the core transport network, access network configuration, and terminal devices. Network slicing enables multiple operators to securely share a mobile network, but separate their own users from other users, and different applications of the users may use different network slices that provide distinct performance characteristics.

Virtualized cellular site router 20A includes virtual router forwarding plane (virtual router) 206A configured with VRFs 212A-212K (collectively, "VRFs 212") for respective network slices implemented by respective L3 VPNs, with VRFs 212A-212K implemented by vCSR 20A and routers 204A-204B using tunnels 231A-231K connecting VRFs 212 to VRFs 210A-210K on routers 204A-204B. Each of tunnels 231A-231K may represent SR-MPLSoIPv6 or other types of tunnels described above. Each of routers 204A-204K may be a gateway router for a data center having one or more servers to execute any one or more of CUs 213A-213K. The data center may include a data center layout to exchange mobile data traffic between routers and CUs. In some cases, one or more servers of the data center may also execute a UPF for the mobile network, in which case the data center layout may also exchange mobile data traffic between the CU and the UPF.

Each of VRFs 212A-212K has a corresponding virtual network interface with DU 22A. Thus, each of the virtual network interfaces of the DU 22A may be mapped to a different L3VPN in the vCSR 20A to support, for example, a different network slice of the plurality of network slices. As described in further detail below, the CNI of server 12A dynamically adds or deletes a virtual network interface between the pod (where DU 22A is deployed) and virtual router 206A, which may also be deployed as a container in some examples, when triggered by a pod event from orchestrator 50. The CNI also dynamically updates cRPD 24A (the control plane of vCSR 20A) in the form of route specifiers and route targets using host routes for each DU 22A/pod virtual network interface and corresponding third layer VPN mappings. Further, cRPD 24A programs virtual router 206A (the data plane of vCSR 20A) accordingly, optionally using the gRPC interface. In this way, vCSR 20A is introduced into the datapath as a cloud native router, e.g., to support an F1 interface with CUs 213A-213K, which may be executing in an edge or regional datacenter site, for example. In various examples, virtual router 206A may represent a SmartNIC-based virtual router, a kernel-based virtual router, or a DPDK-based virtual router.

Fig. 3A-3B are block diagrams illustrating example examples of servers implementing virtualized cell site routers in accordance with the techniques of this disclosure. Servers 300, 350 may each represent any of servers 12 of fig. 1. In some cases, the servers 300, 350 are configured to implement both virtualized cell site routers and distributed units for in-box forwarding of mobile data traffic between the DU 22A and the data plane of the virtualized cell site router 20A. The servers 300, 350 may each be bare metal servers or virtual machines. An example hardware architecture of the servers 300, 350 is depicted in fig. 8.

The servers 300, 350 include one or more Network Interface Cards (NICs) 321A-321B (collectively, "NICs 321"), each having one or more hardware interfaces. In a 5G radio access network deployment, interface 320 of NIC 321A may be coupled to an RU via physical cabling. Interface 320 may implement an F2 interface. The interface 322 of the NIC 321B may be coupled to the intermediate network via physical cabling for sending and receiving mobile data traffic to and from the CUs. Interface 322 may implement the F1 interface.

At a higher level, DPDK-based virtual router data or forwarding plane ("virtual router") 206A is programmed by virtual router agent 314 with forwarding information for implementing packet fast paths. Virtual router agent 314 may be a user space process. Virtual router agent 314 may have a northbound interface 340 for receiving configuration and routing information from a control plane process, such as cRPD 324. cRPD 324 may be an example of cRPD 24A of fig. 1. Virtual router agent 314 has a southbound interface 341 that is used to program virtual router 206A. An example implementation of interface 340 is described in more detail with reference to fig. 5. References herein to a "virtual router" may refer specifically to a virtual router forwarding plane, or to a combination of a virtual router forwarding plane (e.g., virtual router 206A) and a corresponding virtual router agent (e.g., virtual router agent 314).

cRPD 324 may have a northbound interface for exchanging configuration and routing information with SDN controller 70. The containerized network interface 312 may be a CNI plug-in that configures the interface of the container workload (in this example DUs 22A-1 through 22A-N) with the DPDK based virtual router 206A. The orchestrator 50 may orchestrate the DPDK-based virtual router 206A, cRPD 324 and/or the DU 22 workload. In some cases, the workload may have multiple interfaces and multiple types of interfaces (e.g., some with virtual router 206A, some with NIC 321A). Thus, CNI 312 may represent a CNI or a combination of unified CNIs that are capable of configuring a workload through multiple types of interfaces. Multiple CNIs may be controlled by a master CNI such as Multus. Where orchestrator 50 is Kubernetes master, customized Resource Definition (CRD) may be implemented for orchestrator 50 to support multi-tenant and network isolation.

The orchestrator 50 orchestrates pods comprising the container workload. CNI 312 configures a virtual interface between the pod and a data plane, which may be DPDK-based virtual router 206A, a kernel-based virtual router, or a SmartNIC-based virtual router. In some examples, CNI 312 configures the virtio interface of each pod as a vhost user interface of DPDK-based virtual router 206A. In some examples, CNI 312 configures the veth pair for each pod to virtual router 206A. Virtual router agent 314 may collect telemetry data, e.g., in the form of a system log, and output it to a telemetry collector. In some examples, virtual router 206A has a binding interface with NIC 321B, which may be an Intel-based NIC that supports DPDK. The binding interface facilitates packet load balancing between the topology interfaces. Additional description of configuring virtual interfaces may be found in U.S. patent 10,728,145 issued at 7.28 in 2020, which is incorporated herein by reference in its entirety.

In Kubernetes deployment, CNI 312 provides a network for application workloads. This includes, for example, setting interfaces, IP address management, and access control lists; advertising reachability of workloads within the Kubernetes cluster including any of the servers 300, 350 (examples of Kubernetes slave nodes); and setting a network name space.

CNI 312 may utilize cRPD 324 to extend its control plane capabilities and facilitate virtualized cell site router 20A onboard with application workloads DUs 22A-1 through 22A-N. cRPD 324 may incorporate elements of network service grid architecture (NSM), service discovery, external endpoints, and tunnels. cRPD 324 may use an external routing protocol, such as Border Gateway Protocol (BGP), to advertise pod reachability both inside and outside the Kubernetes cluster. cRPD 324 may use interior gateways and other routing protocols such as IS-IS, OSPF, label Distribution Protocol (LDP), etc. to participate in the underlying network. cRPD 324 may also use protocols/technologies such as MPLS, MPLSoUDP or MPLSoGRE tunneling; vxLAN; SR-MPLS, SRv6, SRv4 and/or IPSec to provide support for advanced L3VPN coverage.

cRPD 324 operates as the control plane for vCSR 20A, while virtual router 206A operates as the data or forwarding plane for vCSR 20A. Thus, CNI 312 utilizing cRPD 324 can facilitate multi-tenants using an L3VPN, e.g., implementing network slices for different tenants; the ACL and network policy applied; and IPSec for high security.

Fig. 10 is a block diagram illustrating an example implementation of cRPD 324 or any other cRPD of the present disclosure that an orchestrator may pod to deploy. cRPD 1440 may be deployed as a micro-service in Docker, coreOS (rkt) or other container platform.

cRPD 1440 includes a management interface 1400, which may represent a Command Line Interface (CLI), netconf, secure Shell (SSH), PCEP, or Simple Network Management Protocol (SNMP) interface. Management interface 1400 may support YANG, openConfig or other configuration data formats. The management interface 1400 may receive configuration data from the automation system 1420 and may output telemetry data to the telemetry system 1422.

The cRPD 1440 implementation may include BGP, OSPF, IS-IS, LDP, routing protocol 1402 for segment routing, and may receive static routing (represented by programmability 1424) for programming from a controller or an automation system. cRPD 1440 includes routing infrastructure 1404 to support routing protocol 1402. Routing infrastructure 1404 may include a Routing Information Base (RIB), a RIB manager, a Label Information Base (LIB), a LIB manager. Routing infrastructure 1404 may implement Bidirectional Forwarding Detection (BFD). The cRPD 1440 includes a Forwarding Information Base (FIB) adaptation layer (1406) to integrate the cRPD 1440 into the data plane by configuring forwarding information in the data plane. For example, FIB adaptation layer 1406 may implement a gRPC, netlink, or rtsock interface to program a virtual router (e.g., a DPDK-based virtual router). The FIB adaptation layer 1406 may implement another type of interface to program a virtual router, kernel-based vSwitch, smartNIC, network processor, ASIC-based forwarding chip, or other data plane.

FIG. 3B illustrates an example implementation of a server with disjoint data planes. Kernel 380 may represent a Linux kernel, other Unix variant kernels, or other operating system kernels that include networking stacks and are capable of packet forwarding.

Server 350 has two data planes for packet forwarding, a first data plane implemented by core 380 and a second data plane implemented by virtual router 206A. The DPDK-based virtual router 206A is configured with "ownership" of the physical interface 322. Physical interface 322 may be a VPN attachment circuit for VRF 212. The physical interfaces 322 may be associated with respective interfaces of the virtual router 206A through which the virtual router 206A sends and receives traffic via the physical interfaces 322.

In accordance with the techniques of this disclosure, virtual router 206A exposes respective interfaces 382 to kernel 380 of physical interface 322. That is, for each physical interface, virtual router 206A exposes the interface to kernel 380. Each interface 382 may be a vHost interface. Thus, the core 380 may send and receive network packets with the virtual router 206A via the interface 382.

In some examples, cRPD 324 runs routing protocols and needs to exchange routing protocol messages with routers external to server 350. In addition, cRPD 324 relies on kernel 380 networking stacks to obtain network topology information for the underlying network, which cRPD 324 needs to establish routing protocol adjacencies with external routers. Interface 382 provides cRPD 324 with access to physical interface 322 via kernel 380 and virtual router 206A, and thus to the underlying network accessible via physical interface 322. Such underlying networks may include a midplane network, a switching fabric for a local data center in which server 350 is located, and the like. Virtual router 206A is configured with a route that causes virtual router 206A to forward network packets received at one of physical interfaces 322 and destined for the IP address of a corresponding one of interfaces 382 to core 380 via the corresponding one of interfaces 382.

Core 380 outputs network packets to cRPD 324 via interface 384. Interface 384 may represent a system call interface/API, file system, pthread, socket, or other mechanism by which a process, such as cRPD 324, may receive packets from core 380 and inject packets into the core, as exposed by core 380. In this manner, cRPD 324 may operate as a control plane for executing routing protocols for virtualized cellular site router 20A in a manner that incorporates networking stacks, routing protocol infrastructure, and other network features of kernel 380; while virtual router 206A may operate as a data plane for forwarding data traffic between DUs 22A-1-22A-N and physical interface 322 in a manner that excludes kernel 380. Thus, because DPDK-based virtual router 206A operates in user space and generally provides better performance capabilities than core-based forwarding, these disjoint data planes (core 380+ virtual router 206A) and (individual virtual router 206A) can provide fast path data packet processing through virtual router 206A as well as complete control plane routing functions for virtualized cellular site router 20A.

Fig. 6 is a block diagram illustrating an example server having example control and data traffic flows within the server in accordance with the techniques of this disclosure. The server 600 may be similar to the server 350 of fig. 3B or other servers described herein. Server 600 differs from server 350 in that pods 422A-422L are not necessarily DUs (e.g., DU microservices), although pods 422A-422L may be DUs in some cases. cRPD 324 operates as the control plane of the router implemented by server 600 and DPDK-based virtual router 206A operates as the fast path forwarding plane of the router. From the perspective of virtual router 206A, pods 422A-422L are endpoints, and may represent, in particular, overlay endpoints of one or more virtual networks that have been programmed into virtual router 206A. A single vhost interface vhost0 interface 382A may be an example of any of interfaces 328 of fig. 3B and is exposed by virtual router 206A to kernel 380 and in some cases by kernel 380 to virtual router 206A. The vhost interface 382A has an associated underlying host IP address for receiving traffic "at host". Thus, the core 380 may be a network endpoint of an underlying network that includes the server 600 as a network device, the network endpoint having an IP address of the vhost interface 382A. The application layer endpoint may be cRPD 324 or other process managed by kernel 380.

An underlying network refers to the physical infrastructure that provides connectivity between nodes (typically servers) in the network. The underlying network is responsible for delivering packets across the infrastructure. The underlying network devices use routing protocols to determine IP connectivity. Conventional routing protocols used for routing purposes on the underlying network devices are OSPF, IS-IS, and BGP. An overlay network refers to a virtual infrastructure that provides connectivity between virtual workloads (typically VMs/pods). This connectivity builds on top of the underlying network and allows virtual networks to be built. Overlay traffic (i.e., virtual networks) is typically encapsulated in IP/MPLS tunnels or other tunnels, which are routed by the underlying network. The overlay network may run across all or a subset of the underlying network devices and implement multi-tenancy via virtualization.

Control traffic 700 may represent routing protocol traffic for one or more routing protocols performed by cRPD 324. In server 600, control traffic 700 may be received through physical interface 322 owned by virtual router 206A. The virtual router 206A is programmed with a route for the host IP address of the vhost0 interface 382A and receives the next hop, which causes the virtual router 206A to send traffic received at the physical interface 322 and destined for the host IP address of the vhost0 interface 382A to the core 380 via the vhost0 interface 382A. From the perspective of cRPD 324 and core 380, all such control traffic 700 appears to be from vhost0 interface 382A. Accordingly, the cRPD 324 route will designate the vhost0 interface 382A as the forwarding next hop for the route. cRPD 324 selectively installs some routes to virtual router agent 314 and the same (or other) routes to core 380, as described in further detail below. Virtual router agent 314 will receive Forwarding Information Base (FIB) updates corresponding to some of the routes received by cRPD 324. These routes will point to the vHost0 interface 382A, and the virtual router 206A may automatically translate or map the vHost0 interface 382A to the physical interface 322.

As explained above with respect to fig. 3B, the routing information programmed by cRPD 324 may be categorized into bottom layers and overlays. The cRPD 324 will install the underlying routes to the core 380 because the cRPD 324 may need this reachability to establish additional protocol adjacencies/sessions with external routers, e.g., BGP multi-hop sessions over reachability provided by IGPs. cRPD 324 supports the use of routing policy constructs that allow matching with RIB, routing instances, prefixes, or other attributes to selectively filter FIB updates to a particular data plane (e.g., to kernel 380 or virtual router 206A).

Control traffic 700 sent by cRPD 324 to virtual router 206A over vhost0 interface 382A may be sent by virtual router 206A from corresponding physical interface 322 for vhost0 interface 382A.

As shown, cRPD-based CNI 312, upon notification by orchestrator 50 via orchestration agent 310, will create a virtual network (here "pod") interface for each application pod 422A, 422L. One end of the pod interface terminates in a receptacle included in the pod. CNI 312 may request that virtual router 206A begin monitoring the other end of the pod interface and cRPD 324 facilitate forwarding of traffic from physical interface 322 destined for the application container of DPDK-based pods 422A, 422L using DPDK, exclusively and without involving core 380. The reverse procedure applies to traffic initiated by the containers 422A, 422L.

However, because the DPDK-based virtual router 206A manages these virtual network interfaces for pods 422A, 422L, the kernel 380 is unaware of these virtual network interfaces. Server 600 may use a tunnel dedicated to the DPDK forwarding path to perform the DPDK based clustering 422A, 422L; virtual router 206A; and NIC 312B, to internally transmit and receive overlay data traffic 800.

Thus, in server 600, cRPD 324 interfaces with two disjoint data planes: a core 380 and a DPDK based virtual router 206A. The cRPD 324 utilizes the kernel 380 networking stack to set up routes specifically for DPDK fast paths. The routing information received by cRPD 324 includes underlying routing information and overlay routing information. The cRPD 324 runs a routing protocol on the vpest interface 382A visible in the core 380, and the cRPD 324 may install FIB updates in the core 380FIB corresponding to IGP learned routes (underlying routing information). This may allow for the establishment of a multi-hop iBGP session to the destination indicated in such IGP learned routes. Again, the cRPD 324 routing protocol adjacency involves kernel 380 (and vHost interface 382A) because kernel 380 executes the networking stack.

Virtual router agent 314 of virtual router 206A notifies cRPD 324A of the application pod interface of pods 422A, 422L. These pod interfaces are created by CNI 312 and managed exclusively by virtual router agent 314 (i.e., without involving kernel 380). These pod interfaces are not known to the core 380. cRPD 324 may advertise reachability of these container interfaces to the rest of the network as L3VPN routes that include Network Layer Reachability Information (NLRI). In the 5G mobile network context, such L3VPN routes may be stored in VRFs of virtual router 206A for different network slices. Via interface 340 with virtual router agent 314, the corresponding MPLS route may be programmed by cRPD 324 only to virtual router 206A, and not to core 380. This is so because the next hop of these MPLS labels is the pop and forward of the pod interface to one of pods 422A, 422L; these interfaces are only visible in virtual router 206A and not visible in core 380. Similarly, reachability information received over BGP L3VPN may be selectively programmed to virtual router 206A by cRPD 324, as only such routing is required for forwarding traffic generated by pods 422A, 422. The kernel 380 does not have any application that requires such reachability. The above routes programmed into virtual router 206A constitute overlay routes for the overlay network.

Fig. 4 is a block diagram illustrating an example server in accordance with the techniques of this disclosure. The server 400 may be similar to the server 600 of fig. 6. The first data plane 394 includes the kernel 380. The second data plane 392 includes virtual router 206A virtual router agent 314 of virtual router 206A. The first data plane 394 and the second data plane 392 are disjoint. The first data plane 394 and the second data plane 392 may store different routes for the underlying network and the overlay network, respectively. The first data plane 394 and the second data plane 392 may independently perform forwarding lookups and forward traffic using respective different storage routes. cRPD 324 is a routing protocol process directed to handling both underlying and overlay routes. After learning routes, crpd 324 may selectively program (via virtual router agent 314) underlying routes to kernel 380 and overlay routes to virtual router 206A, either through routing protocols or from SDN controller 70.

Fig. 5 is a block diagram illustrating an example virtual router agent in accordance with the techniques of this disclosure. Virtual router agent 314 includes a gRPC server 520 for exchanging data with cRPD 324 (gRPC client) via generic interface 340. The APIs of gRPC server 520 include a Virtual Machine Interface (VMI) API 530 for exchanging virtual network interface data and requests, a configuration API 532 for exchanging configuration data and requests, and a routing API 534 for exchanging routes and requests-including for enabling cRPD 324 to program routes to virtual router 206A via virtual router agent 314. The synchronization module 544 programs the virtual router 206A with a virtual network interface (e.g., a portion of the veth pair or virtio-vhost interface between the DPDK pod and the DPDK-based virtual router 206A) and programs the virtual router 206A with routing information.

Interface 540 may represent a data structure that stores data describing the virtual network interface of an application pod executing on a server executing virtual router agent 314. Port service 542 listens for requests from CNI 312, such as a request to add a new "port" to the application pod, which port service 542 may translate to subscribing to requests for cRPD 324 via interface 340 to obtain virtual network interface configuration information from cRPD 324. Port service 542 may be implemented using a REST server.

In this manner, virtual router agent 314 provides generic interface 340 to the data plane for overlay traffic originated by or destined for an application pod on the server. The generic interface 340 may be implemented by any controller, routing protocol process, or other agent, as it relies on the gRPC rather than a proprietary interface.

Example implementation details of virtual router 314 of fig. 5 are described further below.

In an aspect of the disclosure, the generic data plane model is decoupled from a network controller of the virtualized computing infrastructure. For example, the number of the cells to be processed, the data plane in accordance with this aspect may expose Application Programming Interfaces (APIs) 530, 532, 534 that may be implemented by any control plane services. In some examples, the data plane can also be used with multiple types of CNIs 312. The data plane may use a virtual router based on DPDK the gRPC interface 340 is implemented and exposed for exchanging control data. For example, virtual router agent 314 for virtual router data plane may operate as a gRPC server 520 exposing a gRPC API for programming virtual router data plane 206A. These techniques include a workflow for configuring a virtual network interface for a pod, where virtual router agent 314 obtains information from a container routing protocol daemon (cRPD) 324 in response to a port request from CNI 312.

The present disclosure describes a generic data plane model decoupled from an SDN controller and may expose APIs that may be implemented by any control plane service, such as virtual router agent 314. The proposed data plane (e.g., virtual router 206A and virtual router agent 314) will also have the capability to work with any CNI. In some cases, the data plane will work with the Plater CNI. The software component includes a containerized routing protocol daemon (cRPD) to support network services grid (NSM) architecture the set of software components supports NSM architecture.) furthermore, a solution that potentially optimizes the data plane to be modular and less space consuming will be considered. The generic data plane presented herein may be an extension of the current track-based data plane. The proposed design will include a range for both virtual routers and DPDK forwarding planes, while also meeting the requirements to support the technology to be pushed out (e.g. eBPF and XDP) and to support virtual router and DPDK based forwarding planes. The design will also pave the way for the same general data plane to work with multiple control planes simultaneously. Thus, the compute node data plane may become more loosely coupled with the SDN control plane than existing control plane and data plane integration schemes.

In the model of the existing data plane, there are two building blocks implementing the data plane—a virtual router agent and a virtual router (or dpdk) forwarding plane. The virtual router agent may be a user space process running on Linux. It acts as a local lightweight control plane and is responsible for the following functions:

● It uses XMPP to exchange control states such as routes with the control nodes.

● It receives low-level configuration states, such as routing instances and forwarding policies, from the control node using XMPP.

● It reports analysis status (such as logs, statistics, and events) to the analysis node.

● It installs the forwarding state into the forwarding plane.

● It cooperates with orchestration agents to discover the presence and properties of VMs.

● It applies a forwarding policy to the first packet of each new flow and installs the flow entry in the flow table of the forwarding plane.

● It proxies DHCP, ARP, DNS and MDNS. Additional agents may be included in the future.

Each virtual router agent is connected to at least two control nodes to implement redundancy in a dual active redundancy model.

The virtual router forwarding plane is an existing system, can operate as a loadable kernel module in Linux, and is responsible for the following functions:

● It enables encapsulation of packets to be sent to the overlay network and de-encapsulation of packets to be received from the overlay network.

● It assigns the packet to the routing instance: packets received from the overlay network are assigned to the routing instance based on MPLS labels or Virtual Network Identifiers (VNIs). The virtual interface with the local virtual machine is bound to the routing instance.

● It looks up the destination address in a Forwarding Information Base (FIB) (also called forwarding table) and forwards the packet to the correct destination. The route may be a layer three IP prefix or a layer two MAC address.

● Forwarding policies may be applied using a flow table: it matches the packet with the flow table and applies the flow action.

It sends the packet for which no flow rule was found (i.e. the first packet for each flow) to the virtual router agent, which then installs the rule in the flow table.

It sends certain packets (such as DHCP, ARP, MDNS) to the virtual router proxy for proxy to SDN controller.

In accordance with techniques of one or more aspects of the present disclosure, a virtual router forwarding plane may be implemented using a DPDK-based router, which may exhibit the following properties:

● DPDK-based forwarding is implemented entirely in user space

● Virtual router applications run as multiple logical cores

O logic (Lcore) is pthread with core relevance

Lcore operates in polling mode and processes packet bursts to obtain optimal performance

● DPDK provides a lock-free ring for communication between Lcore

● Highly optimized for Intel architecture

Effective use of CPU cache

Large page to reduce TLB misses

○NUMAaware

The generic data plane interface of any type of virtual router to the control plane can be accomplished by enhancing the current model in one of the following ways:

(1) Virtual router proxy + virtual router/DPDK forwarding plane and XMPP northbound interface. The current model is kept unchanged by using the virtual router proxy as the data plane and the control plane is made to implement XMPP (as the trace control node does). However, not all control planes support XMPP and it may not be the preferred method. The virtual router agent provides a great deal of support for legacy openstack functionality that may not be necessary, resulting in greater footprint.

(2) Virtual router proxy + virtual router/DPDK forwarding plane and GRPC northbound interface. The current data plane and forwarding plane are maintained, but a common open source protocol (such as GRPC) is implemented as an interface with the control plane. Using a more widely available protocol, such as GRPC, as a northbound interface would present more opportunities. The use of a control plane may be increased. Nonetheless, the virtual router agent provides a great deal of support for traditional openstack functionality that may not be necessary, resulting in greater footprint.

(3) Virtual router/DPDK forwarding plane + lean virtual router proxy and GRPC northbound interface. The virtual router/DPDK forwarding plane is kept unchanged but the footprint of the virtual router agent is reduced by employing only the functions strictly required by the forwarding plane. The northbound interface may be XMPP or preferably GRPC. The virtual router agent footprint may be reduced in two ways:

● Modularizing and lean current virtual router agents

● Implementing a new library that will represent lean agents and only have the necessary functionality in this scenario, as shown in (2), using a more widely applicable protocol (such as GRPC) as the northbound interface would present more opportunities and the adoption of the control plane could be increased. Furthermore, lean and modular agents reduce overall footprint and provide flexibility in choosing and selecting functions as desired.

(4) Virtual router/DPDK forwarding plane + northbound interface (GRPC). The virtual router/DPDK forwarding plane is directly exposed to the control plane. This approach eliminates the need for a separate virtual router agent, but it loses the current hardware abstraction, where the virtual router is masked from the control plane. Furthermore, the intelligence provided by the virtual router agents must be absorbed by the control plane or virtual router, which makes it more complex.

The combination of schemes (2) and (3) may facilitate a generic data plane. In case the virtual router based data plane can be used in combination with the cRPD based control plane, the proposed architecture will be as shown in fig. 3A to 3B and fig. 4 to 6. It can be seen that in these examples there is a virtual router agent 314, a virtual router/DPDK forwarding plane 206A, and a gRPC northbound interface 340 as a generic data plane interface. Then, by decoupling the northbound interface from virtual router agent 314 to cRPD 324, the data planes of virtual router 206A and virtual router agent 314 become "generic". In some examples, virtual router 206A and virtual router agent 314 may operate in a single container and/or as separate pieces of software. The use of gRPC reduces the dependence on any particular control plane. The following support may be provided: gRPC interface as an abstraction layer over virtual router agent 314, interface to handle configuration+routing, gRPC interface to provide abstraction for configuration objects, standard data model for configuration northbound interface to control plane, and conversion to agent understandable format on southbound interface.

Port Add (Port Add) and Port Down sequences may be accomplished primarily via cRPD or via virtual router agents. Examples of such sequences of port addition and port closure are as follows:

Overlay routing in VRF and inet.0 or inet.3

In one example aspect, the system supports VRF functionality, where the overlay route and the underlay route remain separate in different VRFs.

Object format and protocol buffering (WIP)

In gRPC, a client may call a method on a server application. While virtual router agent 314 is a client in the XMPP model, it is a server in accordance with the techniques of this disclosure and invokes a function in the client when it must push an object.

Separate services are defined for routing and configuration programming. In addition, a separate service is defined for agents to notify cRPD of port add/delete/close events.

/>

Virtual machine interface (e.g., virtual network interface)

/>

route/Next Hop (NH)

/>

/>

/>

IPv4 routing 10.10.10.1/32 for native VMI

/>

MPLS routing of local VMI, label = 100

/>

Overlay remote routing 10.10.10.2/32, service label 200,DIP 100.100.100.1

The following is an example message structure for an Access Control List (ACL) policy. .

/>

Client/server model

The server may be implemented with two completion queues to operate in an asynchronous mode—one for the VMI subscription service 530 and the other for the routing/configuration services 532, 534. An abstract class ServiceData may be defined from which individual service data types inherit. The address of the service data object may be used as a tag that is added to the completion queue of various calls.

/>

/>

Fig. 7 is a conceptual diagram depicting a sequence of operations added to a port that results in route programming in a virtual router, according to example aspects of the present disclosure. The sequence of operations is described with respect to components of server 300, but may be performed by components of any of the servers (e.g., servers 12, 350, 400, 600) described in this disclosure. The sequence of operations in fig. 7 may be similar to the operation of the CNI-agent (option 2) described above. CNI 312 reserves a block of IP addresses for the pod. Virtual router agent 314 listens for port add and port delete messages, e.g., on the tlift service, where "port" corresponds to a virtual network interface. CNI 312 sends a port add message to virtual router agent 314 (702). The port add message includes an identifier of the virtual network of the port and the IP address assigned to the pod by CNI 312. (CNI 312 may configure the pod alone with the other end of the virtual network interface). Virtual router agent 314 creates a virtual network interface (referred to herein as a virtual machine interface or VMI, which is an example of a virtual network interface) in interface 540 (704). Virtual router agent 314 configures the virtual network interface in virtual router 206A with the default VRF identifier and VMI add message (706). Virtual router agent 314 subscribes to cRPD 324, but not the SDN controller, with VMI subscription messages that include the virtual network name and IP address received in the port add message (708). cRPD 327 sends VMI configuration message with the correct VRF identifier for the virtual network of the virtual network interface to virtual router agent 314 (712), optionally adding the VRF to virtual router agent 314 using a VRF add message, if necessary (710). Virtual router agent 314 sends a VMI update message with the correct VRF identifier to virtual router 206A to cause virtual router 206A to attach the virtual network interface to the correct VRF (714). cRPD 324 assigns a service label and uses the route add message to add the route and next hop (e.g., MPLS route for BGP IP-VPN) to virtual router agent 314 (716). cRPD 324 also advertises routes for reaching the pod to its peer routers (718), which may include other cRPD, routers in the underlying network, or other routers. Virtual router agent 314 configures virtual router 206A with forwarding information for the route received in the route add message from cRPD 324 (720). The example message structure and data structure of the message described above with respect to fig. 7 are defined.

Fig. 11 is a block diagram illustrating example components of a virtual router agent and an example sequence of operations and messages for creating and advertising new ports for pods in accordance with the techniques of this disclosure. The example sequence may be similar in some respects to the sequence described with respect to fig. 7.

Fig. 8 is a block diagram of an example computing device (e.g., host) in accordance with the techniques described in this disclosure. Computing device 800 of fig. 2 may represent a real or virtual server, and may represent an example instance of either server 12 or server 350 or 400 of fig. 1. In this example, the computing device 800 includes a bus 242 that couples the hardware components of the computing device 800 hardware environment. Bus 242 couples a Network Interface Card (NIC) 230, a memory disk 246, and one or more microprocessors 210 (hereinafter referred to as "microprocessors 810"). NIC 230 may have SR-IOV capability. In some cases, a front side bus may couple microprocessor 810 and memory device 244. In some examples, bus 242 may couple memory device 244, microprocessor 810, and NIC 230. Bus 242 may represent a Peripheral Component Interface (PCI) express (PCIe) bus. In some examples, a Direct Memory Access (DMA) controller may control DMA transfers between components coupled to bus 242. In some examples, components coupled to bus 242 control DMA transfers between components coupled to bus 242.

Microprocessor 810 may include one or more processors, each including a separate execution unit to execute instructions conforming to an instruction set architecture, the instructions stored to a storage medium. The execution units may be implemented as separate Integrated Circuits (ICs) or may be combined within one or more multi-core processors (or "many-core" processors), each implemented using a single IC (i.e., a chip multiprocessor).

Disk 246 represents computer-readable storage media including volatile and/or nonvolatile, removable and/or non-removable media implemented in any method or technology for storage of information such as processor-readable instructions, data structures, program modules, or other data. Computer-readable storage media includes, but is not limited to, random Access Memory (RAM), read Only Memory (ROM), EEPROM, flash memory, CD-ROM, digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by microprocessor 810.

Main memory 244 includes one or more computer-readable storage media that may include Random Access Memory (RAM), such as various forms of Dynamic RAM (DRAM), e.g., DDR2/DDR3 SDRAM or Static RAM (SRAM), flash memory, or any other form of fixed or removable storage medium, that may be used to carry or store desired program code and program data in the form of instructions or data structures and that may be accessed by a computer. Main memory 244 provides a physical address space comprised of addressable memory locations.

A Network Interface Card (NIC) 230 includes one or more interfaces 232 configured to exchange packets using links of an underlying physical network. Interface 232 may include a port interface card having one or more network ports. NIC 230 may also include on-card memory to store, for example, packet data. Direct memory access transmissions between NIC 230 and other devices coupled to bus 242 may be read from and written to NIC memory.

Memory 244, NIC 230, storage disk 246, and microprocessor 810 may provide an operating environment for a software stack that includes an operating system kernel 380 executing in kernel space. Kernel 380 may represent, for example, linux, berkeley software release (BSD), another Unix variant kernel, or a Windows server operating system kernel available from microsoft corporation. In some cases, an operating system may execute a hypervisor and one or more virtual machines managed by the hypervisor. Example hypervisors include kernel-based virtual machines (KVM) for Linux kernels, xen, ESXi available from VMware, window hyper-V available from Microsoft, and other open source and proprietary hypervisors. The term hypervisor may encompass a Virtual Machine Manager (VMM). An operating system including kernel 380 provides an execution environment for one or more processes in user space 245.

The kernel 380 includes physical drivers 225 to use the network interface card 230. The network interface card 230 may also implement an SR-IOV to enable sharing of physical network functions (I/O) among one or more virtual execution elements, such as containers 229A-229B or one or more virtual machines (not shown in fig. 2). A shared virtual device, such as a virtual function, may provide dedicated resources such that each virtual execution element may access the dedicated resources of NIC 230, and thus the NIC appears to each virtual execution element as a dedicated NIC. The virtual functions may represent lightweight PCIe functions that share physical resources with physical functions used by the physical drivers 225 and with other virtual functions. For an SR-IOV capable NIC 230, the NIC 230 may have thousands of available virtual functions according to the SR-IOV standard, but for I/O intensive applications, the number of configured virtual functions is typically much smaller.

The computing device 800 may be coupled to a physical network switching fabric that includes an overlay network that extends the switching fabric from a physical switch to software or "virtual" routers, including the virtual router 206A, that are coupled to physical servers of the switching fabric. The virtual router may be a process or thread or component thereof executed by a physical server (e.g., server 12 of fig. 1) that dynamically creates and manages one or more virtual networks that are available for communication between virtual network endpoints. In one example, the virtual router implements each virtual network using an overlay network that provides the ability to decouple the virtual address of the endpoint from the physical address (e.g., IP address) of the server on which the endpoint is executing. Each virtual network may use its own addressing and security scheme and may be considered orthogonal to the physical network and its addressing scheme. Various techniques may be used to transport packets within and across virtual networks through a physical network. The term "virtual router" as used herein may encompass Open VSwitch (OVS), OVS bridges, linux bridges, docker bridges, or other devices/software located on a host device and performing switching, bridging, or routing of packets between virtual network endpoints of one or more virtual networks, where the virtual network endpoints are hosted by one or more servers 12. In the example computing device 800 of fig. 2, the virtual router 206A executes within the user space as a DPDK-based virtual router, but the virtual router 206A may execute within a hypervisor, host operating system, host application, or virtual machine within various implementations.

The virtual router 206A may replace and contain virtual routing/bridging functions of the Linux bridge/OVS module typically used for Kubernetes deployment of pods 202. Virtual router 206A may perform bridging (e.g., E-VPN) and routing (e.g., L3VPN, IP-VPN) of virtual networks. Virtual router 206A may perform network services such as applying security policies, NAT, multicasting, mirroring, and load balancing.

The virtual router 206A may be implemented as a kernel module or as a user space DPDK procedure (the virtual router 206A is shown here as being in user space 245). Virtual router agent 314 may also execute in user space. In the example computing device 800 of fig. 2, the virtual router 206A executes within the user space as a DPDK-based virtual router, but the virtual router 206A may execute within a hypervisor, host operating system, host application, or virtual machine within various implementations. Virtual router agent 314 connects to network controller 24 using a channel that is used to download configuration and forwarding information. Virtual router agent 314 programs the forwarding state to the virtual router data (or "forwarding") plane represented by virtual router 206A. Virtual router 206A and virtual router agent 314 may be processes.

The virtual router 206A may replace and contain virtual routing/bridging functions of the Linux bridge/OVS module typically used for Kubernetes deployment of pods 202. Virtual router 206A may perform bridging (e.g., E-VPN) and routing (e.g., L3VPN, IP-VPN) of virtual networks. Virtual router 206A may perform network services such as applying security policies, NAT, multicasting, mirroring, and load balancing.

Virtual router 206A may be multi-threaded and execute on one or more processor cores. Virtual router 206A may include multiple queues. Virtual router 206A may implement a packet processing pipeline. Depending on the operations to be applied to the packet, virtual router agent 314 may stitch the pipeline from the simplest to the most complex. Virtual router 206A may maintain multiple instances of forwarding libraries. The virtual router 800 may use an RCU (read copy update) lock to access and update the table.

To send packets to other computing nodes or switches, virtual router 206A uses one or more physical interfaces 232. In general, virtual router 206A exchanges overlay packets with a workload such as a VM or pod 202 (in fig. 2). The virtual router 206A has a plurality of virtual network interfaces (e.g., vif). These interfaces may include a kernel interface vhost0 for exchanging packets with the host operating system; interface kt0 with virtual router agent 314p to obtain forwarding state from the network controller and send exception packets. There may be one or more virtual network interfaces corresponding to one or more physical network interfaces 232.

Other virtual network interfaces of virtual router 206A are used to exchange packets with the workload. The virtual network interfaces 212, 213 of the virtual router 206A are shown in fig. 2. The virtual network interfaces 212, 213 may be any of the types of virtual interfaces described above. In some cases, the virtual network interfaces 212, 213 are drop interfaces.

In a kernel-based deployment of virtual router 206A (not shown), virtual router 206A is installed as a kernel module within the operating system. Virtual router 206A registers itself with the TCP/IP stack to receive packets from any desired operating system interface it wants. The interface may be bond, physical, tap (for VM), veth (for container), etc. The virtual router 206A in this mode relies on the operating system to send and receive packets from different interfaces. For example, the operating system may expose a tap interface supported by the vhost-net driver to communicate with the VM. Once virtual router 206A registers packets from the tap interface, the TCP/IP stack sends all packets to it. Virtual router 206A sends the packet via the operating system interface. In addition, the NIC queues (physical or virtual) are handled by the operating system. Packet processing may operate in an interrupt mode, which generates interrupts and may result in frequent context switches. When higher packet rates are present, the overhead of frequent interrupts and context switches can overwhelm the operating system and lead to performance degradation.

In a DPDK based deployment of virtual router 206A (as shown in fig. 2), virtual router 206A is installed as a user space 245 application linked to a DPDK library. This may result in faster performance than core-based deployments, especially in the presence of high packet rates. The physical interface 232 is used by the Polling Mode Driver (PMD) of the DPDK rather than by the interrupt-based driver of the kernel. Registers of physical interface 232 can be exposed in user space 245 to be accessible to PMD; the physical interfaces 232 bound in this way are no longer managed by or visible to the host operating system, and the DPDK-based virtual router 206A manages the physical interfaces 232. This includes packet polling, packet processing, and packet forwarding. In other words, the user packet processing step is performed by the virtual router 206A DPDK data plane. The nature of this "polling mode" makes virtual router 206A DPDK data plane packet processing/forwarding more efficient than interrupt mode when the packet rate is high. There are relatively few interrupts and context switches during packet I/O as compared to the kernel-mode virtual router 206A, and in some cases interrupts and context switches during packet I/O may be completely avoided.

In general, each of pods 202A-202B may be assigned one or more virtual network addresses for use within a respective virtual network, where each of the virtual networks may be associated with a different virtual subnet provided by virtual router 206A. Pod 202B may be assigned its own virtual layer three (L3) IP address, e.g., for sending and receiving communications, but may not be aware of the IP address of computing device 800 on which pod 202B is executing. Thus, the virtual network address may be different from the logical address of the underlying physical computer system (e.g., computing device 800).

Computing device 800 includes virtual router agent 314, which controls the coverage of virtual networks for computing device 800 and coordinates the routing of data packets within computing device 800. In general, virtual router agent 314 communicates with network controller 24 of the virtualization infrastructure, which generates commands to create a virtual network and configure network virtualization endpoints, such as computing device 800, and more particularly virtual router 206A and virtual network interface 212. By configuring virtual router 206A based on information received from network controller 24, virtual router agent 314 may support configuring network isolation, policy-based security, gateways, source Network Address Translation (SNAT), load balancer, and service linking capabilities for orchestration.

In one example, network packets generated or consumed by containers 229A-229B within the virtual network domain, such as layer three (L3) IP packets or layer two (L2) ethernet packets, may be encapsulated in another packet (e.g., another IP or ethernet packet) that is transmitted by the physical network. Packets transmitted in a virtual network may be referred to herein as "inner packets" while physical network packets may be referred to herein as "outer packets" or "tunnel packets. Encapsulation and/or decapsulation of virtual network packets within physical network packets may be performed by virtual router 206A. This functionality is referred to herein as tunneling and may be used to create one or more overlay networks. In addition to ipineip, other example tunneling protocols that may be used include Generic Routing Encapsulation (GRE) -based IP, vxLAN, GRE-based multiprotocol label switching (MPLS), user Datagram Protocol (UDP) -based MPLS, and the like. Virtual router 206A/performs tunnel encapsulation/decapsulation of packets originating/destined for any of the containers of pod 202, and virtual router 206A exchanges packets with pod 202 via bus 242 and/or the bridge of NIC 230.

As described above, network controller 24 may provide a logically centralized controller to facilitate the operation of one or more virtual networks. Network controller 24 may, for example, maintain a routing information base, such as one or more routing tables that store routing information for the physical network and one or more overlay networks. Virtual router 206A implements one or more virtual routing and forwarding instances (VRFs) 222A-222B for the respective virtual networks, wherein virtual router 206A operates as a respective tunnel endpoint. In general, each VRF 222 stores forwarding information for the corresponding virtual network and identifies where the data packet is to be forwarded and whether the packet is to be encapsulated in a tunneling protocol, such as with a tunneling header, which may include one or more headers for different layers of the virtual network protocol stack. Each of VRFs 222 may include a network forwarding table that stores routing and forwarding information for the virtual network.

NIC 230 may receive the tunnel packet. Virtual router 206A processes the tunnel packet to determine the virtual network of the source endpoint and the destination endpoint of the inner packet from the tunnel encapsulation header. Virtual router 206A may strip the layer two header and tunnel encapsulation header to forward the inner packet only internally. The tunnel encapsulation header may include a virtual network identifier (such as a VxLAN label or MPLS label) that indicates the virtual network (e.g., the virtual network corresponding to VRF 222A). VRF 222A may include forwarding information for the internal packet. For example, VRF 222A may map the destination third layer address of the inner packet to virtual network interface 212. In response, VRF 222A forwards the internal packet to pod 202A via virtual network interface 212.

Container 229A may also initiate an internal packet as a source virtual network endpoint. For example, container 229A may generate a third tier internal packet destined for a destination virtual network endpoint executed by another computing device (i.e., not computing device 800) or destined for another container. Container 229A may send the third layer internal packet to virtual router 206A via virtual network interface 212 attached to VRF 222A.

Virtual router 206A receives the inner packet and the layer two header and determines the virtual network of the inner packet. Virtual router 206A may determine a virtual network using any of the virtual network interface implementation techniques described above (e.g., macvlan, veth, etc.). Virtual router 206A uses VRF 222A corresponding to the virtual network of the inner packet to generate an outer header of the inner packet that includes an outer IP header for the overlay tunnel and a tunnel encapsulation header identifying the virtual network. Virtual router 206A encapsulates the inner packet with an outer header. The virtual router 206A may encapsulate the tunnel packet with a new layer two header having a destination layer two address associated with a device external to the computing device 800 (e.g., one of the TOR switches 16 or the servers 12). If external to computing device 800, virtual router 206A outputs the tunnel packet with the new layer two header to NIC 230 using physical function 221. NIC 230 outputs packets on the outbound interface. If the destination is another virtual network endpoint executing on computing device 800, virtual router 206A routes the packet to the appropriate one of virtual network interfaces 212, 213.

In some examples, a controller of computing device 800 (e.g., network controller 24 of fig. 1) configures a default route in each of pods 202 to cause virtual machine 224 to use virtual router 206A as an initial next hop for an outbound packet. In some examples, NIC 230 is configured with one or more forwarding rules to cause all packets received from virtual machine 224 to be switched to virtual router 206A.

Pod 202A includes one or more application containers 229A. Pod 202B includes an instance of cRPD 324. Container platform 804 includes container runtime 208, orchestration agent 310, service agent 211, and CNI 312.

Container engine 208 includes code executable by microprocessor 810. The container runtime 208 may be one or more computer processes. Container engine 208 runs containerized applications in the form of containers 229A-229B. Container engine 208 may represent a Dockert, rkt, or other container engine for managing containers. In general, container engine 208 receives requests and manages objects such as images, containers, networks, and volumes. The image is a template with instructions for creating a container. A container is an executable instance of an image. Based on instructions from controller agent 310, container engine 208 may obtain images and instantiate them as executable containers in pods 202A-202B.

Service agent 211 includes code executable by microprocessor 810. Service agent 211 may be one or more computer processes. Service agent 211 monitors the addition and removal of services and endpoint objects and it maintains the network configuration of computing device 800 to ensure communication between the pods and the containers, such as using the services. Service proxy 211 may also manage iptables to capture traffic to the virtual IP address and port of the service and redirect the traffic to proxy ports of the pods supported by the proxy. Service proxy 211 may represent a kube proxy for a subordinate node of the Kubernetes cluster. In some examples, container platform 804 does not include service agent 211, or service agent 211 is disabled to facilitate CNI 312 configuring virtual router 206A and pod 202.

Orchestration agent 310 comprises code executable by microprocessor 810. Orchestration agent 310 may be one or more computer processes. Orchestration agent 310 may represent kubrelet for the subordinate nodes of the Kubernetes cluster. Orchestration agent 310 is an agent of an orchestrator (e.g., orchestrator 23 of fig. 1) that receives container specification data for the containers and ensures that the containers are executed by computing device 800. The container specification data may be in the form of manifest files sent from orchestrator 23 to orchestration agent 310 or received indirectly via a command line interface, HTTP endpoint, or HTTP server. The container specification data may be a container specification (e.g., pod spec—yaml (yet another markup language) or JSON object describing the container) of one of the pods 202 of the container 229. Based on the container specification data, orchestration agent 310 instructs container engine 208 to obtain and instantiate a container image of container 229 to execute container 229 by computing device 800.

Orchestration agent 310 instantiates or otherwise invokes CNI 312 to configure one or more virtual network interfaces for each of pods 202. For example, orchestration agent 310 receives the container specification data for pod 202A and instructs container engine 208 to create pod 202A with container 229A based on the container specification data for pod 202A. Orchestration agent 310 also invokes CNI 312 to configure pod 202A with a virtual network interface of the virtual network corresponding to VRF 222A. In this example, pod 202A is a virtual network endpoint of a virtual network corresponding to VRF 222A.

CNI 312 may obtain interface configuration data for configuring the virtual network interfaces of pod 202. Virtual router agent 314 operates as a virtual network control plane module for enabling network controller 24 to configure virtual router 206A. Unlike orchestration control planes (including container platforms 804 for slave nodes and master nodes, e.g., orchestrator 23) that manage provisioning, scheduling, and managing virtual execution elements, virtual network control planes (including network controllers 24 and virtual router agents 314 for slave nodes) manage the configuration of the virtual network implemented in the data plane in part by the slave node's virtual router 206A. Virtual router agent 314 communicates interface configuration data for the virtual network interface to CNI 312 to enable orchestration control plane elements (i.e., CNI 312) to configure the virtual network interface according to the configuration state determined by network controller 24, bridging the gap between the orchestration control plane and the virtual network control plane. Furthermore, this may enable CNI 312 to obtain interface configuration data for and configure multiple virtual network interfaces of the pod, which may reduce communication and resource overhead inherent in invoking a separate CNI 312 to configure each virtual network interface.

FIG. 9 is a block diagram of an example computing device operating as an instance of an orchestrator master node that virtualizes a cluster of computing infrastructure. The computing device 300 of fig. 9 may represent one or more real or virtual servers. As such, the computing device 300 may, in some instances, implement one or more master nodes for the respective clusters.

The scheduler 1322, API server 1320, network controller manager 1326, network controller 1324, network controller manager 1325, and configuration store 1328, while shown and described as being executed by a single computing device 300, may be distributed among multiple computing devices 300 that establish a computing system or hardware/server cluster. In other words, each of the plurality of computing devices 300 may provide a hardware operating environment for one or more instances of any one or more of the scheduler 1322, API server 1320, network controller manager 1326, network controller 1324, network controller manager 1325, or configuration store 1328. Network controller 1324 may represent an example instance of network controller 24 of fig. 1. Scheduler 1322, API server 1320, controller manager 1326, and network controller manager 1325 may implement example instances of orchestrator 23. The network controller manager 1325 may represent an example implementation of a Kubernetes cloud controller manager or a Kube manager. Network controller 1324 may represent an example instance of network controller 24.

In this example, computing device 300 includes a bus 1342 that couples the hardware components of the computing device 300 hardware environment. The bus 1342 couples a Network Interface Card (NIC) 1330, a memory disk 1346, and one or more microprocessors 1310 (hereinafter "microprocessors 1310"). In some cases, a front side bus may couple microprocessor 1310 and memory device 1344. In some examples, bus 1342 may couple memory device 1344, microprocessor 1310, and NIC 1330. Bus 1342 may represent a Peripheral Component Interface (PCI) express (PCIe) bus. In some examples, a Direct Memory Access (DMA) controller may control DMA transfers between components coupled to bus 242. In some examples, components coupled to bus 1342 control DMA transfers between components coupled to bus 1342.

The microprocessor 1310 may include one or more processors, each including a separate execution unit to execute instructions conforming to an instruction set architecture, the instructions stored to a storage medium. The execution units may be implemented as separate Integrated Circuits (ICs) or may be combined within one or more multi-core processors (or "many-core" processors), each implemented using a single IC (i.e., a chip multiprocessor).

The disk 1346 represents computer-readable storage media including volatile and/or nonvolatile, removable and/or non-removable media implemented in any method or technology for storage of information such as processor-readable instructions, data structures, program modules, or other data. Computer-readable storage media includes, but is not limited to, random Access Memory (RAM), read Only Memory (ROM), EEPROM, flash memory, CD-ROM, digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by microprocessor 1310.

Main memory 1344 includes one or more computer-readable storage media, which can include Random Access Memory (RAM) such as various forms of Dynamic RAM (DRAM), e.g., DDR2/DDR3 SDRAM or Static RAM (SRAM), flash memory, or any other form of fixed or removable storage medium, which can be used to carry or store desired program code and program data in the form of instructions or data structures and which can be accessed by a computer. Main memory 1344 provides a physical address space comprised of addressable memory locations.

A Network Interface Card (NIC) 1330 includes one or more interfaces 3132 configured to exchange packets using links of an underlying physical network. Interface 3132 may include a port interface card having one or more network ports. NIC 1330 may also include on-card memory to store packet data, for example. Direct memory access transmissions between NIC 1330 and other devices coupled to bus 1342 may be read from and written to NIC memory.

Memory 1344, NIC 1330, storage 1346, and microprocessor 1310 may provide an operating environment for a software stack that includes an operating system kernel 1314 executing in kernel space. Kernel 1314 may represent, for example, linux, berkeley software release (BSD), another Unix variant kernel, or a Windows server operating system kernel available from microsoft corporation. In some cases, an operating system may execute a hypervisor and one or more virtual machines managed by the hypervisor. Example hypervisors include kernel-based virtual machines (KVM) for Linux kernels, xen, ESXi available from VMware, window hyper-V available from Microsoft, and other open source and proprietary hypervisors. The term hypervisor may encompass a Virtual Machine Manager (VMM). An operating system including kernel 1314 provides an execution environment for one or more processes in user space 1345. The kernel 1314 includes a physical driver 1325 to use the network interface card 230.

The computing device 300 may be coupled to a physical network switching fabric that includes an overlay network that extends the switching fabric from a physical switch to software or "virtual" routers, such as the virtual router 220 of fig. 2, that are coupled to physical servers of the switching fabric. The computing device 300 may configure the slave nodes of the cluster using one or more private virtual networks.

The API server 1320, scheduler 1322, controller manager 1326 and configuration store may implement the master node of the cluster and may alternatively be referred to as a "master component". The cluster may be a Kubernetes cluster and the master node is a Kubernetes master node, in which case the master component is a Kubernetes master component.

The API server 1320 includes code executable by the microprocessor 1310. The API server 1320 can be one or more computer processes. For example, the API server 1320 validates and configures data of objects, such as virtual execution elements (e.g., pods of containers), services, and copy controllers. A service may be an abstraction of a logical set defining a pod and a policy for accessing the pod. The set of pods implementing the service is selected based on the service definition. The service may be implemented in part as or otherwise include a load balancer. The API server 1320 may implement a representational state transfer (REST) interface to handle REST operations and provide a front end to the shared state of the corresponding cluster stored to the configuration store 1328. The API server 1320 may validate and authorize the request. The API server 1320 communicates with other components to instantiate virtual execution elements in the computing infrastructure 8. API server 1320 may represent a Kubernetes API server.

Configuration store 1328 is a backing store for all cluster data. The cluster data may include cluster state and configuration data. The configuration data may also provide a back-end for service discovery and/or provide a locking service. Configuration store 1328 may be implemented as a key-value store. Configuration store 1328 may be a central database or a distributed database. Configuration store 1328 may represent an etcd store. Configuration store 1328 may represent a Kubernetes configuration store.

Scheduler 1322 includes code that may be executed by microprocessor 1310. Scheduler 1322 may be one or more computer processes. Scheduler 1322 monitors newly created or requested virtual execution elements (e.g., pods of containers) and selects the subordinate nodes on which the virtual execution elements will run. The scheduler 1322 may select subordinate nodes based on resource requirements, hardware constraints, software constraints, policy constraints, locality, and the like. Scheduler 1322 may represent a Kubernetes scheduler.

In general, the API server 1320 may call the scheduler 1322 to schedule the virtual execution element, which may select a subordinate node and return an identifier of the selected subordinate node to the API server 1320, which may write the identifier to the configuration store 1328 associated with the virtual execution element. The API server 1320 may call the orchestration agent 310 of the selected dependent node, which may cause the container engine 208 of the selected dependent node to obtain virtual execution elements from the storage server and create virtual execution elements on the dependent node. Orchestration agent 310 for the selected slave node may update the state of the virtual execution element to API server 1320, which saves the new state to configuration store 1328. In this way, computing device 300 instantiates new virtual execution elements in computing infrastructure 8.

The controller manager 1326 includes code executable by the microprocessor 1310. The controller manager 1326 may be one or more computer processes. The controller manager 1326 may embed a core control loop to monitor the shared state of the cluster by retrieving notifications from the API server 1320. The controller manager 1326 may attempt to move the state of the cluster to a desired state. Example controllers (not shown) managed by controller manager 1326 may include a replication controller, an endpoint controller, a namespace controller, and a service account controller. The controller manager 1326 may perform lifecycle functions such as namespace creation and lifecycle, event garbage collection, termination of pod garbage collection, cascade delete garbage collection, node garbage collection, and the like. The controller manager 1326 may represent a Kubernetes controller manager for a Kubernetes cluster.

The network controller 1324 includes code executable by the microprocessor 1310. The network controller 1324 may include one or more computer processes. Network controller 1324 may represent an example instance of network controller 24 of fig. 1. The network controller 1324 may be a logically centralized but physically distributed Software Defined Network (SDN) controller that is responsible for providing management, control and analysis functions of the virtualized network. In particular, the network controller 1324 may be a logically centralized control plane and management plane of the computing infrastructure 8 and orchestrate virtual routers for one or more slave nodes.

The network controller 1324 may provide cloud networking for computing architectures operating on a network infrastructure. The cloud network may include private clouds of enterprises or service providers, infrastructure as a service (IaaS), and Virtual Private Clouds (VPC) of Cloud Service Providers (CSP). Private cloud, VPC, and IaaS use cases may involve a multi-tenant virtualized data center, such as that described with respect to fig. 1. In such cases, multiple tenants in the data center share the same physical resources (physical servers, physical storage, physical networks). Each tenant is assigned its own logical resources (virtual machines, containers, or other forms of virtual execution elements; virtual storage; virtual networks). These logical resources are isolated from each other unless specifically allowed by security policies. Virtual networks in the data center may also be interconnected to physical IP VPNs or L2 VPNs.

The network controller 1324 may provide Network Function Virtualization (NFV) to networks such as business edge networks, broadband subscriber management edge networks, and mobile edge networks. NFV involves orchestration and management of network functions in virtual machines, containers, or other virtual execution elements, rather than on physical hardware devices, such as firewalls, intrusion detection or prevention systems (IDS/IPS), deep Packet Inspection (DPI), caches, wide Area Network (WAN) optimizations, and so forth. The main driving force for the virtualization of web services in this market is time to market and cost optimization.

The network controller 1324 programs the network infrastructure elements to create a virtual network and may create interface configurations for virtual network interfaces of the virtual network.

Additional information regarding example network controllers can be found in international application number PCT/US2013/044378, and U.S. patent application 14/226,509, incorporated by reference above.

The network controller manager 1325 includes code executable by the microprocessor 1310. The network controller manager 1325 may be one or more computer processes. The network controller manager 1325 operates as an interface between orchestration-oriented elements (e.g., scheduler 1322, API server 1320, controller manager 1326, and configuration store 1328) and the network controller 1324. In general, the network controller manager 1325 monitors the new objects (e.g., pods and services) of the cluster. The network controller manager 1325 may isolate the pod in the virtual network and connect the pod with the service.

The network controller manager 1325 may be implemented as a container of master nodes of the cluster. In some cases, the use of the network controller manager 1325 enables disabling the service proxy (e.g., kubernetes kube-proxy) of the slave node such that all pod connectivity is implemented using virtual routers, as described herein.

The network controller manager 1325 may use the controller framework of the orchestration platform to monitor (or otherwise monitor) changes to objects defined in the APIs, and add annotations to some of these objects. The annotation may be a tag or other identifier (e.g., "virtual network green") that specifies an object attribute. The network controller manager 1325 may use the interface to the network controller 1324 to create networking solutions for applications to define network objects such as virtual networks, virtual network interfaces, and access control policies. The network controller 1324 may implement networking solutions in the computing infrastructure by, for example, configuring one or more virtual networks and virtual network interfaces in the virtual router.

The following example deployment configuration for this application consists of pods and virtual network information for the pods:

/>

this metadata information is copied to each pod copy created by controller manager 1326. When the network controller manager 1325 is notified of these pods, the network controller manager 1325 may create virtual networks (in the above example, "red network", "blue network", and "default/extended network") as listed in the notes and create a virtual network interface per pod copy (e.g., pod 202A) for each virtual network by a unique private virtual network address from the cluster-wide address block (e.g., 10.0/16) of the virtual network.

Additional techniques in accordance with the present disclosure are described below. Track (Contrail) is an example network controller architecture. The track CNI may be a CNI developed for the track. The trajectory controller may be an example of an SDN controller described in this disclosure, such as SDN controller 50.

Additional details of the control plane and data plane operations of the server according to the techniques of this disclosure are as follows. For example, these details may be implemented in server 600 of FIG. 6.

As part of the use case of vCSR, vCSR 20 may support IPv6 in the bottom layer and IPv6 tunnel-based SR-MPLS on virtual router 206. cRPD 324 controls plane traffic such as: OSPF, ISIS, etc. may be routed using IPv6 underlying support provided by virtual router 206. Overlay traffic from the customer pods may be routed by virtual router 206 through SR-MPLSoIPv6 or other tunnels. The overlay traffic may be identified using an L3VPN service label (programmed by cRPD 24). The SR-MPLS tunnel may be represented using a "label stack" programmed by cRPD 24.

In server 600 of fig. 6, the virtual network interface for pod 422 to virtual router 206A may be a virtual host interface for a separate VRF configured in virtual router 206A. Virtual router agent 314 and virtual router 206A may be configured with multiple interfaces for communicating with each other: pkt0 interfaces and Unix domain sockets (e.g., sandish). vHost0 382A is described elsewhere in this disclosure. In some cases, the cRPD 324 control plane traffic path via the IPv6 bottom layer may be as follows:

● The Vhost0 interface 382A of virtual router 206A will host IPv6 addresses used by cRPD324 to send and receive control plane traffic for BGP, IS-IS, OSPF, or other routing and control protocols, for example.

● Before cRPD324 can send the actual unicast control plane packet, cRPD324 can attempt to resolve the next-hop MAC address via an IPv6 neighbor solicitation request.

● The virtual router 206A will transparently send these IPv6 ND requests through the physical/layout interface (e.g., to one of the IF 322) attached thereto. Similarly, upon receiving a response to the solicitation request, virtual router 206A may send a response packet to cRPD324 and virtual router agent 314.

● Once cRPD324 resolves the next hop link layer address, the actual unicast control plane packets may flow.

● The actual unicast control plane packets may be routed by virtual router 206A to the physical/topology interface and vice versa. Routing will be based on routing programmed by cRPD324 and virtual router agent 314.

● Control plane multicast packets sent by cRPD324 over vhost0 interface 382A may be forwarded by virtual router 206A over a physical interface. Similarly, any multicast packets from the physical interface may be sent to cRPD324 using vhost0 interface 206A.

The overlay data path traffic (SR process) is traffic transmitted and received by the user pod 422 created in the server 600 (compute node). This may be IPv4 or IPv6 traffic.

Ingress SR-MPLS processing at vCSR 20 using PHP: in the transport direction, virtual router 206A may perform ingress SR-MPLS processing for traffic sent by pod 422. Fig. 12 shows an example configuration of servers 1200, 1220 and forwarding packets from pod 422A on server 1200 to pod 422M on server 1220. Cloud native router 1202 includes cRPD 324 for the control plane and an instance of virtual router 206A for the data plane.

Interface 1204A has an IP address of 10.1.1.1 and a label L1, and interface 1204B has an IP address of 20.1.1.1 and a label L2. Containers 422M, 422N have similar such interfaces, and CNR is not shown on server 1220.

In this example of fig. 12, which relies on segment routing:

● The cRPD 324 and router 1210 and intermediate SR-capable routers 1206, 1208 are configured with SR-capable ISIS/OSPF to exchange SR Segment Identifiers (SIDs) according to label. This results in cRPD 324 and router 1210 knowing the SIDs in the network and what these SIDs correspond to.

● cRPD 324 and router 1210 are also configured with BGP with inet and inet6 VPNs for exchanging overlay L3 VPN routes for the pod ("virtual") network. Thus, service labels that cover the routes are exchanged between cRPD 324 and router 1210.

● cRPD 324 now programs overlay routes, service labels, and underlying SR-MPLS next hop information to virtual router 206A via a virtual router agent (not shown in fig. 12). The mechanism of selecting the SR path is responsible for the cRPD 324 and optional SDN controller/path computation engine.

● In virtual router 206A, the overlay route is programmed in the cluster VRF and is associated with a service label and SR-MPLS next hop.

● Virtual router 206A SR-MPLS next hop consists of a list of SR labels to push, as well as L3 (IPv 6), L2 (ethernet) header information, and outgoing interfaces, all of which can be used to encapsulate and send packets out as packets 1230.

● As shown in fig. 12, once a packet is received from container 422A, a route lookup occurs in VRF red. This results in retrieving the service label to be used for the destination (here pod 422M) and the SR-MPLS next hop to be used for sending the packet out. The virtual router then encapsulates the packet with SR-MPLS next hop and sends it out.

● The packet then reaches the subsequent SR enabled router, where the SID tag is popped up and used to forward the packet to the next router in the path.

● Finally, when the packet arrives at 1210, the NULL tag and service tag are popped, which results in the actual overlay packet being forwarded to the O-CU destination pod.

The process without Penultimate Hop Pop (PHP) is similar to that described above, except that all SR nodes will also receive the SID they advertise in the top of the stack SID. It then pops the top of the stack SID and looks at the next SID to forward the packet further. The final SR endpoint will pop its SID and service tag up and forward the overlay packet using the service tag.

Outlet SR treatment at vCSR by PHP: in the receive direction, virtual router 206A will perform egress SR processing on traffic incoming through the fabric interface of the pod. Fig. 13 shows an example configuration of servers 1200, 1220 and forwarding packets from pod 422M to pod 422A.

● Virtual router agent 314 (not shown in fig. 13) will receive NH for vhost0 IPv6 address installation L3. After reading the proxy conf file that will contain the vhost0 IP address, this will be done at initialization of virtual router agent 314.

● The routing process will occur in both cRPD 324 and router 1210, as given in the first two steps of ingress processing.

● Receiving NH results in virtual router 206A being able to receive incoming traffic destined for the vhost0 IP address and process it further.

● For incoming traffic destined for the vhost0 IP address, virtual router 206A will check if the packet is an SR-MPLS packet and if so it will pop up the external NULL/vCSRS ID (in the absence of PHP) label.

● In addition, virtual router 206A will pop up the service label in the packet. The service tag will point to the pod VMI next hop.

● After the necessary L2 overwrite has been made, virtual router 206A will then forward the packet to the pod using the next hop of pod VMI.

Implicit NULL for external SR tags will also be supported. In this case, there will be no SR tag in the packet and processing will occur directly on the internal service tag.

Fig. 14 is a conceptual diagram illustrating example operations 1400 for programming virtual router forwarding information in accordance with the techniques of this disclosure.

Fig. 15 is a conceptual diagram illustrating example operations 1500 for configuring and advertising virtual network interfaces in a server with a cloud native router in accordance with the techniques of this disclosure. Operation 15 may be similar in some respects to the sequence described with respect to fig. 7.

As explained with respect to fig. 3A-3B, the DU 22 container may receive 5G radio traffic from Port0 that uses single root I/O virtualization (SR-IOV) to create multiple Virtual Functions (VFs) or instances (ports) of physical functions, where each VF terminates in its own pod (one of the DUs 22). These VFs are visible to the Linux kernel 380, however, they are not routing protocols running on them. Their sole purpose is to pull radio traffic into the DU 22. The DU 22 processes radio traffic and wishes to send the processed traffic through a tunnel (SR-MPLS) to a CU 5G function running in a data center, as described with respect to fig. 1.

To provide reachability on tunnels, cRPD 324 may be configured with the necessary protocols (IGP, BGP, etc.). The DPDK virtual router 206A will manage physical port 1-sending and receiving routing traffic through physical port 1.

cRPD 324 can be configured with the necessary protocols through Netconf via the domain controller. cRPD 324 will establish adjacencies for the various protocols; routing information (including reachability of application containers) is learned and advertised using its routing protocol. cRPD 324 needs to program this learned routing information to virtual router agent 314. Virtual router 206A will provide a bi-directional gRPC channel 340 for communication back and forth with cRPD 324. Data objects (routes, VRFs, interfaces, etc.) may be modeled in protocol buffers.

Control traffic will come from a different physical port (e.g., port 1) than port 0. However, virtual router 206A will detect that this is control traffic (non-tunnel traffic) and forward it over vhost0 interface 382A. From the perspective of cRPD 324, all traffic appears to come from vhost0 interface 382A. Thus, all cRPD routes will reference the vhost0 interface 382A. cRPD 324 will install these routes to virtual router agent 314 and core 380 (in some cases this may involve using RIB/instance policies to selectively install the underlying routes in inet.0 to the core). Virtual router agent 314 may automatically translate the route to vhost0 to port 1 as shown in fig. 14. The route that cRPD 324 will install to core 380 is because cRPD 324 may need reachability to establish additional protocol adjacencies/sessions, such as BGP multi-hop sessions for reachability provided by IGP. Control traffic sent by cRPD 324 to virtual router 206A over vhost0 interface 382A must be sent out of port 1 without any further action.

cRPD 324 may communicate with virtual router agent 314 by one of the following:

crpd 324 will continue to issue netlink messages. An external (to cRPD 324) converter will convert these into corresponding gRPC messages. The introduction of this converter may introduce some additional delay. The converter may be an in-situ stateless entity.

The crpd 324 directly begins using these gRPC APIs with a kernel routing table multi-channel or some version of FDM. In some cases, cRPD 324 may directly begin using these gRPC APIs to program Linux kernels as well as kernels for programming sonics through another kernel routing table on top of the existing channel.

As depicted in fig. 15, cRPD-based CNI 312 will create veth pairs for each of the application containers upon notification by Kubernetes/orchestration agent 310. CNI 312 is responsible for assigning IP addresses to these interfaces. One end of the veth pair will terminate at the interface of the application container. As for the other end, CNI 312 will request virtual router 206A to begin monitoring this end of the veth interface. This helps the DPDK forward all tunnel traffic going from the physical port to the application container without involving the core 380. Finally, the CNI 312 will inform the pod to begin using the DPDK/memory mapped interface.

However, these interfaces are not visible from the kernel 280 because the virtual router 206A now manages one end of these veth interfaces. Thus, these interfaces are not visible to cRPD 324, and thus cRPD 324 cannot announce reachability information to the outside world. To address this problem, cRPD 324 is made visible with a veth equivalent interface. This would not be the interface through which cRPD 324 can run routing protocols (as it would require the use of kernel facilities as sockets, TCP/IP stacks, etc.). This interface is used to inform cRPD 324 of the reachability it needs to advertise.

In some cases, virtual router 206A may directly notify cRPD 324 of the interface. This may be preferable because it is similar in some ways to the way the current VRF is handled in cRPD 324. In addition, virtual router 206A may notify cRPD 324 if the interface fails. If cRPD starts, virtual router 206A may let cRPD know all interfaces it monitors again.

Through these interfaces, cRPD 324 may advertise the reachability of MPLS to the application container. The cRPD 324 may advertise vrf-table-label or per next hop tags (where the next hop represents a veth equivalent) or per prefix tags. When the MPLS route may be installed to virtual router 206A, virtual router agent 314 will have the ability to translate the veth equivalent to an actual veth interface.

The following is a further example sequence of operations between various components in accordance with one or more aspects of the present disclosure:

I. interactions between components for creating initial connectivity

Crpd learns vhost0 from the kernel through netlink.

2. The domain controller configures (IGP and BGP) protocol configurations on cRPD via Netconf. Alternatively, the operator may perform this operation manually using CLI on cRPD.

Crpd establishes IGP adjacencies and learns network reachability and segment routing information.

Crpd programs this reachability information to the host kernel through the existing netlink channel.

Crpd establishes BGP sessions through IGP-learned connectivity.

BGP learns l3vpn routes through this BGP session.

Crpd learns the workload interface from the virtual router. The cRPD creates a subnet (e.g./30) and an interface route (/ 32) corresponding to the interface.

Cni configures the workload interface under a specific vrf on cRPD.

Crpd sends vrf interface mappings to virtual routers.

Crpd imports the l3vpn routes received in step 6 to the corresponding VRFs and parses them through the SR tunnel in step 3.

Crpd installs these tunnel routes (in the vrf. Inet (6). 0 table) to the virtual router. (virtual router needs to make a translation of vhost0 to physical port 0).

12. Furthermore, cRPD advertises the l3vpn route for the VRF route in step 7.

Crpd installs mpls.0 route with pop tag and forwarding to workload semantics.

Interaction between the various components when the workload interface fails.

1. The virtual router notifies cRPD interface deletion.

Crpd deletes subnets and interface routes.

Crpd sends reachability revocation to workload.

Crpd deletes the mpls.0 tunnel route with pop and forward to workload semantics from the virtual router.

Interaction between components at the time of vrf deletion.

Crpd deletes internally the l3vpn route received from the corresponding VRF in step 6 (interaction i.).

Crpd sends the deletion of these routes (in the vrf. Inet (6) 0 table) where the tunnel next hops to the virtual router.

3. In addition, cRPD withdraws the l3vpn route of the VRF route from step 8 (interaction i.).

Crpd sends to the virtual router a delete with pop label and mpls.0 route forwarded to the workload semantics.

In an aspect of the present disclosure, to provide high availability of network connectivity, CNI 312 may also add a second backup interface to the application pod when adding the DPDK interface to the application pod instantiated on the compute node. The backup interface may be configured on a backup data plane within the computing node that is different from the active data plane on which the active interface is configured. For example, the active data plane may be a DPDK-based virtual router and the backup data plane may be a kernel-based virtual router, similar to server 350, but with a kernel-based virtual router in addition to DPDK virtual router 206A.

High availability of DPDK forwarding plane

Use of backup interfaces for pods using different forwarding planes

DPDK supports building applications that can bypass the kernel for packet I/O. Applications can send/receive packets directly from the NIC and can achieve high performance by using polling. Bypassing the kernel for packet I/O results in better performance (due to reduced number of context switches, duplication of packet content, and infeasibility/non-ideal polling modes in the kernel).

The DPDK virtual router 206A will own/take over one or more (physical) network ports on the system. The kernel 280 will not be able to utilize these ports for normal network I/O as the virtual router 206A does.

In Kubernetes (K8 s) clusters, DPDK applications run within pods. K8s is responsible for the orchestration (lifecycle management) of these pods. Since these applications in the pod require network connectivity, K8s uses a component called CNI to set up network interfaces, IP address assignment and routing.

After the DPDK pod is started (interface added by CNI), it may happen that the DPDK data path is not available for some reason (Datapath has crashed, is restarting or is being upgraded/maintained). Applications requiring high network availability, applications pods with fallback or alternative traffic forwarding methods are desired.

In order to provide high availability of network connectivity, CNI 312 will also add an additional (backup) interface to each application pod when adding a DPDK interface to the pod, but via a different data plane than the one currently not running.

During this window, the application (or the enhanced DPDK library running as part of the application procedure) will detect that the primary (DPDK) interface has been turned off and switch to using the kernel (backup) interface.

When DPDK virtual router 206A fails to operate due to software problems or ongoing maintenance, the DPDK virtual router 206A physical port can be released back to the core 380. This will allow the core 380 to begin forwarding traffic using these ports until the DPDK virtual router 206A returns and asserts these ports.

To enable this, CNI 312 and/or the routing stack programs the same route into DPDK virtual router 206A and core 380 forwarding table, albeit over different next-hop interfaces. The routing stack may detect that DPDK virtual router 206A is out of service (using a TCP connection between the routing stack and DPDK virtual router 206A) and update the next hop information and call out core-oriented (physical) interface states accordingly.

Similarly, when DPDK virtual router 206A recovers, the routing stack can detect availability and recover routing and interface status so that application pod traffic begins via DPDK virtual router 206A.

This will result in minimal disruption to control plane routing protocol states (such as BGP) by ensuring that core-oriented interfaces previously managed by virtual router 206A are assigned the same interface name and IP address.

In an aspect of the disclosure, the set of software components provides CNI functionality that addresses network requirements specific to the cloud native 5G network environment. The software components include a containerized routing protocol daemon (cRPD) to support network services grid (NSM) architecture. The set of software components support the NSM architecture and may provide additional functionality such as hybrid networking (between physical and virtual infrastructure), direct access to the pods from outside the computing node cluster (e.g., advertised by BGP or like protocols), dynamic tunneling using various techniques such as MPLS, SRv6, IP-IP/VxLAN/GRE, IPsec, etc.

In use cases of this aspect, a 5G O-RAN network may be deployed using cloud-native technology and follow a 5G split, where DUs (distributed units) and CSRs (cell site routers) are virtualized and run on compute nodes. The set of software components may operate as a cell site router to provide L3 reachability for a 5G network's neutral transport.

The software component uses cRPD to distribute layer three (L3) network reachability information for pods not only inside the cluster but also outside the cluster. cRPD also programs the data plane on each compute node. For better network packet I/O performance, DU applications may run in the application pod to bypass the kernel networking stack and abstraction, and thus use, for example, a zero copy mechanism to send/receive packets directly from the physical NIC. A Data Plane Development Kit (DPDK) is one such framework, and DPDK-based virtual routers can be used as a user space data plane that exploits the DPDK to achieve high forwarding performance for this purpose.

The software components may include virtual routers based on DPDK to support DPDK applications. The CNI plug-in manages the DPDK configuration of the application and programs the virtual router. This may include setting up a vhost control channel and assigning IP (e.g., both IPv4 and IPv 6) and MAC addresses, advertising pod IP addresses, and detecting and withdrawing routes when a pod is considered closed or deleted.

Most existing use cases of cRPD provide control plane only routing functions such as BGP route reflectors, or drive forwarding planes: a kernel-based or ASIC-based forwarding plane on a white-box platform. The rise of containers and cloud computing has led to the need for container orchestration platforms to manage the lifecycle of the containers. Kubernetes (K8 s) is an orchestration platform for running containerized applications in clustered computing environments. It provides for automatic deployment, extension, networking, and management of containerized applications. The K8s pod consists of one or more containers representing application instances and is the smallest unit that K8s can handle. All containers in the pod share the same network name space. The Container Networking Interface (CNI) provides a network for application pods in Kubernetes. It is responsible for setting up the pod interfaces, address allocation between pods in the k8s cluster and network isolation between the network and different workloads.

CNI 312 may have CNI functionality and capabilities useful for supporting network services grid (NSM) architecture.

While there are a variety of CNI solutions that primarily cater to data center use cases, the techniques described herein may address network requirements specific to the cloud-native 5G environment by interacting with cRPD 324 to provide NSM functionality. CNIs supporting NSM architecture provide additional functionality such as hybrid networking (between physical and virtual infrastructure), direct access to pods from outside the cluster, for example: tunnels are dynamically set up by advertising through protocols such as BGP, using various technologies such as MPLS, SRv6, IP-IP/VxLAN/GRE, IPsec, etc.

The 5G O-RAN network can be deployed using cloud native technology and follow the 5g 7.2 split, where DUs (distributed units) and CSRs (cell site routers) are virtualized and run on servers. CNI 312 acts as a cell site router to provide L3 reachability for the intermediate transfer.

The cRPD 324 distributes the pod third tier network reachability information not only within (in Kubernetes deployments) the Kubernetes cluster, but also outside the cluster. The cRPD 324 is also responsible for programming the forwarding plane on each computing node/server.

For better network packet I/O performance, DU applications run in the application pods to bypass the kernel networking stack and abstraction, and use a (zero copy) mechanism to send/receive packets directly from the physical NIC. A Data Plane Development Kit (DPDK) is one such framework.

DPDK virtual router 206A is a user space data plane that utilizes DPDK to achieve high forwarding performance. Virtual router 206A supports DPDK applications. The CNI 312 will be responsible for setting up the DPDK configuration for the application and programming the virtual router 206A. This includes setting up the vhost control channel and assigning IP (IPv 4 and IPv 6) and MAC addresses, advertising pod IP addresses, detecting and withdrawing routes when the pod is considered closed or removed.

Other features provided by aspects described in this disclosure include:

● Advertising network reachability of pods using L3 routing protocols such as BGP and IS-IS

● Advertising reachability inside and outside a cluster using BGP

● Network namespaces (supported using L3VPN and EVPN Type-5)

● ACL and network policy for security and QoS

● Support for tunnels: MPLS, SR-MPLS, SRv6, SR-MPLSOPv 6, vxLAN, IPIP, GRE

● Signaling dynamic tunnels using BGP

● IPsec tunnel for traffic encryption

● Network policy for providing security and quarantine

● Integration with DPDK virtual router 206A to achieve higher forwarding performance, encapsulation, packet filtering, and QoS

● The YAML specification file may be used to deploy delivery of a collection of containers in K8 s.

The collection of components that make up CNI 312 and the cloud native router may together be considered a Kubernetes CNI, referred to herein as a Platter CNI.

To meet the objectives of the 5G use case as described with respect to fig. 1, CNI 312 and cloud native router provide the following functions:

● Network namespaces: the application pods should be accessible via a non-default network name space or a routing instance implemented using an L3 VPN.

● IPv6 bottom layer: the IPv6 bottom layer is supported according to the requirement of the use case. The IGP protocol should be able to exchange IPv6 routes. The BGP protocol session should be set up using the IPv6 address.

● IPv6 coverage: IPv6 coverage is supported by assigning IPv6 addresses to pods and advertising them through BGP.

● BGP: plater runs on each node in the k8s cluster and uses BGP to advertise pod reachability to the network. Routes advertised by BGP may carry SRv label stacks or other tunnel encapsulation attributes.

● IGP: each node will participate in the IGP infrastructure to learn the reachability of other BGP peers and route reflectors. IS-IS may be used to advertise host/node addresses to the network.

● SRv6: pod traffic may be carried over SRv6 tunnels. IS-IS used to learn segment routing SID information.

● Virtual router-dpdk: for better packet I/O performance, a virtual router-dpdk is supported as the data plane. This includes assigning IP and MAC addresses, generating the appropriate DPDK configuration for the application, virtual router programming, and advertising routes.

Deployment in the K8s cluster is performed using YAML files containing various detailed information about all containers as CNIs: store of hosted images, initialization order, environment variables, configuration, and license key information. Typically, YAML files must be custom-defined to accommodate K8s deployment. An example YAML configuration of the player CNI (player. Yml) is provided below:

/>

/>

/>

/>

/>

/>

/>

/>

/>

DPDK application configuration

In the case of using kernel networking functionality, CNI 312, cRPD 324, and virtual router 206A set up network interfaces, assign IP addresses, and set up routes, there is no direct interaction with the application as part of the application pod.

Such a user space data plane as virtual router 206A provides network functionality for application pods using non-standard mechanisms. This requires coordination of configuration details between CNI 312, cRPD 324, and virtual router 206A components and pods.

When an application uses DPDK, a Unix Domain Socket (UDS) (referred to as a vhast-user adapter) may be used as a control channel between the application running in the pod and virtual router 206A for negotiating data channels (virtio interface in the pod and vhast interface on virtual router 206A) to transmit packets.

On the host, a configuration file is generated that should be installed by the volume into the application pod at the appropriate location accessible/known to the application. For example:

example dpdk-config. Json profiles are provided below/var/run/cni/player/< pod-id >/dpdk-config- < pod-interface >. Json:

the application pod volume will be installed and create a configuration file as specified by the following parameters in the configmap section of the YAML file:

dpdkConfigBaseDir: "/var/run/cni/player" # maps to paths on hosts in the pod

dpdkConfigFileName:“dpdk-conFIGjson”

The DPDK application may know the location and name of the configuration file. Applications in the pod can access the pod-id as an environment variable. The system will set the rights for the path so that the contents of the directory can be accessed only if the pod-id is known.

Pod YAML configuration

When the components described herein are used as CNIs, to take advantage of advanced functions such as DPDK and VRF support, the application pod YAML configuration should include other detailed information such as environment variables and notes.

Net pod UID

The DPDK application configuration may be stored in the installed volume and path. For security reasons, the path will be inserted into the pod UID and the DPDK application should be aware of the UID.

An example path is as follows:

/var/run/cni/platter/<pod-uid>/dpdk-config-<pod-ifname>.json

Pod YAML should derive pod UID as kubiernetes_pod_uid that may be required for DPDK application.

Annotating

The notes can be used to set the following optional configuration details required for the Plater:

● VRF name: for adding interfaces to route instances

● VRF target: for publishing instance routes through BGP

Configuration example

An example application YAML configuration with environment variables and annotations is shown below:

node configuration

An initial version of the planter would use a statically defined pod network configuration, which is loaded using a configuration map file. This configuration map is read during the Plater CNI installation and stored as a file on each node. The configuration file stores detailed information of each interface of each application, including IP address and routing instance detailed information. Each time the planter CNI is invoked to set the pod interface, the pod name and interface name are used as keys, which find the interface configuration details required to start the interface.

Example node configuration diagram

An example application YAML configuration (player-node-config. YAML) with environment variables and annotations is shown below:

/>

/>

/>

the techniques described herein may be implemented in hardware, software, firmware, or any combination thereof. The various components, functional units, and/or modules illustrated in the figures and/or shown or described elsewhere in the disclosure may use software, hardware, firmware, or a mixture of hardware, software, and firmware residing in and/or executing at one or more computing devices to perform the described operations. For example, a computing device may execute one or more such modules by multiple processors or multiple devices. The computing device may execute one or more such modules as virtual machines executing on underlying hardware. One or more such modules may be executed as one or more services of an operating system or computing platform. One or more such modules may be executed as one or more executable programs at an application layer of a computing platform. In other examples, the functionality provided by the modules may be implemented by dedicated hardware devices. Although certain modules, data stores, components, programs, executable files, data items, functional units, and/or other items may be shown separately included within one or more storage devices, one or more such items may be combined and operated as a single module, component, program, executable file, data item, or functional unit. For example, one or more modules or data stores may be combined or partially combined such that they operate as a single module or provide functionality. Furthermore, one or more modules may operate in conjunction with each other such that, for example, one module acts as a service or extension to another module. Furthermore, each module, data storage, component, program, executable, data item, functional unit, or other item illustrated within a storage device may include multiple components, sub-components, modules, sub-modules, data storage, and/or other components or modules or data storage not illustrated. Furthermore, each module, data storage, component, program, executable, data item, functional unit, or other item shown within a storage device may be implemented in various ways. For example, each module, data store, component, program, executable, data item, functional unit, or other item shown within a storage device may be implemented as part of an operating system executing on a computing device.

If implemented in hardware, the present disclosure may relate to an apparatus such as a processor or an integrated circuit device such as an integrated circuit chip or chipset. Alternatively or additionally, if implemented in software or firmware, the techniques may be realized at least in part by a computer-readable data storage medium comprising instructions that, when executed, cause a processor to perform one or more of the methods described above. For example, a computer-readable data storage medium may store such instructions for execution by a processor.

The computer readable medium may form part of a computer program product, which may include packaging material. The computer-readable medium may include computer data storage media such as Random Access Memory (RAM), read Only Memory (ROM), non-volatile random access memory (NVRAM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory, magnetic or optical data storage media, and the like. In some examples, an article of manufacture may comprise one or more computer-readable storage media.

In some examples, the computer-readable storage medium may include a non-transitory medium. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier wave or propagated signal. In some examples, a non-transitory storage medium may store data (e.g., in RAM or cache) that may change over time.

The code or instructions may be software and/or firmware executed by a processing circuit comprising one or more processors, such as one or more Digital Signal Processors (DSPs), general purpose microprocessors, application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Thus, the term "processor" as used herein may refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. Furthermore, in some aspects, the functionality described in this disclosure may be provided within software modules or hardware modules.

Example 1: a system, comprising: a container workload; a containerized routing protocol daemon comprising processing circuitry and configured to receive routing information from an external network controller; a kernel network stack including processing circuitry and configured to route packets for the container workload based on first routing information; a DPDK-based virtual router including processing circuitry and configured to route packets for the container workload based on second routing information; and a container networking interface plug-in comprising processing circuitry and configured to configure a first virtual network interface for the workload to interface with the DPDK based virtual router and a second virtual network interface for the workload to interface with the kernel network stack.

Example 2: the system of example 1, further comprising: a virtual router agent for the virtual router, the virtual router agent comprising the processing circuitry and configured to receive the second routing information from the containerized routing protocol daemon.

Example 3: the system of example 1, wherein the second routing information includes routing information for an overlay network of the computing infrastructure.

Example 4: the system of example 1, wherein the system operates as a virtualized cell site router for a mobile network.

Example 5: the system of example 1, wherein the workload is a Distributed Unit (DU) for a 5G mobile network.

Example 6: the system of example 1, wherein the system is a single computing node.

Example 7: the system of example 1, wherein the container networking interface plug-in is configured to receive virtual network interface information for the virtual router from a Kubernetes infrastructure.

Example 8: the system of example 1, wherein the system interfaces with Kubernetes infrastructure as a container networking interface.

Example 9: the system of example 1, wherein the routing information comprises segment routing information.

Example 10: the system of example 1, wherein the virtual router agent is configured to interface with a plurality of different types of control planes.

Example 11: a computing device, comprising: a container networking interface plug-in comprising processing circuitry; an orchestration agent comprising processing circuitry, wherein the orchestration agent is an agent for an orchestrator of a computing infrastructure comprising the computing device; a kernel network stack, the kernel network stack comprising processing circuitry; a virtual router comprising a virtual router data plane and a virtual router agent, the virtual router comprising processing circuitry; and a logically related group of one or more containers, the computing device configured to operate to implement a backup network interface for the one or more containers.

Claims (15)

1. A computing device, comprising:

a containerized routing protocol process configured for execution on processing circuitry of the computing device and configured to receive routing information;

a containerized workload set configured for execution on the processing circuitry;

A virtual router of a data plane-based development suite, the virtual router configured for execution on the processing circuitry and configured to forward traffic from the workload based on the routing information from the containerized routing protocol process; and

a virtual router agent for the virtual router, the virtual router agent executing on the processing circuitry and configured to expose a generic data plane interface.

2. The computing device of claim 1, wherein the generic data plane interface comprises a remote procedure call application programming interface.

3. The computing device of claim 1, wherein the virtual router agent is a remote procedure call application programming interface server.

4. The computing device of claim 1, wherein the computing device operates as a virtualized cell site router for a mobile network.

5. The computing device of claim 1, wherein the workload is a distributed unit for a 5G mobile network.

6. The computing device of claim 1, further comprising:

a container networking interface plug-in configured for execution on the processing circuitry and configured to receive virtual network interface information for the virtual router from an orchestrator.

7. The computing device of any of claims 1-6, wherein the computing device interfaces with an orchestrator using a container networking interface for Kubernetes.

8. The computing device of any of claims 1-6, wherein the routing information comprises segment routing information.

9. The computing device of any of claims 1-6, wherein virtual router agent is configured to interface with a plurality of different types of control planes via the generic data plane interface.

10. The computing device of any of claims 1-6, wherein the virtual router agent is configured to receive virtual network interface information from the containerized routing protocol process via the generic data plane interface for configuring the workload with a virtual network interface.

11. The computing device of any one of claim 1-6,

wherein the container networking interface plug-in reserves a block of internet protocol, IP addresses for a workload, allocates IP addresses from the block of IP addresses and configures the workload in the workload, and triggers a Port Add command to the virtual router agent with a network name and the IP address,

Wherein the virtual router agent is configured to send virtual machine interface subscription commands to the containerized routing protocol process, the containerized routing protocol process responding with updates that provide virtual routing and forwarding instance identifiers,

wherein the containerized routing protocol process is configured to assign labels, add routes and next hops to the virtual router, and derive the routes.

12. The computing device of claim 11, wherein the route advertises the workload as reachable at the system.

13. The computing device of any of claims 1-6, wherein the containerized routing protocol process is configured to receive the routing information from an external network controller or from a peer router.

14. A computer networking method performed by the computing device of any of claims 1-13.

15. A computer-readable storage medium encoded with instructions for causing one or more programmable processors to perform the method performed by the computing device of any of claims 1-13.