CN106462469B

CN106462469B - Framework for network technology independent multi-cloud elastic expansion and isolation

Info

Publication number: CN106462469B
Application number: CN201580033476.8A
Authority: CN
Inventors: 马苏姆·Z·哈桑; 路易斯·威利·图克
Original assignee: Cisco Technology Inc
Current assignee: Cisco Technology Inc
Priority date: 2014-06-22
Filing date: 2015-06-12
Publication date: 2020-08-04
Anticipated expiration: 2035-06-12
Also published as: EP3158439A1; CN106462469A; WO2015200012A1

Abstract

An MCEE logical structure is established relating tenant resources of a tenant site, first non-tenant cloud resources at a first non-tenant cloud site, and second non-tenant cloud resources at a second non-tenant site. The MCEE logical structure nodes are mapped to a segmented end-to-end virtual network structure (E2E-VNS) such that resources at each node of the MCEE logical structure are in separate virtual networks of the E2E-VNS. An extension and isolation (EXI) domain is established in the MCEE logical structure that associates at least one node of the tenant resource with at least one node of the first non-tenant cloud and at least one node of the second non-tenant cloud. The E2E-VNS virtual networks of the nodes of the EXI domain are connected for network communications, and the EXI domain is used to isolate the resources of the nodes of the EXI domain from other resources of the MCEE logical structure in the EXI virtual network.

Description

Framework for network technology independent multi-cloud elastic expansion and isolation

RELATED APPLICATIONS

This application claims priority from U.S. provisional patent application No.62/015,514 entitled "a frame For network technology Agnostic Multi-Cloud Elastic Extension And Isolation" filed on 22.6.2014; and this application was continued from the section of U.S. patent application No.14/538,765 entitled "Cloud Framework For Multi-Cloud Extension" filed 11/2014, which claimed the benefit of U.S. provisional patent application No.62/015,516 entitled "An Advanced Cloud Representation and Visual Framework For Multi-Cloud Elastic Extension" filed 22/6/2014. The entire disclosure of each of the above applications is incorporated herein by reference.

Technical Field

The disclosed technology relates to cloud computing, and more particularly, to a framework for elastic expansion of clouds.

Background

"cloud computing" may refer to a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage devices, applications, and services) that can be quickly provisioned and released with minimal administrative work or service provider interaction. The cloud model may be described as consisting of five features, three service models, and four deployment models.

The features include the following.

On-demand self-service — when needed, a customer can automatically unilaterally provide computing capabilities (e.g., server time and network storage devices) without requiring human interaction with each service provider.

Broad network access-capability is available over a network and is accessed through standard mechanisms that facilitate the use of heterogeneous thin client platforms or heterogeneous thick client platforms (e.g., mobile phones, tablets, laptops, and workstations).

Resource pool-the provider's computing resources are centralized to serve multiple customers using a multi-tenant model, and different physical and virtual resources that are dynamically allocated and reallocated according to customer demand. The significance of the position independence is that: customers typically do not have any control or knowledge of the exact location of the provided resources, but can specify locations at a higher level of abstraction (e.g., country, state, or data center). Examples of resources include storage devices, processing, memory, and network bandwidth.

Quick elasticity-the ability can be elastically provided and released (in some cases, automatically) to quickly zoom out and in commensurate with the need. For the customer, the capabilities that can be offered generally appear to be unlimited and can be occupied in any number at any time.

Metering service-cloud systems automatically control and optimize resource usage by leveraging metering capabilities at certain levels of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Typically, this process is done on a pay-per-use basis or a charge-per-use basis. Resource usage can be monitored, controlled, and reported, providing transparency to both the provider and the customer of the utilized service.

The service model may include the following:

software as a service (SaaS). Customers use the provider's applications running on the cloud infrastructure. The cloud infrastructure is a collection of hardware and software that enables five features of cloud computing. The cloud infrastructure can be viewed as combining both physical and abstract layers. The physical layer includes hardware resources necessary to support the provided cloud services and typically includes servers, storage devices, and network components. The abstraction layer includes software deployed across the physical layer that manifests basic cloud characteristics. Conceptually, the abstraction layer is located above the physical layer.

Platform as a service (PaaS). Customer-created or acquired applications created using programming languages, function libraries, services, and tools are deployed on the cloud infrastructure and supported by the provider. This capability does not necessarily preclude the use of compatible programming languages, function libraries, services, and tools from other resources. The customer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage devices, but controls the deployed applications and possible configuration settings of the application hosting environment.

Infrastructure as a service (IaaS). The capabilities provided to the customer are to provide processing, storage, networking, and other basic computing resources, wherein the customer is able to deploy and run arbitrary software, which may include operating systems and applications. Customers do not manage and control the underlying cloud infrastructure, but control the operating system, storage, and deployed applications; and possibly limited control of select network components (e.g., host firewalls).

The deployment model includes the following:

a private cloud. A cloud infrastructure is provided for exclusive use by a single organization that includes multiple customers (e.g., business units). The cloud infrastructure may be owned, managed, and operated by the organization, a third party, or some combination thereof, and may be built-in (on premise) or external (off premise).

A community cloud. The cloud infrastructure may be provided for use by a specific community of users from organizations with shared concerns (e.g., tasks, security requirements, policies, and compliance considerations). The cloud infrastructure may be owned, managed, and operated by one or more of the community, third parties, or organizations in some combination thereof, and may be internal or external.

A public cloud. A cloud infrastructure may be provided for open use by the public. The cloud infrastructure may be owned, managed, and operated by a business, academic, or government organization, or some combination thereof. The cloud infrastructure may exist on the facilities of a cloud provider.

A mixed cloud. A cloud infrastructure is a combination of two or more different cloud infrastructures (private, community, or public) that hold unique entities but are tied together by standardized or proprietary techniques that enable data portability and application portability (e.g., clouds that explode due to load balancing between the clouds).

Drawings

FIG. 1 is a block diagram depicting a communication and processing architecture, in accordance with certain example embodiments;

fig. 2 illustrates a multi-cloud resilient extension (MCEE) structure without extension and isolation (EXI) nodes, and a cloud logical structure mapped to a Virtual Network (VN), according to some example embodiments;

FIG. 3 illustrates an EXI specification with isolation of two segments and propagation through a VN, in accordance with certain example embodiments;

FIG. 4 illustrates the elasticity, on-demand, dynamics, incremental addition, and propagation through VNs of EXI, in accordance with certain example embodiments;

fig. 5 illustrates EXI across two Cloud Service Provider (CSP) domains and propagation through a VN, in accordance with certain example embodiments;

FIG. 6 is a flow diagram depicting a process for multi-cloud elastic expansion and isolation, according to certain example embodiments;

FIG. 7 is a flow diagram depicting a process for multi-cloud elastic expansion and isolation, according to certain example embodiments;

FIG. 8 is a flow diagram depicting a process for multi-cloud elastic expansion and isolation, according to certain example embodiments;

FIG. 9 is a block diagram depicting a computing machine and modules, according to some example embodiments.

Detailed Description

Overview

Embodiments of the technology disclosed herein include network technology independent distributed elastic structures and related operations that allow for the implementation of multi-cloud elastic extensions in network end-to-end regardless of the network technology used in the cloud infrastructure end-to-end.

Example System architecture

In an example architecture of the present technology, although each cloud, server, system, and device shown in the architecture is represented by one instance of a cloud, server, system, or device, multiple instances of each cloud, server, system, or device may be used. Moreover, while certain aspects of the operation of the present technology are presented in the examples associated with the figures to assist in implementing the claimed invention, additional features of the present technology that assist in implementing the claimed invention are also disclosed elsewhere herein.

As shown in fig. 1, fabric 100 includes

computing networks

110, 120, 130, and 140, each including one or more network computing devices, and some networks including a cloudless elasticity extensions (MCEE) engine 150 and seamless cloud resources 160. Each

network

110, 140, each MCEE engine 150, and each seamless cloud resource 160 can be configured to communicate with each other via a communication network 199. In some embodiments, a user associated with a device on a network must install an application and/or make a feature selection to obtain the benefits of the techniques described herein.

For example, network 199 may include one or more of a local area network (L AN), a Wide Area Network (WAN), AN intranet, the Internet, a Storage Area Network (SAN), a Personal Area Network (PAN), a Metropolitan Area Network (MAN), a wireless local area network (W L AN), a Virtual Private Network (VPN), a cellular or other mobile communication network, a network 199, a network device, a network,

Wireless technology connections, Near Field Communication (NFC) connections, any combination thereof, and any other suitable architecture or system that facilitates communication of signals, data, and/or messages. Throughout the discussion of the example embodiments, it should be understood that the terms "data" and "information" are used interchangeably herein to refer to text, images, audio, video, or any other form of information that may be present in a computer-based environment.

Each

network device

110, 140, each MCEE engine 150, and each seamless cloud resource 160 can include a communication module capable of sending and receiving data over network 199. For example, each

network device

110, 140, each MCEE engine 150, and each seamless cloud resource 160 may include a server, a desktop computer, a laptop computer, a tablet computer, a television with one or more processors embedded and/or coupled thereto, a smartphone, a handheld computer, a Personal Digital Assistant (PDA), a gateway, a router, a network bridge, a switch, a hub, and a repeater, or any other wired or wireless processor driven device.

The network connections shown are examples and other means of establishing a communications link between the computer and the device may be used. Moreover, those having ordinary skill in the art and access to the benefit of the present disclosure will appreciate that the network device illustrated in FIG. 1 may have any of several other suitable computer system configurations. For example, a network device embodied as a mobile phone or handheld computer may not include all of the above components.

In certain embodiments, network 110 may be a tenant cloud, each network 120 may be a non-tenant private cloud, each network 130 may be a non-tenant private cloud, and network 140 may include tenant enterprise resources. In such embodiments, each MCEE engine 150 can be operated by an operator of the associated network/resource, and each seamless cloud resource 160 is available and can be allocated to tenant resource requirements.

In an example embodiment, the network computing device and any other computing machines associated with the techniques presented herein may be any type of computing machine, such as (but not limited to) those discussed in more detail with respect to fig. 9. Moreover, any of the modules associated with any of these computing machines, for example, the modules described herein or any other modules associated with the techniques presented herein (scripts, web content, software, firmware, or hardware) may be any of the modules discussed in more detail with respect to fig. 9. The computing machines discussed herein may communicate with each other, as well as with other computer machines or communication systems over one or more networks (e.g., network 199). Network 199 may include any type of data or communication network, including any of the network technologies discussed with respect to fig. 9.

Example embodiments

The example embodiments illustrated in the following figures are described below with respect to example operating environments and components of example structures described elsewhere herein. Example embodiments may also be implemented using other systems or in other environments. For example, MCEE engine 150 associated with tenant resources (e.g., tenant cloud 110 and tenant enterprise resources 140) may be deployed in a stand-alone connection with communications network 199.

The future of cloud computing is towards a multi-cloud environment, where tenants can flexibly extend one or more of their own private clouds 110, select a subset of private cloud sites, and select a subset of resources in those selected sites (referred to herein generally as tenant clouds 110). Such seamless cloud resources 160 can be widely distributed across multiple clouds and constitute the resources of the seamless cloud instance to be acquired by the tenant. The tenant's resources may include non-cloud tenant enterprise resources, such as tenant enterprise resources 140. The MCEE framework securely and seamlessly ties these distributed resources. Tenants may extend their private cloud 110 to any one or more of the following: one or more public clouds, a select subset of public cloud sites, and a select subset of resources in those sites (referred to herein generally as non-tenant clouds 120a.. 120 n). This extension is similar to a hybrid cloud in the multi-cloud context defined by the National Institute of Standards and Technology (NIST).

Tenants may extend their private cloud 110 to any one or more of the following: other private clouds, select subsets of their sites, and select subsets of resources in those sites (referred to herein generally as non-tenant private clouds 130a.. 130n), possibly via one or more public clouds having select subsets of their sites and select subsets of resources in those sites. This extension is similar to the community cloud in the multi-cloud context defined by NIST.

Note that even if a tenant does not have its own private cloud, the tenant may want to extend its enterprise (intranet) 140 into multiple clouds. Given a set of clouds, sites, and resources in those sites, a tenant may want to create multiple instances of such a multi-cloud extension. Tenants, or system operators or other stakeholders may further want to keep each instance isolated from other instances, as well as merge or separate instances as needed based on conditions or requirements.

Implementation of MCEE in infrastructure (implementation in infrastructure or network) becomes challenging due to the reason that a complex potential infrastructure may belong to multiple organizations, etc. complex network E2E may include many segments each of which may include multiple network technologies, e.g., L inux bridging, virtual local area network (V L AN), virtual extensible L AN (VX L AN), Virtual Routing and Forwarding (VRF) -L ite, VRF, multi-protocol label switching (MP L S) Virtual Private Network (VPN), Internet Protocol Security (IPSEC), virtual private L AN service (VP L S), Overlay Transport Virtualization (OTV), etc.

Consider that either tenant-specific cloud 110 or tenant enterprise resources 140 are extended into one public cloud 120a to form a 1: 1 hybrid cloud. The same tenant may extend into another public cloud 120b, but separately. Embodiments of the technology disclosed herein focus on extension into multiple clouds and consider the extension as one single seamless extension.

Embodiments of the techniques disclosed herein include network technology independent distributed elastic structures and related operations that allow for the implementation of multi-cloud elastic extensions in the network end-to-end, regardless of the network technology used in the cloud infrastructure end-to-end.

Tenants can implement MCEE via the following two levels of distributed elastic structures and related operations.

Cloudy Elastic Extended Structure (MCEES): the distributed elastic fabric includes logical representations of built-in and external clouds, cloud sites, site resources, and an extended and isolated representation from a tenant's private cloud 110 or tenant enterprise 140 to two or more other clouds (e.g., non-tenant public cloud 120 and non-tenant private cloud 130). The structure represents a logical topology, where each node in the topology has certain semantics.

E2E virtual network architecture (E2E-VNS): the MCEES is mapped to a network technology independent end-to-end resilient distributed virtual network structure. E2E-VNS has the following properties. The architecture includes multiple virtual network elements that facilitate resiliency (either adding or deleting on demand or dynamically). The fabric facilitates isolation of the cloud extension both locally (in cloud DC) and globally (in MAN/WAN). E2E-VNS facilitates the assignment of a single network domain to a multi-cloud extension instance. For example, a single subnet may be assigned to an instance even if the instance spans multiple network segments including multiple ases.

Resilient mapping to multi-AS MP L S L3 VPN network-E2E-VNS (which is network technology independent) can be mapped to any suitable potential network isolation technology, e.g., VP L S, or L MP L S VPN, or station-to-station IPSEC, AS well AS other isolation technologies, e.g., V L AN or VX L AN or GRE tunnel.

In a cloud environment, each level of cloud infrastructure should be elastically programmable however, elastically (dynamically or on-demand) programming in MP L S L3 VPN networks (widely deployed technologies) that also include multi-AS or multi-Service Provider (SP) MP L S L3 VPN networks is challenging example embodiments of the technology disclosed herein provide a MP L S L3 VPN elastic programming method employing elastic MCEES and E2E-VNS structures and methods thereof, AS an illustration of principles regarding embodiments generally used by isolation technology

Embodiments of the technology disclosed herein (the above structures, operations, and mappings) are distributed. The MCEE has multiple segments corresponding to multiple cloud and network segments. For example, a tenant (controlling tenant cloud 110 and tenant enterprise resources 140) and a first cloud service provider (CSP1 — controlling non-tenant public cloud 120a or non-tenant private cloud 130a) control one segment of the network, and the tenant and a second CSP2 (controlling non-tenant public cloud 120b or non-tenant private cloud 130b) together control a different segment of the network spanned by a single instance of the tenant's MCEE. Thus, their operation and mapping may be distributed. The fabric is defined among and maintained by a plurality of MCEE engines 150, at least one of the plurality of MCEE engines 150 being associated with tenant resources (shown in fig. 1 as part of tenant cloud 110, but also readily deployed as part of tenant enterprise resources 140), and at least one of the plurality of MCEE engines 150 being associated with each non-tenant cloud to be expanded into (shown in fig. 1 as part of each non-tenant cloud 120, 130, but also readily deployed outside of the non-tenant cloud). In an example embodiment, there is one MCEE engine 150 for each non-tenant cloud provider DC. The various MCEE engines 150 operate together in a distributed manner to implement the present techniques.

Multi-cloud elastic expansion structure

The structure described below corresponds to the MCEE of the tenant. This structure grows or shrinks elastically as tenants elastically expand/extract to/from multiple clouds or their sites (or even independent resources). A particular tenant may have multiple instances of such structure corresponding to multiple MCEEs. The logical structure and semantics of the MCEE network include logical nodes and edges that logically connect the nodes.

Referring to fig. 2 and with continuing reference to fig. 1 in context, a multi-cloud resilient extensions (MCE) architecture without extended isolation (EXI) nodes, and a cloud logical architecture 200 mapped to a Virtual Network (VN) are shown in accordance with an example embodiment. The MCEE engine 150, distributed across managed resources, can establish, operate, and maintain MCEES and E2E-VNS.

In an MCEES cloud logical structure (C L S), MCEE engine 150 (distributed across tenants 'resources and the DC sites of CSP1 and CSP 2) may establish one or more seamless administrative domain nodes 210, logical nodes for managing built-in resources (ONP)220 (e.g., some or all of the resources of each of tenant' S private cloud 110 and tenant 'S enterprise resources 140), one or more logical nodes for managing extensions to each built-in site (ONS)230, and one or more logical nodes for managing extensions to each built-in site' S resources (ONR) 240.

The MCEE engine 150 can establish logical nodes for managing external resources (OFPs) 260 (e.g., some or all of the resources of each of the non-tenant public cloud 120a and the non-tenant private cloud 130a), one or more logical nodes for managing extensions to each external site (OFS)270, and one or more logical nodes for managing extensions to each external site's resources (OFRs) 280. In a similar manner, the MCEE engine 150 can establish edges 250, each edge 250 representing a logical connection. For example, MCEE engine 150 uses edge 250 to logically connect each OFP260 to one or more OFSs 270, and uses edge 250 to logically connect each OFS270 to one or more OFRs 280.

If the tenant creating the MCEE instance desires a single network domain for the entire MCEE, the tenant uses the tenant's MCEE engine 150 to specify a domain addressing scheme, e.g., a private IP address prefix.

E2E virtual network architecture (E2E-VNS)

With continuing reference to fig. 2 and with reference to fig. 1 in the context, MCEE engine 150 may map MCEEs logical structures to network-independent Virtual Networks (VNs) and logical topologies containing such VNs MCEE engine 150 may establish built-in site virtual networks (ON-SVN)232 (e.g., each of V1-V3 and V9) corresponding to ONS 230 each ON-SVN 232 may be characterized by a site identifier (SiteID) and an identifier (SiteID. resourcesetid) of each resource 240 connected to ON-SVN 232 each ON-SVN 232 may be mapped to an address from a desired domain if single domain addressing (e.g., private IP address) is desired each ON-SVN 232 may be mapped to an address from a desired domain each ON-SVN 232 may use network address translation (cvnat) at a VN for example MCEE engine 150 may establish built-in cloud virtual networks (ON-n) 222 (e.g., each of V4 and V10) corresponding to an embedded cloud virtual network (ON-n) 150 may be connected to an MCEE edge mapping (cvon-SVN) 150 may be used by a MCEE engine 150 as a potential VPN interface map an example cvon network edge mapping technique 250 between a pair of MCEE-SVN 150.

The MCEE engine 150 establishes an external site virtual network (OF-SVN)272 (e.g., V6-V8, V12) corresponding to the OFS 270. Each OF the OF-SVN 272 may be characterized by a site identifier (SiteID) and an identifier (SiteID. resourcesetid) for each resource 280 connected to the OF-SVN 272. If single domain addressing is desired (e.g., private IP address), then each OF-SVN 272 is mapped to an address from the desired domain. For example, the address will be translated using Network Address Translation (NAT) at the VN. MCEE engine 150 can establish an external cloud virtual network (OF-CVN)262 (e.g., V5, V11) corresponding to OFP 260. Each OF-CVN 262 may be assigned a cloud ID (CloudID). The MCEE engine 150 can establish edges 252, each edge 252 representing a logical connection. For example, the MCEE engine 150 uses the edge 252 to logically connect each OF-CVN 262 to one or more OF-SVNs 272. The MCEE engine 150 maps each such connection based on the potential network isolation technique used by the resource.

MCEE engines 150 of resources at each site (e.g., tenant 'S site, and CSP 1' S site) may map a set of resources to the virtual network based on a particular network isolation technique (e.g., V L AN, VX L AN, or MP L S L3 VPN), thus providing semantics in the network for the set of resources.

Each

edge

250, 252 corresponds to two directions of reachability to other VN cells when two VNs (e.g., V1 and V2) are "connected," MCEE engines 150 exchange node-related information between them, thus, when a node attached to V1 and a node attached to V2 are mapped to a particular network technology, the node attached to V1 becomes reachable to a node attached to V2, and vice versa.

Multi-cloud extended functionality

Referring to fig. 3 and with continued reference to the previous figures for context, the MCEES of fig. 2 with two extension and isolation (EXI) nodes (EXI1, EXI2), and the corresponding E2E-VNS 300 are shown in accordance with certain example embodiments. In such embodiments, tenants may extend their resources in the context of the MCEES and E2E-VNS of fig. 2 to the resources of the cloud service provider (CSP1) and CSP2 as follows

Extension-1-distributed MCEE engine 150 (one associated with the tenant's resources and one associated with the first site of CSP1) can use EXI1 to connect ONS 230a and ONS 230b to OFP260 a of CSP1 based on user input, thereby extending the tenant's built-in CA-SJ resources into the external public cloud of CSP 1.

Extension-2-distributed MCEE engine 150 (one associated with the tenant's resources and one associated with the CSP2 site) can use EXI2 to connect ONP220 a and OFP260 b according to user input specifying the node to connect, thereby extending the tenant's built-in TX resources into the private cloud of CPS 2.

The multi-cloud extended functionality is mapped to a virtual network. For each new EXI, a virtual network is created. Each such VN logically spans multiple network domains. Such as internal and external or multiple CSP network domains. The semantics are the same as those described with respect to E2 EVNS. In the case of FIG. 3, EXI-VN V13 and V14 are created. V13 spans the domain of tenant's CA-SJ resources and all CSP 1's resources in the MCEE.

Resilient mapping to multi-AS MP L S L3 VPN network

The following processes, along with distributed MCEES and E2E-VNS, provide a resilient layer on top of the multi-AS MP L S L3 VPN network when the MP L S L3 VPN mapping function is turned on in the MCEE engine 150, the following operations are performed resiliently when nodes and edges are dynamically added (deleted) AS needed.

Referring to fig. 4 and with continuing reference to the preceding figures OF the context, the addition OF an ONS or OFS to an existing set OF extended and disjoint resources 400 is shown according to an example embodiment, in particular, the following description, in conjunction with fig. 4, shows the addition OF resources OF an ONS 230b to an isolated and extended domain EX1 linking an ONS 230a and an OFP260 a by establishing an Eg1 in the MCEES and an Eg2 in the E2E-VNS, when an ONS or OFS such as an ONS 230b is selected for addition to the extended and isolated domain, the MCEE engine 150 creates a SiteID and a set OF SiteID. resourcesetid OF resource nodes connected to the ONS 230b, if specified, the MCEE engine 150 assigns a specified single domain address (SDA, e.g., a private IP address or prefix) to these nodes, and an address translation module (e.g., NAT) is created and associated with this site engine 150 creates a new path identifier (cvpe) corresponding to a specific MP identifiable prefix or prefix 64 OF the MCEE MP 120 b-VPN) and provides the new path identifier (cvpe — VPN ID — an ecpe) to the site engine 150-wo 32-csee engine 150 and adds the new path identifier (cvpe) to the MCEE engine 150.

Referring to fig. 5 and with continuing reference to the previous figures of the context, adding extensions and isolation domains 500 is shown according to an example embodiment, in particular, modifying the MCEES and E2E-VNS of fig. 4 by introducing an EXI3 connecting the extensions and isolation domains of EXI1 and EXI2 at the MCEES and by changing the E2E-VNS after creating EXI3, the MCEE engine 150 creates an EXI virtual network, in which case the v15. the MCEE engine 150 creates an MP L S L3 VPN routing and forwarding (vrvn) table that the MCEE engine 150 attaches to VN15. the MCEE engine 150 propagates the (cloudi, resotid, ecrtd, SDA, CEID, PEID) created at the constituent VN to the vn15. if there is no VPN f for the constituent MCEE engine 150, the MCEE engine 150 writes the (cloudi, resotid, ecold in the vnm, the MCEE engine 150 outputs from the VRFs.

With respect to the MCEE engine 150, the MCEE engine for tenant resources may reside on a tenant facility. The tenant MCEE engine builds and maintains a global view of the MCEE, which is shown at the top of each of fig. 2-5. The tenant MCEE engine also maps the built-in segment of MCEE to the built-in segment of E2E-VNS. E2E-VNS is shown at the bottom of each of fig. 2-5. In the example figure, E2EVNS segment V4 (along with V1, V2, V3) and V10 (and V9) reside in the tenant MCEE engine. Shared VNs (V13, V14, V15) corresponding to EXI also reside in the tenant MCEE engine. This engine, which maintains a global view of MCEEs across multiple clouds and cloud sites, is coordinated among multiple MCEE engines (e.g., an MCEE engine associated with the CSP1 cloud and an MCEE engine associated with the CSP2 cloud). In the examples of fig. 2-5, one MCEE engine is associated with the CSP1 cloud and another MCEE engine is associated with the CSP2 cloud. Cloud-specific VNs (e.g., V5, V6, V7, and V8 of CSP1) and shared EXI VNs (V13, V14, V15) also reside in each cloud VN. The expansion and contraction is performed automatically, dynamically, and elastically through a combined set of MCEE engines based on the demand of resources from tenants for cloud services.

Referring to fig. 6 and with continuing reference to the previous figures in context, a process 600 of cloudy elastic expansion and isolation is shown in accordance with certain example embodiments. In such embodiments, a set of MCEE engines 150 establishes an MCEE logical structure relating to at least one tenant resource of at least one tenant site, at least one resource of a first non-tenant cloud at a first non-tenant cloud site, and at least one resource of a second non-tenant cloud at a second non-tenant cloud site-block 610.

In particular, establishing the MCEE logical structure may include: the tenant resources (e.g., tenant cloud 110), the resources of a first non-tenant cloud (e.g., non-tenant public cloud 120), and the resources of a second non-tenant cloud (e.g., non-tenant private cloud 130) are associated in a hierarchical tree topology by resource operators, resource operator sites, and a set of resources at each resource operator site as shown in fig. 2.

The architecture of FIG. 2 associates the built-in resources 240 of tenants at san Jose, Calif. (CA-SJ), los Angeles, Calif. (CA-L A), and Las Vegas, Nevada (NV-L V) with the built-in resources 240 of tenants, Texas, and the external resources of two non-tenant cloud providers.

The set of MCEE engines 150 may map the nodes of the MCEE logical structure to a segmented end-to-end virtual network structure (E2E-VNS) such that the set of resources at each node of the MCEE logical structure is in a separate virtual network of E2E-VNS-block 620. In particular, mapping the MCEE logical structure to the E2E-VNS may include: the mapping is performed such that at each level of E2E-VNS, the set of resources at each node of the MCEE logical structure is in a separate virtual network of that E2E-VNS. For example, in FIG. 3, V1-V4, V5-V8, V9-V10, and V11-V12 are in separate E2E-VNS virtual networks.

The set of MCEE engines 150 can establish an extension and isolation (EXI) domain in the MCEE logical structure that associates at least one node of the tenant resource with at least one node of the first non-tenant cloud and at least one node of the second non-tenant cloud-block 630. FIG. 3 illustrates EXI domain EXI1 associating resources of tenants at CA-SJ with resources 270 of CSP 1. FIG. 3 also shows EXI domain EXI2 associating built-in resources 220a of a group of tenants with resources 270 of CSP 2. Note that EXI2 may have connected the TX resources of the site-level tenant with the resources of the site-level CSP2 to achieve the same effect. In embodiments employing a hierarchical tree structure for the MCEE structure, establishing the EXI domain may include associating at least one node of the tenant resource portion of the hierarchy with at least one node of the first non-tenant cloud portion of the hierarchy and at least one node in the second non-tenant cloud portion of the hierarchy.

The set of MCEE engines 150 may connect the E2E-VNS virtual network of nodes of the EXI domain for network communications, the EXI domain to isolate resources of the nodes of the EXI domain from other resources of the MCEE logical structure in the EXI virtual network — block 640. In the example of FIG. 3, MCEE engine 150 has used V13 to connect the resources of V1 (the tenant's CA-SJ resources) and CSP1 resources.

Referring to fig. 7 and with continuing reference to the preceding figures in context, a process 700 of multi-cloud resilient extension and isolation is illustrated according to certain example embodiments in such embodiments, although block 630 may be performed as described above, the set of MCEE engines may establish MCEE logical structures (e.g., block 610) relating to tenant resources and resources of two non-tenant clouds, wherein the network isolation technique of the first non-tenant cloud is a multi-protocol label switching layer 3 virtual private network (MP L S L3 VPN) network isolation technique — block 710.

In such embodiments, mapping the MCEE logical structure to the E2E-VNS includes several processes for each first non-tenant cloud site virtual network of the E2E-VNS. In such embodiments, the set of MCEE engines 150 can assign a site identifier to each first non-tenant cloud site virtual network and a resource identifier to each resource of each first non-tenant cloud site virtual network-block 721.

In the case where the network isolation technology of the first non-tenant cloud is a multi-protocol label switching layer 3 virtual private network (MP L S L3 VPN) network isolation technology, the set of MCEE engines 150 may create an MP L S L3 VPN extended community path target (ECRT) identifier for each first non-tenant cloud site virtual network-block 722.

The cloud provider infrastructure may include both Customer Edge (CE) routers and Provider Edge (PE) routers. In the illustrated embodiment of fig. 7, as part of mapping the MCEE logical structure to the E2E-VNS, the set of MCEE engines 150 can assign a CE identifier to each CE router attached to each first non-tenant cloud site virtual network and a PE identifier to each PE router attached to each first non-tenant cloud site virtual network — block 723. The set of MCEE engines 150 may then communicate each identifier to a corresponding first non-tenant cloud virtual network in the E2E-VNS — block 724.

After establishing an EXI domain in the MCEE that associates at least one node of the tenant resource with at least one node of the first non-tenant cloud and at least one node of the second non-tenant cloud (block 630), the MCEE engine 150 may connect the E2E-VNS virtual networks of the nodes of the EXI domain for network communication.

Although the example of fig. 7 involves the MP L S L3 VPN network isolation technique of the first non-tenant cloud, the set of MCEE engines 150 may implement protocols of other network protection techniques for connecting E2E-VNS virtual networks of nodes of the EXI domain, thereby resource isolation from nodes of other resources.

Referring to fig. 8 and with continuing reference to the previous figures in context, a process 800 of cloudy elastic expansion and isolation is shown in accordance with certain example embodiments. In such embodiments, blocks 610-640 may be performed as described above. In such an approach, after connecting the E2E-VNS virtual networks of the nodes of the EXI domain (for network communications and for isolating the resources of these nodes from other resources), the set of MCEE engines 150 may receive a request to increase the available resources in the EXI domain — block 850. For example, where the cloud resources are storage and the tenant provides additional storage through self-service, the set of MCEE engines 150 may receive a request to extend the tenant's cloud 110 to new cloud resources.

Upon receiving the request, the set of MCEE engines 150 may map at least one additional non-tenant cloud resource to the E2E-VNS of the EXI domain — block 860. Once mapped to the E2E-VNS of the EXI domain, the MCEE engine can connect the E2E-VNS virtual network of the mapped resources to the EXI domain-block 870.

Other exemplary embodiments

Fig. 9 depicts a computing machine 2000 and a module 2050, according to some example embodiments. The computing machine 2000 may correspond to any one of the following: various computers, servers, mobile devices, embedded systems, or computing systems presented herein. The module 2050 may include one or more hardware or software elements configured to assist the computing machine 200 in performing the various methods and processing functions set forth herein. The computing machine 2000 may include various internal or attached components, such as a processor 2010, a system bus 2020, a system memory 2030, a storage medium 2040, an input/output interface 2060, and a network interface 2070 for communicating with a network 2080.

The computing machine 2000 may be implemented as a conventional computer system, an embedded controller, a laptop computer, a server, a mobile device, a smartphone, a set-top box, a kiosk, a vehicle information system, one or more processors associated with a television, a customized machine, any other hardware platform, or any combination or multiple combinations thereof. The computing machine 2000 may be a distributed system configured to operate with multiple computing machines interconnected via a data network or bus system.

The processor 2010 may be a general purpose processor, a processor core, a multiprocessor, a reconfigurable processor, a microcontroller, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Graphics Processing Unit (GPU), a Field Programmable Gate Array (FPGA), a programmable logic device (P L D), a controller, a state machine, gated logic, discrete hardware components, any other processing unit, or any combination or multiple combinations thereof.

The system memory 2030 may include a non-volatile memory such as a Read Only Memory (ROM), a Programmable Read Only Memory (PROM), an Erasable Programmable Read Only Memory (EPROM), a flash memory, or any other device capable of storing program instructions or data with or without input power. The system memory 2030 may also include volatile memory such as Random Access Memory (RAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), and Synchronous Dynamic Random Access Memory (SDRAM). Other types of RAM may also be used to implement system memory 2030. The system memory 2030 may be implemented using a single memory module or a plurality of memory modules. Although the system memory 2030 is depicted as being part of the computing machine 2000, those skilled in the art will recognize that the system memory 2030 may be separate from the computing machine 2000 without departing from the scope of the subject technology. It should also be appreciated that the system memory 2030 may include, or operate in conjunction with, a non-volatile storage device, such as the storage media 2040.

The storage medium 2040 may include a hard disk, a floppy disk, a compact disk read only memory (CD-ROM), a Digital Versatile Disk (DVD), a blu-ray disk, a magnetic tape, a flash memory, other non-volatile memory devices, a Solid State Disk (SSD), any magnetic storage device, any optical storage device, any electrical storage device, any semiconductor storage device, any physical-based storage device, any other data storage device, or any combination or multiple combinations thereof. The storage media 2040 may store one or more operating systems, application programs and program modules (e.g., the module 2050), data, or any other information. The storage medium 2040 may be part of the computing machine 2000, or be connected to the computing machine 2000. The storage media 2040 may also be part of one or more other computing machines in communication with the computing machine 2000, e.g., a server, a database server, a cloud storage device, a network-connected storage device, etc.

The module 2050 may include one or more hardware or software elements configured to assist the computing machine 2000 in performing the various methods and process functions set forth herein the module 2050 may include one or more sequences of instructions stored as software or firmware associated with the system memory 2030, the storage medium 2040, or both the storage medium 2040 may thus represent an example of a machine or computer readable medium on which instructions or code for execution by the processor 2010 may be stored the machine or computer readable medium may generally refer to any medium or media for providing instructions to the processor 2010 the machine or computer readable medium associated with the module 2050 may include a computer software product, it being understood that a computer software product including the module 2050 may also be associated with one or more processes or methods for communicating the module 2050 to the computing machine 2000 via the network 2080, any signal bearing medium, or any other communication or communication technique the module 2050 may also include hardware circuitry or information for configuring hardware circuitry, such as micro code for an FPGA or other P L D configuration information.

The input/output (I/O) interface 2060 may be configured to couple one or more external devices to receive data from the one or more external devices and to transmit data to the one or more external devices. Such external devices, along with various internal devices, may also be referred to as peripheral devices. The I/O interface 2060 may include both electrical and physical connections for operatively coupling various peripheral devices to the computing machine 2000 or the processor 2010. The I/O interface 2060 may be configured to transfer data, addresses, and control signals between a peripheral device, the computing machine 2000, or the processor 2010. The I/O interface 2060 may be configured to implement any standard interface, such as Small Computer System Interface (SCSI), serial SCSI (sas), fibre channel, Peripheral Component Interconnect (PCI), PCI express (pcie), serial bus, parallel bus, Advanced Technology Attachment (ATA), serial ATA (sata), Universal Serial Bus (USB), Thunderbolt (Thunderbolt), FireWire (FireWire), various video buses, and the like. The I/O interface 2060 may be configured to implement only one interface or bus technology. Alternatively, the I/O interface 2060 may be configured to implement multiple interface or bus technologies. The I/O interface 2060 may be configured as part of the system bus 2020, all, or operate in conjunction with the system bus 2020. The I/O interface 2060 may comprise one or more buffers for buffering transmissions between one or more external devices, internal devices, the computing machine 2000, or the processor 2010.

The I/O interface 2060 may couple the computing machine 2000 to various input devices including a mouse, a touch screen, a scanner, an electronic digitizer, a sensor, a receiver, a touchpad, a trackball, a camera, a microphone, a keyboard, any other pointing device, or any combination thereof. The I/O interface 2060 may couple the computing machine 2000 to various output devices including video displays, speakers, printers, projectors, haptic feedback devices, automation controls, robotic components, actuators, motors, fans, solenoids, valves, pumps, transmitters, signal transmitters, lights, and the like.

The computing machine 2000 may operate in a networked environment using logical connections to one or more other systems or computing machines through a network interface 2070 to a network 2080 the network 2080 may include a wide area WAN, a local area network (L AN), AN intranet, the Internet, a wireless access network, a wired network, a mobile network, a telephone network, AN optical network, or a combination thereof the network 2080 may be packet-switched, circuit-switched in any topology and may use any communication protocol the communication links in the network 2080 may involve various digital or analog communication media, such as fiber optic cables, free space optical systems, waveguides, electrical conductors, wireless links, antennas, radio frequency communications, and so forth.

The processor 2010 may be coupled to the computing machine 2000 or other elements of the various peripheral devices discussed herein by a system bus 2020. It is to be appreciated that the system bus 2020 can be internal to the processor 2010, external to the processor 2010, or both. According to certain example embodiments, any of the processor 2010, other elements of the computing machine 2000, or various peripherals discussed herein may be integrated in a single device, such as a system on a chip (SOC), a System On Package (SOP), or an ASIC device.

Embodiments may include a computer program embodying the functionality described and illustrated herein, wherein the computer program is implemented in a computer system comprising instructions stored in a machine-readable medium and a processor executing the instructions. It should be understood, however, that there may be many different ways of implementing embodiments in computer programming, and embodiments should not be construed as limited to any one set of computer program instructions. Furthermore, a skilled programmer would be able to write such a computer program to implement embodiments of the disclosed embodiments based on the accompanying flow charts and associated description in the application text. Therefore, disclosure of a particular set of program code instructions is not necessarily considered necessary for an adequate understanding of how to make and use the embodiments. Furthermore, those skilled in the art will appreciate that one or more aspects of the embodiments described herein, as may be embodied in one or more computing systems, may be performed by hardware, software, or a combination thereof. Moreover, any reference to an action being performed by a computer should not be construed as being performed by a single computer, as more than one computer may perform the action.

The example embodiments described herein may be used with computer hardware and software that performs the methods and processing functions described above. The systems, methods, and programs described herein may be embodied in a programmable computer, computer-executable software, or digital circuitry. The software may be stored on a computer readable medium. For example, the computer readable medium may include floppy disks, RAM, ROM, hard disks, removable media, flash memory, memory sticks, optical media, magneto-optical media, CD-ROMs, and the like. The digital circuitry may comprise integrated circuits, gate arrays, building logic, Field Programmable Gate Arrays (FPGAs), etc.

The example systems, methods, and acts described in the foregoing embodiments are illustrative, and in alternative embodiments some acts may be performed in a different order, in parallel with each other, omitted entirely, and/or combined between different example embodiments, and/or some additional acts may be performed, without departing from the scope and accuracy of the various embodiments. Accordingly, such alternate embodiments are included within the scope of the following claims, which are to be accorded the broadest interpretation so as to encompass such alternate embodiments.

Although specific embodiments have been described in detail above, the description is for illustrative purposes only. Thus, it should be understood that many of the aspects described above are not intended as required or essential elements unless explicitly described as such.

Modifications of various aspects of the disclosed example embodiments, in addition to those described above, as well as equivalent components or actions corresponding to various aspects of the disclosed example embodiments having the benefit of the present disclosure, may be made by those skilled in the art without departing from the spirit and scope of the embodiments as defined by the following claims, the scope of which is to be accorded the broadest interpretation so as to encompass such modifications and equivalent structures.

Claims

1. A method for multi-cloud elastic expansion and isolation, comprising:

establishing a multi-cloud resilient extension (MCEE) logical structure in a set of MCEE engines, the MCEE logical structure relating to at least one tenant resource of at least one tenant site, at least one resource of a first non-tenant cloud at a first non-tenant cloud site, and at least one resource of a second non-tenant cloud at a second non-tenant cloud site, wherein each of the tenant site, the first non-tenant cloud, and the second non-tenant cloud hosts an MCEE engine in the set of MCEE engines;

mapping, by the set of MCEE engines, the nodes of the MCEE logical structure to a segmented end-to-end virtual network structure (E2E-VNS) such that a set of resources at each node of the MCEE logical structure are in separate virtual networks of the E2E-VNS;

establishing, by the set of MCEE engines, an extended and isolated (EXI) domain in the MCEE logical structure that associates at least one node of the tenant resource with at least one node of the first non-tenant cloud and at least one node of the second non-tenant cloud; and

connecting, by the set of MCEE engines, the E2E-VNS virtual networks of nodes of the EXI domain for network communications, the EXI domain to isolate resources of the nodes of the EXI domain from other resources of the MCEE logical structure in the EXI virtual network.

2. The method of claim 1, wherein,

establishing the MCEE logical structure comprises the following steps: associating the at least one tenant resource, the at least one resource of the first non-tenant cloud, and the at least one resource of the second non-tenant cloud in a hierarchical tree topology through resource operators, resource operator sites, and a set of resources at each resource operator site;

mapping the MCEE logical structure to an E2E-VNS comprises: performing mapping such that at each level of the E2E-VNS, the set of resources at each node of the MCEE logical structure are in separate virtual networks of the E2E-VNS; and

establishing the EXI domain includes: establishing an EXI domain in the MCEE logical structure that associates at least one node of the tenant resource portion of the hierarchy with at least one node of a first non-tenant cloud portion of the hierarchy and at least one node in a second non-tenant cloud portion of the hierarchy.

3. The method of claim 2, wherein,

mapping the MCEE logical structure to the E2E-VNS comprises: performing, for each first non-tenant cloud site virtual network of the E2E-VNS:

assigning a site identifier to each first non-tenant cloud site virtual network;

assigning a resource identifier to each resource of each first non-tenant cloud site virtual network;

assigning a Customer Edge (CE) identifier to each CE router connected to each first non-tenant cloud site virtual network;

assigning a Provider Edge (PE) identifier to each PE router connected to each first non-tenant cloud site virtual network;

transmitting each identifier to a respective first non-tenant cloud virtual network in the E2E-VNS; and

connecting the E2E-VNS virtual networks of nodes of the EXI domain for network communication includes: transmitting, from each E2E-VNS virtual network connected to the EXI virtual network, each identifier of each E2E-VNS virtual network connected to the EXI virtual network.

4. The method of claim 2, wherein,

the network isolation technology of the first non-tenant cloud is a multi-protocol label switching layer 3 virtual private network (MP L S L3 VPN) network isolation technology, and

creating an MP L S L3 VPN extended community path target (ECRT) identifier for each first non-tenant cloud site virtual network;

connecting E2E-VNS virtual networks of nodes of the EXI domain for network communication includes creating MP L S L3 VPN Virtual Routing and Forwarding (VRF) table instances of the EXI virtual network, attaching the VRF table instances to the EXI virtual network, and transmitting each identifier of each E2E-VNS virtual network connected to the EXI virtual network from each E2E-VNS virtual network connected to the EXI virtual network.

5. The method of claim 4, wherein mapping the MCEE logical structure to the E2E-VNS further comprises: for each first non-tenant cloud site virtual network of the E2E-VNS, assigning a single domain address to each first non-tenant cloud site virtual network.

6. The method of claim 1, wherein,

the at least one tenant resource comprises a tenant cloud operating under a first network isolation technology, and the first non-tenant cloud comprises a cloud operating under a second network isolation technology different from the first network isolation technology.

7. The method of claim 1, further comprising:

receiving, by the set of MCEE engines, a demand-based request requesting an increase in available resources in the EXI domain;

upon receiving the request, mapping, by the set of MCEE engines, at least one additional non-tenant cloud resource to an E2E-VNS of the EXI domain; and

connecting the E2E-VNS virtual network of the mapped resource to the EXI domain through the set of MCEE engines.

8. One or more computer-readable storage media encoded with software, the software comprising computer-executable instructions operable when executed to perform the steps of:

establishing a multi-cloud resilient extension (MCEE) logical structure related to at least one tenant resource of at least one tenant site, at least one resource of a first non-tenant cloud at a first non-tenant cloud site, and at least one resource of a second non-tenant cloud at a second non-tenant cloud site, wherein each of the tenant site, the first non-tenant cloud, and the second non-tenant cloud hosts an MCEE engine of a set of MCEE engines;

mapping nodes of the MCEE logical structure to a segmented end-to-end virtual network structure (E2E-VNS) such that a set of resources at each node of the MCEE logical structure are in separate virtual networks of the E2E-VNS;

establishing an extension and isolation (EXI) domain in the MCEE logical structure that associates at least one node of the tenant resources with at least one node of the first non-tenant cloud and at least one node of the second non-tenant cloud; and

connecting the E2E-VNS virtual networks of the nodes of the EXI domain for network communication, the EXI domain to isolate resources of the nodes of the EXI domain from other resources of the MCEE logical structure in an EXI virtual network.

9. The computer-readable storage medium of claim 8,

10. The computer-readable storage medium of claim 9,

11. The computer-readable storage medium of claim 9,

the network isolation technology of the first non-tenant cloud is a multi-protocol label switching layer 3 virtual private network (MP L S L3 VPN) network isolation technology;

mapping the MCEE logical structure to the E2E-VNS comprises, for each first non-tenant cloud site virtual network of the E2E-VNS:

12. The computer-readable storage medium of claim 11, wherein mapping the MCEE logical structure to the E2E-VNS further comprises: for each first non-tenant cloud site virtual network of the E2E-VNS, assigning a single domain address to each first non-tenant cloud site virtual network.

13. The computer-readable storage medium of claim 8,

14. The computer-readable storage medium of claim 8, further comprising:

15. A system for multi-cloud elastic expansion and isolation, comprising:

one or more computer-readable storage media; and

a processor communicatively coupled to a storage device, wherein the processor executes application code instructions stored in the one or more computer-readable storage media to cause the system to perform the steps of:

16. The system of claim 15, wherein,

mapping the MCEE logical structure to an E2E-VNS comprises: performing mapping such that at each level of the E2E-VNS, a set of resources at each node of the MCEE logical structure are in separate virtual networks of the E2E-VNS; and

17. The system of claim 16, wherein,

18. The system of claim 16, wherein,

19. The system of claim 18, wherein mapping the MCEE logical structure to the E2E-VNS further comprises: for each first non-tenant cloud site virtual network of the E2E-VNS, assigning a single domain address to each first non-tenant cloud site virtual network.

20. The system of claim 15, wherein,