CN116615898A - Maintaining quality of service handling of packets using security parameter index values - Google Patents

Abstract

Techniques for load balancing encrypted traffic based on a Security Parameter Index (SPI) value of a packet header and a set of five tuple values of the packet header are described herein. Further, techniques for including quality of service (QoS) type information in an SPI value field of a packet header are described herein. The QoS type information may indicate a particular traffic class from which the packet is to be processed. In addition, techniques for pre-configuring a back-end host such that encrypted traffic may migrate from another back-end host to the back-end host without causing a temporary service disruption are also described herein.

Description

Maintaining quality of service handling of packets using security parameter index values

RELATED APPLICATIONS

The present utility model claims priority from U.S. patent application Ser. No.17/171,604, filed 2/9/2021, which claims priority from U.S. provisional patent application Ser. No.63/124,317, filed 12/11/2020, the entire contents of both of which are incorporated herein by reference.

Technical Field

The present disclosure relates generally to improved techniques for load balancing encrypted traffic using Security Parameter Index (SPI) values of packet headers.

Background

Constructing a cloud delivered software as a service (SaaS) product involves creating a distributed system that is delivered to users in the cloud. Typically, traffic is sent into these services according to one or more routing policies, such as equal cost multi-path (ECMP) routing. ECMP and other routing policies allow streams to be fixed based on "five-tuple" in order to send packets to a particular back-end instance. A five-tuple of a packet generally refers to a set of five different values that contains a transmission control protocol/internet protocol (TCP/IP) connection. The set of five-tuple values includes the source IP address, source port number, destination IP address, destination port number, and the particular protocol being used.

However, since ECMP and other routing policies use quintuple, they do not consider individual flows of encrypted connections, such as internet protocol security (IPsec) connections, which include Internet Key Exchange (IKE) traffic and Encapsulated Security Payload (ESP) traffic. Furthermore, encrypted connections (e.g., IPsec) are difficult to provide traffic classification for due to their encrypted nature. Once the packet is encrypted and encapsulated, it becomes almost impossible to perform any form of quality of service (QoS).

Furthermore, in a networking environment where a load balancer is placed before a pool of worker nodes responsible for handling encrypted traffic, when a worker node goes offline, the encrypted session assigned to the worker node must migrate to one or more other hosts. This typically results in a temporary service outage while the new host worker node(s) and clients negotiate a new encrypted connection.

Drawings

The following detailed description refers to the accompanying drawings. In the drawings, the leftmost digit(s) of a reference number identifies the drawing in which the reference number first appears. The use of the same reference symbols in different drawings indicates similar or identical items. The systems depicted in the drawings are not to scale and components in the drawings may not be depicted as being to scale relative to each other.

Fig. 1 illustrates a schematic diagram of an example system architecture of a networking environment including a tunnel communication session that includes separate control plane and data plane traffic flows.

FIG. 2 illustrates a schematic diagram of an example traffic flow in which a load balancer node sends traffic to a downstream node according to one or more routing policies.

Fig. 3 illustrates a data flow diagram of an example traffic flow between various nodes and/or devices for a communication session that establishes load balancing of traffic using SPI values of packet headers.

Fig. 4A and 4B illustrate data flow diagrams of example traffic flows between various nodes and/or devices of a communication session for indicating QoS type information in an SPI value of a packet header.

Fig. 5A-5C collectively illustrate schematic diagrams of example data flows associated with performing encrypted tunnel migration.

Fig. 6 illustrates a logic flow diagram of an example method for maintaining QoS treatment of a packet by using SPI values.

FIG. 7 illustrates a logic flow diagram of an example method for load balancing traffic based on SPI values of packet headers.

Fig. 8 illustrates a logic flow diagram of an example method for performing encrypted tunnel migration.

Fig. 9 illustrates a logic flow diagram of another example method for performing encrypted tunnel migration.

Fig. 10 illustrates a schematic diagram of an example computer hardware architecture for implementing network nodes and/or devices (e.g., load balancers, control nodes, data nodes, etc.) that may be used to implement various aspects of the various techniques presented herein.

Detailed Description

SUMMARY

Various aspects of the invention are set out in the independent claims and preferred features are set out in the dependent claims. Features of one aspect may be applied to each aspect alone or in combination with other aspects.

The present disclosure describes systems and methods that improve techniques related to load balancing encrypted traffic by using Security Parameter Index (SPI) values of packet headers. By way of example and not limitation, methods in accordance with the various techniques described in this disclosure may include receiving, from a client device and at a network device of a network, a request to establish an encrypted tunnel through the network such that data plane traffic may flow between the client device and a service via the encrypted tunnel. The method may also include generating an SPI value to be used by the client device for data plane traffic and sending an indication of the SPI value to the client device. Additionally, the method may include receiving, at the load balancer, a data packet including an SPI value, and determining to send the data packet to a server in a group of servers supporting the service based at least in part on the SPI value. Accordingly, the load balancer may send data packets to the server.

In some cases, the method may additionally or alternatively include determining that the data plane traffic belongs to a particular traffic class of a set of traffic classes. A particular traffic class may be associated with a particular quality of service (QoS) performance metric. Thus, the method may include generating an SPI value to be used by the client device for data plane traffic. The SPI value may correspond to a particular traffic class. In this way, the load balancer may receive data packets of data plane traffic including SPI values and based at least in part on the data packets including SPI values, the load balancer may send the data packets over the network such that the data packets are processed according to a particular QoS performance metric.

In an additional or alternative example, the method may include receiving, at a load balancer and from a client device, a first data plane traffic having a first SPI value and a set of quintuple values. Based at least in part on the first SPI value and the set of five tuple values, the load balancer may send first data plane traffic to the first node. The method may also include receiving, at the load balancer, an indication that at least a portion of the first data plane traffic is to be sent to the second node. Based at least in part on the indication, the load balancer can prompt the second node to provide one or more interfaces such that at least a portion of the first data plane traffic can be sent to the second node. In this way, the load balancer may receive second data plane traffic from the client device having a second SPI value and the set of five tuple values. Based at least in part on the second SPI value and the set of five tuple values, the load balancer can determine that the second data plane traffic comprises at least the portion of the first data plane traffic, and in response, send the second data plane traffic to the second node.

Additionally, the techniques described herein may be performed as a method and/or by a system having a non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more processors, perform the techniques described herein.

Example embodiment

As described above, traffic is typically sent to various services according to one or more routing policies, such as equal cost multi-path (ECMP) routing. However, since these routing policies use five tuples, they do not consider the individual flows of encrypted connections, such as internet protocol security (IPsec) connections, which include Internet Key Exchange (IKE) traffic and Encapsulated Security Payload (ESP) traffic. This means that the entropy of these flows may be much less than would be achievable if per-tunnel entropy were provided by Security Associations (SAs) for IPsec IKE and ESP flows. For example, IPsec IKE and ESP traffic contain additional identifiers called Security Parameter Indexes (SPI). The SPI value is used to uniquely identify an established IPsec SA.

Accordingly, one aspect of the present disclosure provides techniques for utilizing SPI values to allow load balancing and fixing per IPsec IKE and ESP flows to a particular backend. By performing these techniques, various advantages can be achieved, including the ability to terminate the same encrypted tunnel/SA across multiple systems, which allows for expansion of capacity. Further, higher performance may be achieved by finer granularity control over where control plane and data plane traffic sessions may fall on the backend nodes/servers. In addition, different traffic (e.g., performance levels, allowing customers to own their own backend) may be handled accordingly.

Furthermore, encrypted connections (e.g., IPsec) are difficult to provide traffic classification for due to the encrypted nature of the connection. Once the packet is encrypted and encapsulated, it becomes almost impossible to perform any form of quality of service (QoS). For example, when looking exclusively at Virtual Private Network (VPN) products that build IPsec cloud delivery, you are handling ESPs in IP packets and/or ESPs in UDP packets. Accordingly, another aspect of the present disclosure includes techniques for encoding and mapping QoS type information into SPI values so that QoS may be performed on packets even after the packets are encrypted and encapsulated. Performing these techniques may allow for classification of encrypted traffic, allowing differentiated service-style (DiffServ-style) resource allocation on data nodes, such that individual SAs may be allocated on data nodes that are better suited for that traffic class, and traffic shaping rules within the data nodes may also be adjusted to adjust network throughput in the flows allocated to it.

Furthermore, in a networking environment where a load balancer is placed before a pool of worker nodes responsible for handling encrypted traffic, when a worker node goes offline, the encrypted session assigned to the worker node must migrate to one or more other hosts. This typically results in a temporary service outage while the new host worker node(s) and clients negotiate a new encrypted connection. Accordingly, yet another aspect of the disclosure includes techniques for adding support to a back-end worker node (e.g., a data node) to signal to a load balancer to indicate that a worker node is about to be removed from a back-end worker node queue. In this way, these techniques may reduce the impact of an unscheduled or unplanned shutdown by migrating an encrypted tunnel out of a worker node when the worker node enters an unhealthy state or is set to be replaced by another node. Furthermore, these techniques may reduce the impact of rebalancing loads across server pools.

Thus, improvements in computer-related techniques may be realized in accordance with the various techniques described in this disclosure. As previously mentioned, the entropy for an encrypted stream may be much smaller than would be achievable if per-tunnel entropy was provided. For example, most routing policies (such as ECMP) use a set of five-tuple values to hash. However, utilizing the SPI value of the packet may allow six-tuple logic to be used, thereby better distributing the flows to the head-end nodes. Furthermore, even if the packet has been encapsulated, the SPI value may be further used to indicate QoS type information for the data packet so that individual SAs may be allocated on data nodes that are more appropriate for that traffic class. These are just a few examples of the many improvements that may be achieved in accordance with the techniques described in this disclosure. These modifications and other modifications will be readily apparent to and appreciated by those of ordinary skill in the art.

By way of example and not limitation, methods in accordance with various techniques described in this disclosure may include receiving a packet from a client device, the packet indicating a request to establish an encrypted tunnel over a network such that data plane traffic may flow between the client device and a service via the encrypted tunnel. In some examples, the request packet may be received at the network by a load balancer or router of the network and the load balancer or router may send the request to a control node of the network. In addition, the load balancer or router may send the request packet to the control node based at least in part on an equal cost multi-path (ECMP) routing policy and/or a five-tuple associated with the request packet. In some examples, the request to establish an encrypted tunnel may include a request to establish an IPsec connection and/or a request to establish an IPsec SA or sub-SA.

In some examples, the network may be configured such that the network includes separate control nodes and data nodes. In other words, the network may be configured to split the processing of control plane traffic (e.g., IKE traffic) and data plane traffic (e.g., ESP traffic) into different nodes (e.g., a control node or "IKE" node for processing the control plane and a data node for processing the data plane). This may allow the network to extend each node type individually and/or independently. The control node and the data node may include a headend server associated with the service. In some examples, the control node may operate on a first set of computing resources associated with the network and the data node may operate on a second, different set of computing resources associated with the network.

In some examples, the method may include determining that the data plane traffic belongs to a particular traffic class of a set of traffic classes. A particular traffic class may be associated with a particular quality of service (QoS) performance metric. In some examples, in order for the control node to create an SPI value that matches the correct traffic class, a classifier may be invoked before the SA is established. This can be accomplished in a number of different ways. For example, the load balancer or router may invoke a classifier and inject class information as a header before the control plane traffic is forwarded to the control node. This may be done by using a field in the IP header that is not currently used (e.g., DSCP field) or by creating a new field. Additionally, or alternatively, the control node may invoke a classifier when it initiates an SA. In any of these ways, class information may be provided to the control node so that the control node may create an SPI value corresponding to the traffic class.

In some examples, the method may include generating an SPI value to be used by the client device for data plane traffic. The SPI value may include a combination of bits identifying a particular SA. In some cases, multiple SPI values may be generated, and each individual SPI value of the multiple SPI values may identify a corresponding SA. In addition, the SPI value may be generated by the control node.

As described above, in various examples, the SPI value may include QoS type information (e.g., differentiated services (DiffServ) type information, type of service (ToS), differentiated Services Code Point (DSCP) type information, and/or experimental bit (EXP) type information) that indicates a particular traffic class according to which the packet is processed. Thus, in some examples, generating the SPI value may include generating a first bit combination representing a particular traffic class from which the packet is to be processed, generating a second bit combination representing a particular SA, and masking or combining the first bit combination and the second bit combination such that the first bit combination includes a first portion of the SPI value and the second bit combination includes a second portion of the SPI value. For example, the SPI field of a packet is defined as any value of 32 bits, and the range of 0-255 is defined as reserved. This leaves values from 256 (0 x 00000100) to 4294967295 (Oxffffffff) as SPI values. Thus, in some examples, a first portion (e.g., a "front" portion) of the 32-bit field may be used for QoS mapping, and a first hexadecimal number may be "reserved" for mapping by shifting the SPI value by 4 bits. For example, using hexadecimal values 0x0 < 3 > ec7b2a to 0x < 3 > ec7b2a, hexadecimal numbers 0 to f may represent QoS mappings, and 0x3ec7b2a may represent the actual SPI offset of 4 bits consumed. That is, a first portion of the SPI values (e.g., hexadecimal numbers [0] through [ f ]) may represent a particular traffic class QoS map, and a second portion of the SPI values (e.g., hexadecimal number 3ec7b2 a) may identify a particular SA. This results in the use of 15 mapping values. Additionally or alternatively, an already established 802.1q class of service (CoS) or multiprotocol label switching (MPLS) EXP to DSCP bit map may be followed, as they have similar bit sizes.

After generating the SPI, the method may include sending an indication of the SPI value to the client device. In some examples, the control node may perform a Direct Server Return (DSR) to send the indication to the client device. In some examples, a data packet including an SPI value may be received by a load balancer. The data packets may include data packets of data plane traffic. That is, the protocol associated with the data packets may correspond to a data plane traffic protocol, such as ESP. In some examples, the data packet may include a set of five-tuple values. For example, a set of five-tuple values for a data packet may include a source IP address value, a source port value, a destination IP address value, a destination port value, and a protocol associated with the data packet. As described herein, a set of six tuple values may be used to refer to an SPI value and a set of five tuple values. That is, a set of six-tuple values may include a source IP address value, a source port value, a destination IP address value, a destination port value, a protocol associated with the data packet, and an SPI value. However, different values may be used.

In some cases, the load balancer may determine to send a data packet to a server (e.g., a data node) in a group of servers or nodes supporting the service. For example, based at least in part on the SPI value and/or the set of five tuple values, the load balancer may determine to send the data packet to the server. In some examples, the load balancer may receive data representing an association between the SPI value and the set of five-tuple values associated with the client device, and determining to send the data packet to the server may be further based at least in part on the data. That is, the load balancer may be updated with a map indicating an association between the SPI value and the quintuple value. In some examples, determining to send the data packet to the server may be based at least in part on calculating a hash value that represents the SPI value and/or the set of quintuple values. The load balancer may use a hash function to compute the hash. Additionally or alternatively, the load balancer may determine to send the data packet to the server based at least in part on one or more routing policies (e.g., ECMP).

In some examples, the method may include sending the data packet to a server. In addition, the data packets may be transmitted over the network such that the data packets are processed according to particular QoS performance metrics and/or traffic classes. For example, if the SPI value includes an indication of a particular traffic class and/or QoS performance metric according to which the data packet is to be processed, the load balancer may send the packet over the network according to the particular traffic class and/or QoS performance metric.

In additional or alternative examples, the method may include generating a second SPI value for the client device to use for data plane traffic. The second SPI value may identify the second SA. The second SPI value may be generated by the control node. In some examples, generating the second SPI value may be based at least in part on classifying the request packet to determine a traffic class associated with the request packet, as described above. After generating the second SPI, the method may include sending an indication of the second SPI value to the client device. In some examples, the control node may perform a Direct Server Return (DSR) to send an indication of the second SPI value to the client device.

In some examples, the method may include receiving, at the load balancer, a second data packet including a second SPI value. In addition, the second data packet may include the set of five-tuple values, a portion of the set of five-tuple values, or a new set of five-tuple values. In some cases, the set of five-tuple values may indicate that the second data packet was sent by the client device or a different client device. The load balancer may send a second data packet to a second server (e.g., a second data node) in the server group based at least in part on the second data packet including the second SPI value and/or the set of five tuple values. In some examples, the first portion of the second SPI value may correspond to a second traffic class associated with a second QoS performance metric. In this way, the load balancer may send the second data packet over the network based at least in part on the second data packet including the second SPI value such that the second data packet is processed according to the second QoS performance metric.

As described above, one aspect of the techniques described herein may also include adding support for the back-end worker node (e.g., data node) to send a signal to the load balancer to indicate that the worker node is about to be removed from the back-end worker node queue. Thus, in additional or alternative examples, the method may include receiving, at a load balancer, a first data plane traffic from a client device having a first SPI value and a set of quintuple values. The first SPI value may identify a first Security Association (SA) between the client device and the first node.

In some examples, the method may include sending first data plane traffic to a first node in a set of nodes. In at least one example, the first node may comprise a first data node in a set of data nodes. The first node may be associated with a first encrypted tunnel (e.g., IPsec SA). In some examples, the first data plane traffic to the first node may be based at least in part on the first SPI value and the set of five tuple values. For example, the load balancer may calculate a hash value (e.g., a six tuple) representing the first SPI value and the set of five tuple values. Based at least in part on the hash value, the load balancer may send first data plane traffic to the first node according to an ECMP routing policy.

In various examples, the method may include receiving, at a load balancer, additional data plane traffic to be received from the client device to be sent to a second node in the set of nodes. In at least one example, the second node may comprise a second data node of the set of data nodes. The second node may be associated with a second encrypted tunnel (e.g., a second IPsec SA). In some cases, the indication may include an indication that at least a portion of the first data plane traffic is to be sent to the second node. That is, the indication may inform the load balancer that it needs to adjust where it sends data plane traffic. For example, the load capacity associated with the first node may meet or exceed a threshold load capacity. Additionally or alternatively, the indication may inform the load balancer that the first node is about to be removed (e.g., offline, serviced, etc.) from the set of nodes.

In some examples, a controller associated with the network may send an indication to the load balancer or cause the indication to be sent. For example, the controller may receive telemetry data from the set of nodes. Based at least in part on the telemetry data, the controller may determine a location to which the load balancer will adjust data plane traffic and/or control plane traffic it sends. For example, the telemetry data may indicate a load capacity associated with a respective node in the set of nodes. Additionally or alternatively, telemetry data may indicate a state associated with a respective node in the set of nodes (e.g., whether the node is in an unhealthy, suspended or crashed state, whether the host is to rotate, etc.). Thus, the controller may send an indication to the load balancer and/or, in some examples, a notification to the first node to prompt the first node to send an indication to the load balancer. In some examples, the controller may include a distributed system including a key value store.

Based at least in part on the indication, in some examples, the load balancer and/or the controller may prompt the second node to provide one or more resources such that a portion of the first data plane traffic may be sent to the second node. The one or more resources may include interfaces, channels, computing resources, and the like. In this way, by prompting the second node to provide one or more resources, the second node may "warm up" before data plane traffic is sent to the second node. In examples where a portion of the data plane traffic is redirected from a first node to a second node, pre-heating or pre-configuring the second node may help reduce downtime and/or temporary service disruption while the second node and the client device negotiate a new encrypted connection. In at least one example, prompting the second node to provide the one or more resources may include generating and/or transmitting an empty Encapsulation Security Payload (ESP) packet to the second node. The null ESP packet may include an Internet Protocol (IP) address and a port associated with the client device, as well as other five-tuple values in some cases.

In some examples, the load balancer and/or the controller may send a request to the third node for the third node to generate the second SPI value. The load balancer and/or controller can send a request based at least in part on the indication. In at least one example, the third node includes a first control node (e.g., IKE node) in a set of control nodes. Further, the request may include a "key update (rekey)" request. That is, the request may be that the third node create a second SA between the client device and the second node in place of the first SA between the client device and the first node. Thus, in some examples, the method may include receiving, at the load balancer, an indication of the second SPI value. The indication of the second SPI value may include an indication of an association between the second SPI value and the set of five tuple values.

In some examples, the method may include receiving, at the load balancer and from the client device, second data plane traffic having a second SPI value and the set of five tuple values. The second data plane traffic may include some (e.g., a portion) or all of the first data plane traffic that the load balancer previously sent to the first node. Based at least in part on the second data plane traffic having the second SPI value and the set of five tuple values, the method may include, in some cases, determining that the second data plane traffic includes some (e.g., a portion) or all of the first data plane traffic. For example, the load balancer may not know the second SPI value, and the load balancer may track all SPI values associated with a particular set of five-tuple values. Once the load balancer issues a key update request, it may begin monitoring the new/unknown SPI associated with the set of five-tuple values and sending all data plane traffic with the new/unknown SPI to the second node.

In some examples, based at least in part on the second SPI value and the set of five tuple values, the method may include sending second data plane traffic to the second node. Additionally, in some cases, the method may include removing a first association between the first SPI value and the set of five-tuple values and/or storing a second association between the second SPI value and the set of five-tuple values.

Certain implementations and embodiments of the present disclosure now will be described more fully hereinafter with reference to the accompanying drawings, in which various aspects are shown. The various aspects may, however, be embodied in many different forms and should not be construed as limited to the implementations set forth herein. For example, while many of the examples herein are described with respect to ECMP routing, it should be understood that other routing policies may be used. Moreover, while many examples are shown as distributed systems, it should be understood that the various processes and methods described may be performed by more or fewer devices. As described herein, the present disclosure includes variations of the embodiments. Like numbers refer to like elements throughout.

Fig. 1 illustrates a schematic diagram of an example system architecture 100 of a networking environment 102 that includes a tunnel communication session that includes separate control plane and data plane traffic flows. In general, the networking environment 102 may include equipment housed or located in one or more data centers 104, which data centers 104 may be located in different physical locations. For example, the networking environment 102 may be supported by a network of devices in a public cloud computing platform, a private/enterprise computing platform, and/or any combination thereof. One or more data centers 104 may be physical facilities or buildings located over a geographic area that are designated to store networked devices as part of the networked environment 102. The data center 104 may include various network devices, as well as redundant or backup components and infrastructure for power, data communication connections, environmental control, and various security devices. In some examples, data center 104 may include one or more virtual data centers that are a pool or collection of cloud infrastructure resources specifically designed for enterprise needs and/or cloud-based service provider needs. In general, the data center 104 (physical and/or virtual) may provide basic resources such as processors (CPUs), memory (RAMs), storage devices (disks), and networks (bandwidths). However, in some examples, devices in the networking environment 102 may not be located in a well-defined data center 104, but may be located in other locations or buildings.

The networking environment 102 may be accessed by the client device 106 through one or more networks 108. The networking environment 102 and the network 108 may each comprise one or more networks implemented by any feasible communication technology (e.g., wired and/or wireless modalities and/or technologies), respectively. Networking environment 102 and network 108 may each include a Personal Area Network (PAN), a Local Area Network (LAN), a Campus Area Network (CAN), a Metropolitan Area Network (MAN), an extranet, an intranet, the internet, a short range wireless communication network (e.g., zigBee, bluetooth, etc.), a Virtual Private Network (VPN), a Wide Area Network (WAN) -centralized and/or distributed-and/or any combination, permutation, and/or aggregation thereof. The networking environment 102 may include devices, virtual resources, or other nodes that relay packets from one network segment to another network segment through nodes in a computer network.

In some examples, the networking environment 102 may provide one or more services 110, host one or more services 110, provide a connection to one or more services 110, or otherwise support one or more services 110 for connection and use by the client device 106. Client device 106 may include any type of device configured to communicate over network 108 using various communication protocols (e.g., VPN, SSL, TLS, DTLS and/or any other protocol). For example, client device 106 may include a personal user device (e.g., a desktop computer, a laptop computer, a telephone, a tablet, a wearable device, an entertainment device such as a television, etc.), a network device (e.g., a server, a router, a switch, an access point, etc.), and/or any other type of computing device.

In some examples, networking environment 102 may include edge routers 112 (1) and 112 (2) (hereinafter collectively referred to as "edge routers 112"), load balancers 114 (1) -114 (N) (hereinafter collectively referred to as "load balancers 114") (where N represents any number greater than or equal to one), data nodes 116 (1) -116 (N), control nodes 118 (1) -118 (N), firewall nodes 120 (1) -120 (N), key-value store 122, and controller 124. In various examples, the various systems/devices/nodes of the networking environment 102 may communicate with each other via a management plane and/or message bus associated with the networking environment 102. For example, a common message bus associated with the networking environment 102 may enable the data node to signal to the load balancer that it is about to be removed from the data node queue, that the load balancer needs to adjust where it will flow to occur, and so on. Further, a message bus associated with networking environment 102 may enable any devices/systems/nodes of networking environment 102 to communicate directly with each other.

In some examples, edge router 112 and load balancer 114 may use ECMP, which is a strategy in which next hop packet forwarding to a single destination may occur on multiple "best paths" that are collocated first in the routing metric calculation. In addition, edge router 112 and load balancer 114 may use any routing policy in conjunction with or in lieu of ECMP routing, such as Open Shortest Path First (OSPF), intermediate system-to-intermediate system (ISIS), enhanced Interior Gateway Routing Protocol (EIGRP), domain Name System (DNS) load balancing, and/or Border Gateway Protocol (BGP). Although shown as separate entities in fig. 1, it is to be understood that in some cases, edge router 112 and load balancer 114 may reside on the same hardware device and/or node.

In some cases, edge router 112 may balance traffic 126 based on the hash of the network five-tuple in order to route the packet to load balancer 114. Traffic 126 may include control plane traffic 128 and data plane traffic 130. In addition, load balancer 114 may balance traffic 126 based on the hash of the network six tuples in order to route control plane traffic 128 to control node 118 and data plane traffic 130 to data node 116. The network six-tuple of the packet may include the SPI value, source IP address, source port, destination IP address, destination port, and protocol of the packet.

As shown, the networking environment 102 may include data nodes 116 (1) -116 (N) (hereinafter collectively referred to as "data nodes 116") (where N represents any number greater than or equal to one). In some examples, the data node 116 may process data plane traffic 130 on behalf of the networking environment 102. Data plane traffic 130 may include ESP traffic associated with an IPsec connection. In some examples, data node 116 (1) of data node 116 may be associated with one or more IPsec security associations. In addition, data node 116 may forward data plane traffic 130 to one or more downstream nodes and/or devices, such as firewall nodes 120 (1) -120 (N) (hereinafter collectively referred to as "firewall nodes 120") (where N represents any number greater than or equal to one). In some examples, a first one of the data nodes 116 may be associated with a first traffic class, a second one of the data nodes 116 may be associated with a second traffic class, and so on. Additionally or alternatively, a first interface of a first one of the data nodes 116 may be associated with a first traffic class, a second interface of the first one of the data nodes 116 may be associated with a second traffic class, and so on.

The networking environment 102 may also include one or more control nodes 118 (1) -118 (N) (hereinafter collectively referred to as "control nodes 118") (where N represents any number greater than or equal to one). In some examples, control node 118 may handle control plane traffic 128 on behalf of networking environment 102. Control plane traffic 128 may include IKE traffic associated with an IPsec connection.

As shown, both data node 116 and control node 118 may perform a Direct Server Return (DSR) to send return traffic 132 back to client device 106. That is, the data node 116 and the control node 118 may send return traffic 132 to the client device 106 via the edge router 112 (1), bypassing the load balancer 114. Additionally or alternatively, the data node 116 and the control node 118 may send the return traffic 132 directly to the client device bypassing the edge router 112 (1).

The networking environment 102 may also include a key value store 122 and a controller 124. The key-value store 122 may include one or more databases that are accessible by the various nodes and devices of the networked environment 102. In some examples, load balancer 114, data node 116, control node 118, and other nodes and/or devices of networking environment 102 may read data from key store 122 and write data to key store 122. Key value storage 122 may store an association between SPI values and SAs, a set of SPI values and quintuple values, and so forth. In some examples, controller 124 may receive telemetry data from data node 116 and/or control node 118 and determine a status associated with each of data node 116 and/or control node 118 based at least in part on the telemetry data. For example, the controller 124 may receive telemetry data indicating a load capacity associated with the data node 116 (1). The controller 124 may also determine whether the load capacity meets or exceeds a threshold load capacity, and if so, the controller 124 may prompt the data node 116 (1) to send a notification to the load balancer 114 (1) to request the load balancer 114 (1) to adjust the location to which it sends the data plane traffic 130.

Although depicted in fig. 1 as separate hardware components, it should be appreciated that edge router 112, load balancer 114, data node 116, control node 118, firewall node 120, key store 122, and/or controller 124 may be software components that reside at least partially in memory. In this manner, the one or more processors may execute instructions that cause the one or more processors to perform all of the operations described herein with respect to edge router 112, load balancer 114, data node 116, control node 118, firewall node 120, key store 122, and/or controller 124. In some cases, edge router 112, load balancer 114, data node 116, control node 118, firewall node 120, key store 122, and/or controller 124 may be separate hardware components and/or software components residing in a stand-alone device or stand-alone device system. Additionally or alternatively, edge router 112, load balancer 114, data node 116, control node 118, firewall node 120, key store 122, and/or controller 124 may comprise any type of network device, such as a server, switch, router, hub, bridge, gateway, modem, repeater, access point, etc.

Fig. 2 shows a schematic diagram of an example traffic flow 200 in which the load balancer 114 (1) sends traffic to downstream nodes according to one or more routing policies. For example, load balancer 114 (1) may receive incoming tunnel traffic 202 from a client device (e.g., one or more client devices 106). In some cases, the incoming tunnel traffic 202 may include control plane traffic 128 and/or data plane traffic 130. In addition, incoming tunnel traffic 202 may include an SPI value and a set of five tuple values.

In some examples, when load balancer 114 (1) receives incoming tunnel traffic 202, load balancer 114 (1) may calculate a hash value representing the SPI value of incoming tunnel traffic 202 and the set of five-tuple values. The load balancer 114 (1) may then determine a particular node of the data nodes 116 or the control nodes 118 to which the incoming tunnel traffic 202 is to be sent based at least in part on the hash value and using the ECMP routing policy. For example, if the incoming tunnel traffic 202 includes control plane traffic 128 (e.g., IKE traffic), the load balancer 114 (1) may send the control plane traffic 128 to one of the control nodes 118 based at least in part on the hash value. Likewise, if the incoming tunnel traffic 202 includes data plane traffic 130 (e.g., ESP traffic), the load balancer 114 (1) may send the data plane traffic 130 to one of the data nodes 116 based at least in part on the hash value.

Fig. 3 illustrates a data flow diagram of an example traffic flow 300 between various nodes and/or devices for establishing a load balanced communication session of traffic using SPI values of packet headers. The example traffic flow 300 includes a client 302, a router/load balancer 304, a first head-end 306, and a second head-end 308. In an example, the first head end 306 and the second head end 308 may include data nodes, control nodes, servers, and the like. For example, the first head-end 306 may include a control node and the second head-end 308 may include a data node.

To begin the example traffic flow 300, a client 302 sends a connection request packet 310 to a router/load balancer 304. Connection request packet 310 may indicate a request to establish an encrypted tunnel so that traffic may flow from client 302 to second head-end 308. Connection request packet 310 may include a set of five-tuple values. The router/load balancer 304, upon receiving the connection request packet 310, may send the connection request packet 310 to the first head-end 306. The router/load balancer 304 may determine to send the connection request packet 310 to the first head-end 306 based at least in part on calculating a hash value representative of the set of five-tuple values included in the connection request packet 310. Additionally or alternatively, the router/load balancer 304 may determine to send the connection request packet 310 to the first head-end 306 based at least in part on the ECMP routing policy.

After receiving the connection request packet 310, the first head-end 306 may establish an IKE session 314 with the client 302. In this way, IKE traffic may flow between the client 302 and the first head-end 306. In some cases, establishing IKE session 314 may include authenticating user 302 associated with the client, such as by determining the identity of the user. Once the IKE session is established, the first head-end may send a reply packet 316 to the client 302. Reply packet 316 may indicate that an IKE session has been established.

Client 302 may then send ESP traffic 318 to router/load balancer 304, and router/load balancer 304 may forward ESP traffic 318 to second head-end 308. After receiving ESP traffic 318, second head-end 308 may generate SPI value 320 for use by clients 302 in transmitting data plane traffic over the ESP channel. Second head-end 308 may further associate an SPI value with the set of five-tuple values. In this way, second head-end 308 may update router/load balancer 304 with five-tuple and SPI mapping 322. In some cases, five-tuple and SPI map 322 may include hash values. Additionally or alternatively, the five-tuple and SPI mapping 322 may indicate that future data plane packets including certain five-tuple value sets and certain SPI values are to be sent to the second head end 308. The second head-end 308 may then send a reply packet 324 back to the client 302. Reply packet 324 may indicate that client 302 may begin transmitting data plane traffic 326 using an ESP channel or an encrypted tunnel.

After receiving reply packet 324, client 302 may begin transmitting data plane traffic 326 over the ESP channel. When router/load balancer 304 receives data plane traffic, router/load balancer 304 can calculate a hash value representing a network five tuple and SPI value 328. For example, the packets of data plane traffic 326 may include SPI values and network five tuples. Based at least in part on the calculated hash, the router/load balancer 304 may send data plane traffic 326 to the second head-end 308. For example, router/load balancer 304 may send data plane traffic 326 to second head-end 308 based at least in part on five-tuple and SPI map 322.

Fig. 4A and 4B illustrate data flow diagrams of example traffic flows 400 (1) and 400 (2) between various nodes and/or devices of a communication session for indicating QoS type information in an SPI value and/or an SPI value field of a packet header. Example traffic flows 400 (1) and 400 (2) may include clients 302, router/load balancer 304, IKE nodes 402, and classifier 404.

With respect to fig. 4A, a client 302 may send a connection request packet 406 to a router/load balancer 304. In some examples, the connection request packet 406 may include a IKE SA INIT request packet. The connection request packet 406 may indicate a request to establish an encrypted tunnel (e.g., IPsec connection) for use by the client 302 to send data to and/or receive data from a service. The router/load balancer 304, upon receiving the connection request packet 406, may invoke the classifier 404 to determine the traffic class associated with the connection request packet 406. For example, connection request packet 304 may indicate a type of traffic (e.g., voice, video, audio, network, etc.) that client 302 wishes to send and/or receive, and classifier 404 may be configured to determine what type of traffic that is. Additionally or alternatively, the connection request packet 406 may include a request to establish a plurality of connections, each connection being associated with a different traffic class and/or priority. In some examples, invoking the classifier 404 may include sending, by the router/load balancer 304, a connection request packet 406 to the classifier 404.

In some examples, classifier 404 may operate on data packet 408 to determine a traffic class associated with connection request data packet 406. For example, classifier 404 may determine that connection request packet 406 includes a request to establish one or more of a voice traffic channel, a video traffic channel, an audio traffic channel, a network traffic channel, and the like. Classifier 404 may send classification packet 410 indicating the traffic class associated with connection request packet 406 after determining the traffic class. Classifier 404 may send classification packet 410 to router/load balancer 304. In turn, the router/load balancer 304 may inject an indication of the traffic class classification into the packet header of the connection request packet 406. In this way, the connection request packet 406 may include an updated connection request packet 414 that includes an indication of traffic class classification information in its packet header.

The router/load balancer 304 may send the updated connection request packet 414 to a control node, such as IKE node 402. IKE node 402 may receive updated connection request packet 414. Based at least in part on the traffic class classification information included in the packet header of the updated connection request packet 414, the IKE node 402 may generate one or more SPI values 416. The one or more SPI values 416 may indicate, in whole or in part, the traffic class from which data plane traffic is to be processed. That is, a particular SPI value may include a unique combination of bits, and a first bit combination of the unique combination of bits (e.g., a first portion of the SPI value) may indicate a traffic class, and a second bit combination of the unique combination of bits (e.g., a second portion of the SPI value) may identify a security association between client 302 and one or more hosts associated with an encrypted tunnel connection. In other words, the SPI value field of the data plane packet header may include a first bit combination that indicates the traffic class and a second bit combination that identifies a security association between client 302 and one or more hosts associated with the encrypted tunnel connection.

After generating one or more SPI values 416, IKE node 402 may send a response packet 418. In some cases, the response packet 418 may include an IKE INIT response packet. Additionally or alternatively, the response packet 418 may include some or all of the one or more SPI values 416. In this way, client 302 may use a first one of the one or more SPI values 416 to send first data plane traffic according to a first traffic class associated with a first QoS metric and may use a second one of the one or more SPI values 416 to send second data plane traffic according to a second traffic class associated with a second QoS metric.

With respect to fig. 4B, the client 302 may send a connection request packet 406 to the router/load balancer 304. In some examples, the connection request packet 406 may include a IKE SA INIT request packet indicating a request for the IKE node 402 to establish an IPsec security association. The connection request packet 406 may indicate a request to establish an encrypted tunnel (e.g., IPsec connection) for use by the client 302 to send data to and/or receive data from a service. The router/load balancer 304, upon receiving the connection request packet 304, may send a connection request packet 406 to the IKE node 402. In some examples, router/load balancer 304 may calculate a hash value representing a network five tuple included in connection request packet 406 and send connection request packet 406 to IKE node 402 based at least in part on the hash value. For example, the router/load balancer 304 may use ECMP routing policies and determine to send a connection request packet to the IKE node 402 based at least in part on the hash value.

In order for IKE node 402 to generate one or more SPI values that match the correct traffic class, IKE node 402 may invoke classifier 404 before establishing a connection for client 302. For example, IKE node 402 may send connection request packet 406 or a portion of connection request packet 406 to a classifier so that classifier 404 may determine the traffic class associated with connection request packet 406. For example, the connection request packet 406 may indicate the type of traffic (e.g., voice, video, audio, network, etc.) that the client 302 wishes to send and/or receive, and the classifier 404 may be configured to determine what type of traffic that is. Additionally or alternatively, the connection request packet 406 may include a request to establish a plurality of connections, each connection being associated with a different traffic class and/or priority.

In some examples, classifier 404 may operate on connection request packet 408 to determine a traffic class associated with connection request packet 406. For example, classifier 404 may determine that connection request packet 406 includes a request to establish one or more of a voice traffic channel, a video traffic channel, an audio traffic channel, a network traffic channel, and the like. Classifier 404 may send classification packet 410 indicating the traffic class associated with connection request packet 406 after determining the traffic class. Classifier 404 may send classification packet 410 to IKE node 402.

In some examples, IKE node 402 may receive classification packet 410. Based at least in part on the traffic class associated with connection request packet 406, IKE node 402 may generate one or more SPI values 416. The one or more SPI values 416 may indicate, in whole or in part, the traffic class from which data plane traffic is to be processed. That is, a particular SPI value may include a unique combination of bits, and a first bit combination of the unique combination of bits (e.g., a first portion of the SPI value) may indicate a traffic class, and a second bit combination of the unique combination of bits (e.g., a second portion of the SPI value) may identify a security association between client 302 and one or more hosts associated with an encrypted tunnel connection. In other words, the SPI value field of the data plane packet header may include a first bit combination that indicates the traffic class and a second bit combination that identifies a security association between client 302 and one or more hosts associated with the encrypted tunnel connection.

After generating one or more SPI values 416, IKE node 402 may send response packet 418 to client 302. In some cases, the response packet 418 may include an IKE INIT response packet. Additionally or alternatively, the response packet 418 may include some or all of the one or more SPI values 416. In this way, client 302 may use a first one of the one or more SPI values 416 to send first data plane traffic according to a first traffic class associated with a first QoS metric and may use a second one of the one or more SPI values 416 to send second data plane traffic according to a second traffic class associated with a second QoS metric.

Fig. 5A-5C collectively illustrate a schematic diagram of an example data flow 500 associated with performing encrypted tunnel migration. At "1", the load balancer 114 (1) may receive traffic 502 from one or more client devices 106 and forward the traffic 502 to one or more back end nodes 504 (l) -504 (N) (hereinafter collectively referred to as "back end nodes 504"), where N represents any number greater than or equal to one. Traffic 502 may include a first traffic 502 (1) to be sent to node 504 (1), a second traffic 502 (2) to be sent to node 504 (2), and an nth traffic 502 (N) to be sent to node 504 (N) (where N represents any number greater than or equal to one). Additionally, traffic 502 may include data plane traffic and/or control plane traffic.

In some examples, load balancer 114 (1) may determine that first traffic 502 (1), second traffic 502 (2), and nth traffic 502 (N) are to be sent to nodes 504 (1), 504 (2), and 504 (N), respectively, based at least in part on the ECMP routing policy. The ECMP routing policy can use six-tuple logic to determine which of the back-end nodes 504 sent each packet of the traffic 502. Six-tuple logic may include an SPI value for a single packet and a set of five-tuple values (source address, destination address, source port, destination port, and protocol) for a single packet. For example, each packet of first traffic 502 (1), second traffic 502 (2), and nth traffic 502 (N) may each include a corresponding SPI value and a corresponding set of five-tuple values, and load balancer 114 (1) may calculate a corresponding hash value for each individual packet, the respective hash value representing the corresponding SPI value and the corresponding set of five-tuple values for each individual packet. In this way, each respective hash value of each individual packet of traffic 502 may indicate to which of back-end nodes 504 the individual packet is to be sent.

At "2," the controller 124 may receive telemetry data 506 associated with the backend node 504. For example, node 504 (1) may send first telemetry data to controller 124, node 504 (2) may send second telemetry data to controller 124, and node 504 (N) may send nth telemetry data to controller 124. In some examples, telemetry data 506 may indicate a load capacity associated with each of backend nodes 504. That is, telemetry data 506 may indicate that node 504 (1) is operating at 27% capacity, node 504 (2) is operating at 100% capacity, and node 504 (N) is operating at 17% capacity. In some examples, the load capacity associated with the backend node may include one or more of tunnel load capacity associated with the backend node, an amount of hardware resources available/used by the backend node, an amount of virtual computing resources available/used by the backend node, and the like.

At "3", the controller 124 may send an indication 508 for the load balancer 114 (1) to adjust the data flow. That is, the locations to which the load balancer 114 (1) sends the various portions of traffic 502 (e.g., which node of the back-end nodes 504) are adjusted. For example, based at least in part on telemetry data 506, controller 124 may determine that the load capacity of node 504 (2) exceeds a threshold load capacity. The threshold load capacity may include, for example, a percentage value (e.g., 80%, 85%, 90%, 100%, etc.). Further, the threshold load capacity may be dynamic and vary (e.g., from 80% to 90%) depending on the time of day, day of the week, current demand, etc. In some cases, the controller 124 may send the indication directly to the load balancer 114 (1). Additionally or alternatively, the controller 124 may send an indication to the node 504 (2), as shown in fig. 5B.

At "4", node 504 (2) may send or forward an indication 508 to load balancer 114 (1) based at least in part on receiving the indication from controller 124. The indication 508 may be configured to prompt the load balancer 114 (1) to perform one or more actions to adjust where it sends the data stream. Thus, at "5", based at least in part on receipt of indication 508, load balancer 114 (1) may send indication 510 to node 504 (N) to prompt node 504 (N) to prepare one or more interfaces so that a portion of second traffic 502 (2) may be sent/redirected to node 504 (N). In at least one example, node 504 (N) may comprise a data node for processing ESP traffic and indication 510 may comprise an null ESP packet including a source IP address and port associated with a client device of one or more client devices 106. In this way, when node 504 (N) receives an null ESP packet, node 504 (N) may begin to set up an interface in preparation for receiving an IPsec security association. In addition, load balancer 114 (1) may send a key update request to the control node responsible for the respective IKE session associated with the IPsec security association.

At "6", load balancer 114 (1) may begin sending additional traffic 512 to node 504 (N). The additional traffic 512 may include at least a portion of the second traffic 502 (2) previously sent to the node 504 (2). In this way, the load capacity of node 504 (2) may be reduced (e.g., to 75%) and the load capacity of node 504 (N) may be increased (e.g., to 42%). In some examples, load balancer 114 (1) may begin sending additional traffic 512 to node 504 (N) based at least in part on receiving indication 508 to adjust the data flow. Further, load balancer 114 (1) may determine that additional traffic 512 is to be sent to node 504 (N) based at least in part on the SPI values included in the individual packets of additional traffic 512. For example, in some examples, based at least in part on the load balancer 114 (1) sending the key update request, the load balancer 114 (1) may not be aware of the SPI values included in the individual packets because the IKE node may have issued a new SPI value for use by the client device. Thus, load balancer 114 (1) may identify a set of five-tuple values included in the separate packet and determine that additional traffic 512 is to be sent to node 504 (N) based at least in part on identifying the set of five-tuple values and based at least in part on issuing the update key request. In addition, load balancer 114 (1) may store an association between the new/unknown SPI value and the set of five-tuple values.

Fig. 6, 7, 8, and 9 illustrate logic flow diagrams of various example methods associated with the techniques presented herein for load balancing encrypted traffic based on SPI values. The logical operations described herein with reference to fig. 6, 7, 8, and 9 can be implemented (1) as a series of computer implemented acts or program modules running on a computing system and/or (2) as interconnected machine logic circuits or circuit modules within the computing system.

The implementation of the various components described herein is a matter of choice dependent on the performance and other requirements of the computing system. Accordingly, the logical operations described herein are referred to variously as operations, structural devices, acts or modules. The operations, structural devices, acts and modules may be implemented in software, in firmware, in special purpose digital logic, and any combination thereof. It should also be appreciated that more or fewer operations may be performed than shown in fig. 6, 7, 8, and 9 and described herein. These operations may also be performed in parallel, or in a different order than described herein. Some or all of these operations may also be performed by components other than those specifically identified. Although the techniques described in this disclosure are with reference to particular components, in other examples, the techniques may be implemented by fewer components, more components, different components, or any configuration of components.

Fig. 6 illustrates a logic flow diagram of an example method 600 for maintaining QoS treatment of a packet by using SPI values. The example method 600 begins at operation 602, which includes receiving, from a client device and at a network device of a network, a request to establish an encrypted tunnel through the network such that data plane traffic can flow between the client device and a service via the encrypted tunnel. For example, the load balancer 114 (1) and/or the control node 118 (1) may receive a request from a first client device of the one or more client devices 106. Further, in some examples, the request may include IKE SA INIT requesting packets to establish an IPsec SA between the first client device and the first data node 116 (1) such that data plane traffic 130 may flow between the client device and the service 110.

At operation 604, the example method 600 includes determining that the data plane traffic belongs to a particular traffic class of a set of traffic classes, the particular traffic class associated with a particular quality of service (QoS) performance metric. In some examples, control node 118 (1) may invoke a classifier to determine a particular traffic class. Additionally or alternatively, the load balancer 114 (1) may invoke a classifier to determine a particular traffic class.

At operation 606, the example method 600 includes generating a Security Parameter Index (SPI) value to be used by the client device for the data plane traffic, the SPI value corresponding to the particular traffic class. For example, control node 118 (1) may generate an SPI value to be used by a first client device of the one or more client devices 106. The SPI value and/or a portion of the SPI value field may correspond to a particular traffic class. That is, in some examples, the SPI value may include a unique bit combination, and a portion of the bits of the unique bit combination may correspond to a particular traffic class.

At operation 608, the example method 600 includes sending an indication of the SPI value to the client device. For example, control node 118 (1) may send an indication of the SPI value directly to a first client device of the one or more client devices 106 by performing a Direct Server Return (DSR) to bypass at least load balancer 114. In some examples, the indication may include an IKE INIT response packet indicating that an IPsec SA has been established for a first client device of the one or more client devices 106.

At operation 610, the example method 600 includes receiving, at a load balancing node associated with a network, a data packet of data plane traffic including an SPI value. For example, the load balancer 114 (1) may receive data packets of the data plane traffic 130 from the edge router 112 (1), or directly from the first client device, where load balancing techniques may be performed by the edge router 112 (1). In some cases, edge router 112 (1) may perform an ECMP routing policy based on computing a hash value representing a set of network quintuple values included in the data packet and determine to send the data packet to load balancer 114 (1).

At operation 612, the example method 600 includes transmitting the data packet over the network based at least in part on the data packet including the SPI value such that the data packet is processed according to the particular QoS performance metric. For example, load balancer 114 (1) may send a data packet to data node 116 (1) based on executing an ECMP routing policy that includes calculating a hash value that represents an SPI value and the set of network quintuple values included in the data packet. Additionally, the data node 116 (1) may be associated with a traffic class, and the load balancer 114 (1) may determine to send the data packet to the data node 116 (1) based at least in part on the hash value and/or determining that the SPI value is associated with the traffic class.

Fig. 7 illustrates a logic flow diagram of an example method 700 for load balancing traffic based on SPI values of packet headers. The example method 700 begins at operation 702, which includes receiving, from a client device and at a network device of a network, a request to establish an encrypted tunnel through the network such that data plane traffic can flow between the client device and a service via the encrypted tunnel. For example, the load balancer 114 (1) and/or the control node 118 (1) may receive a request from a first client device of the one or more client devices 106. Further, in some examples, the request may include IKE SA INIT requesting packets to establish an IPsec SA between the first client device and the first data node 116 (1) such that data plane traffic 130 may flow between the client device and the service 110.

At operation 704, the example method 700 includes generating a Security Parameter Index (SPI) value to be used by the client device for data plane traffic. For example, the control node 118 (1) may generate an SPI value to be used by a first client device of the one or more client devices 118 (1). In some examples, the SPI value may identify an IPsec SA between the first client device and one or more data nodes 116 and/or one or more interfaces of data nodes 116.

At operation 706, the example method 700 includes sending an indication of the SPI value to the client device. For example, control node 118 (1) may send an indication of the SPI value directly to a first client device of the one or more client devices 106 by performing a Direct Server Return (DSR) to bypass at least load balancer 114. In some examples, the indication may include an IKE INIT response packet indicating that an IPsec SA has been established for a first client device of the one or more client devices 106.

At operation 708, the example method 700 includes receiving, at a load balancer, a data packet including an SPI value. For example, the load balancer 114 (1) may receive data packets of the data plane traffic 130 from the edge router 112 (1), or directly from the first client device, where load balancing techniques may be performed by the edge router 112 (1). In some cases, edge router 112 (1) may perform an ECMP routing policy based on computing a hash value representing a set of network quintuple values included in the data packet and determine to send the data packet to load balancer 114 (1).

At operation 710, the example method 700 includes determining, by a load balancer and based at least in part on the SPI value, to send the data packet to one of a group of servers supporting the service. In some examples, determining to send the data packet to a server in the set of servers may further include determining an encryption tunnel between the load balancer and the server that is to be used to send the data packet to the server. For example, load balancer 114 (1) may send a data packet to data node 116 (1) based on executing an ECMP routing policy that includes calculating a hash value that represents an SPI value and the set of network quintuple values included in the data packet.

At operation 712, the example method 700 includes transmitting the data packet to a server. For example, load balancer 114 (1) may send data packets of data plane traffic 130 to data node 116 (1) such that data node 116 (1) may forward the data packets to firewall node 120 (1), and firewall node 120 (1) may then forward the data packets downstream to service 110.

Fig. 8 illustrates a logic flow diagram of an example method 800 for performing encrypted tunnel migration. The example method 800 begins at operation 802, which includes receiving, at a load balancer and from a client device, a first data plane traffic having a first Security Parameter Index (SPI) value and a set of quintuple values. For example, the load balancer 114 (1) may receive first data plane traffic from the edge router 112 (1), or directly from the first client device where load balancing techniques may be performed by the edge router 112 (1). In some cases, the edge router 112 (1) may perform an ECMP routing policy based on computing a first hash value representing the set of network quintuple values included in the first data plane traffic and determining to send the first data plane traffic to the load balancer 114 (1).

At operation 804, the example method 800 includes transmitting first data plane traffic to a first node based at least in part on a first SPI value, the first node being associated with a first encrypted tunnel. For example, load balancer 114 (1) may send first data plane traffic 130 to data node 116 (1). For example, the first SPI value may identify an IPsec SA between the client device and the data node 116 (1). In some examples, sending the first data plane traffic to the first node may be based at least in part on calculating a second hash value representing the first SPI value and a set of network quintuple values of the first data plane traffic.

At operation 806, the example method 800 includes receiving, at the load balancer, an indication that additional data plane traffic received from the client device is to be sent to a second node, the second node associated with a second encrypted tunnel. For example, the indication may indicate that the first node is operating at maximum load capacity or that the first node is about to lose connection, leave, be serviced, etc. The load balancer may receive an indication from a controller, such as controller 124, and/or a node, such as one of control node 118 or data node 116. In some examples, the indication may prompt the load balancer to send a key update request to one of the control nodes 118, e.g., to establish a new IPsec SA for the client device. Additionally or alternatively, the indication may prompt the load balancer to send a null ESP packet to the second data node (e.g., data node 116 (N)) that includes the IP address and port of the client device so that the second data node may begin setting up an interface to receive additional data plane traffic.

At operation 808, the example method 800 includes receiving, at the load balancer and from the client device, a second data plane traffic having a second SPI value and the set of five tuple values. For example, the load balancer 114 (1) may receive second data plane traffic from the edge router 112 (1), or directly from the first client device, where load balancing techniques may be performed by the edge router 112 (1). In some cases, edge router 112 (1) may perform the ECMP routing policy based on calculating a third hash value that represents the set of network five-tuple values included in the second data plane traffic. Because the data packet includes the set of network quintuple values, the third hash value may be equal to the first hash value and the edge router 112 (1) may forward the second data plane traffic to the load balancer 114 (1).

At operation 810, the example method 800 includes transmitting second data plane traffic to a second node based at least in part on the second data plane traffic having the set of five-tuple values. For example, load balancer 114 (1) may send second data plane traffic to data node 116 (N). In some examples, load balancer 114 (1) may send second data plane traffic 130 to data node 116 (N) based at least in part on not identifying the second SPI value. Additionally, load balancer 114 (1) may send second data plane traffic 130 to data node 116 (N) based at least in part on identifying that second data plane traffic 130 includes the set of network five-tuple values. For example, load balancer 114 (1) may not have an association stored between the second SPI value and the set of network quintuple values. However, because load balancer 114 (1) may have issued a key update request, load balancer 114 (1) may associate the second SPI value with the set of network five-tuple values. In other words, because load balancer 114 (1) issues a key update request, when load balancer 114 (1) receives data plane traffic that includes a set of known network five-tuple values and new/unknown SPI values, load balancer 114 (1) may associate the new/unknown SPI values with the known set of network five-tuple values and thus send the data plane traffic to data node 116 (N).

Fig. 9 illustrates a logic flow diagram of another example method 900 for performing encrypted tunnel migration. The example method 900 begins at operation 902, which includes receiving, at a load balancer and from a client device, a first data plane traffic having a first Security Parameter Index (SPI) value and a set of quintuple values.

At operation 904, the example method 900 includes transmitting first data plane traffic to the first node based at least in part on the first SPI value and the set of five tuple values. For example, load balancer 114 (1) may send first data plane traffic 130 to data node 116 (1). For example, the first SPI value may identify an IPsec SA between the client device and the data node 116 (1). In some examples, sending the first data plane traffic to the first node may be based at least in part on computing a hash value of the set of network five-tuple values representing the first SPI value and the first data plane traffic.

At operation 906, the example method 900 includes receiving, at a load balancer, an indication that at least a portion of first data plane traffic is to be sent to a second node. For example, the indication may indicate that the first node is operating at maximum load capacity or that the first node is about to lose connection, leave, be serviced, etc. The load balancer may receive an indication from a controller, such as controller 124, and/or a node, such as one of control node 118 or data node 116. In some examples, the indication may prompt the load balancer to send a key update request to one of the control nodes 118, e.g., to establish a new IPsec SA for the client device. Additionally or alternatively, the indication may prompt the load balancer to send a null ESP packet to the second data node (e.g., data node 116 (N)) that includes the IP address and port of the client device so that the second data node may begin to set up an interface to receive the portion of the first data plane traffic.

At operation 908, the example method 900 includes prompting the second node to provide one or more interfaces based at least in part on the indication such that at least the portion of the first data plane traffic may be sent to the second node. For example, load balancer 114 (1) may send an null ESP packet to data node 116 (N). The null ESP data packet may include an IP address and/or port associated with the client device such that data node 116 (N) may begin providing one or more interfaces for the portion of the first data plane traffic.

At operation 910, the example method 900 includes receiving, at the load balancer and from the client device, second data plane traffic having a second SPI value and the set of five tuple values. The second SPI value may comprise a new or unknown SPI value. That is, the load balancer may not have associated the second SPI value with the set of five-tuple values. At operation 912, the example method 900 includes determining that the second data plane traffic includes at least the portion of the first data plane traffic based at least in part on the second SPI value and the set of five tuple values. For example, load balancer 114 (1) may track all SPI values associated with the set of five-tuple values. In this way, if load balancer 114 (1) issues a key update request, load balancer 114 (1) may begin monitoring new/unknown SPI values associated with the set of five-tuple values to determine second data plane traffic comprising the portion of first data plane traffic comprising the set of five-tuple values and the second (new/unknown) SPI value.

At operation 914, the example method 900 includes sending second data plane traffic to a second node. For example, load balancer 114 (1) may send second data plane traffic to data node 116 (N). In some cases, sending the second data plane traffic to the second node may be based at least in part on determining that the second data plane traffic includes the portion of the first data plane traffic. Additionally or alternatively, sending the second data plane traffic to the second node may be based at least in part on the ECMP routing policy and calculate a hash value representing the second SPI value and the set of five tuple values.

Fig. 10 illustrates a schematic diagram of an example computer hardware architecture for implementing network nodes and/or devices (e.g., load balancers, control nodes, data nodes, etc.) that may be used to implement various aspects of the various techniques presented herein. Computer architecture as shown in fig. 10, fig. 10 illustrates a conventional server computer, network device, workstation, desktop computer, laptop computer, tablet computer, network device, electronic reader, smart phone, and/or other computing device, and may be used to execute any of the software components presented herein. The computer 1000 may include networking devices such as servers, switches, routers, hubs, bridges, gateways, modems, repeaters, access points, and the like.

Computer 1000 includes a substrate 1002 or "motherboard," which is a printed circuit board that may be connected to many components or devices by a system bus or other electrical communication path. In one illustrative configuration, one or more central processing units ("CPUs") 1004 operate in conjunction with a chipset 1006. The CPU 1004 may be a standard programmable processor that performs arithmetic and logic operations necessary for the operation of the computer 1000.

The CPU 1004 performs operations by transitioning from one discrete physical state to the next (by manipulating switching elements that distinguish and change these states). The switching elements typically include electronic circuitry, such as flip-flops, that hold one of two binary states, and include electronic circuitry, such as logic gates, that provide an output state based on a logical combination of the states of one or more other switching elements. These basic switching elements may be combined to create more complex logic circuits including registers, adders and subtractors, arithmetic logic units, floating point units, and the like.

The chipset 1006 provides an interface between the CPU 1004 and the remaining components and devices on the substrate 1002. The chipset 1006 may provide an interface to a RAM 1008 that serves as a main memory in the computer 1000. The chipset 1006 may further provide an interface to a computer-readable storage medium, such as a read-only memory ("ROM") 1010 or non-volatile RAM ("NVRAM"), for storing basic routines that help to boot the computer 1000 and transfer information between various components and devices. The ROM 1010 or NVRAM may also store other software components necessary for the operation of the computer 1000, according to the configurations described herein.

The computer 1000 may operate in a networked environment using logical connections to remote computing devices and computer systems through a network (e.g., the network 108 and/or the network 1024). The chipset 1006 may include functionality for providing network connectivity through the NIC 1012, such as a gigabit ethernet adapter. NIC 1012 is capable of connecting computer 800 to other computing devices via a network. It should be appreciated that a plurality of NICs 1012 may be present in the computer 1000 to connect the computer to other types of networks and remote computer systems. In some examples, NIC 1012 may be configured to perform at least some of the techniques described herein, and may include components for performing the techniques described herein.

The computer 1000 may be connected to a storage device 1018 that provides non-volatile storage for the computer. Storage 1018 may store operating system 1020, programs 1022, and data, which are described in greater detail herein. The storage device 1018 may be connected to the computer 1000 through a storage controller 1014 connected to the chipset 1006. Storage 1018 may be comprised of one or more physical storage units. The storage controller 1014 may interface with physical storage units through a serial attached SCSI ("SAS") interface, a serial advanced technology attachment ("SATA") interface, a fibre channel ("FC") interface, or other type of interface for physically connecting and transferring data between a computer and the physical storage units.

The computer 1000 may store data on the storage devices 1018 by transforming the physical state of the physical storage units to reflect the information being stored. In different embodiments of the present description, the specific transition of the physical state may depend on various factors. Examples of such factors may include, but are not limited to, the technology used to implement the physical storage unit, whether storage 1018 features main memory or secondary memory, and the like.

For example, the computer 1000 may store information to the storage device 1018 by issuing instructions via the storage controller 1014 to change magnetic properties at a particular location within the disk drive unit, reflective or refractive properties at a particular location in the optical storage unit, or electrical properties of a particular capacitor, transistor, or other discrete element in the solid state storage unit. Other transformations of physical media are possible without departing from the scope and spirit of the present specification, with the foregoing examples provided merely for convenience of description. Computer 1000 may further read information from storage 1018 by detecting a physical state or characteristic of one or more specific locations within the physical storage unit.

In addition to the mass storage devices 1018 described above, computer 1000 may access other computer-readable storage media to store and retrieve information, such as program modules, data structures, or other data. It should be appreciated by those skilled in the art that computer-readable storage media are any available media that provide non-transitory storage of data and that can be accessed by the computer 1000. In some examples, operations performed by system architecture 100 and/or any components included therein may be supported by one or more devices similar to computer 1000. In other words, some or all of the operations performed by system architecture 100 and/or any components included therein may be performed by one or more computer devices 1000 operating in a cloud-based arrangement.

By way of example, and not limitation, computer-readable storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology. Computer-readable storage media includes, but is not limited to, RAM, ROM, erasable programmable ROM ("EPROM"), electrically erasable programmable ROM ("EEPROM"), flash memory or other solid state memory technology, optical disk ROM ("CD-ROM"), digital versatile disks ("DVD"), high definition DVD ("HD-DVD"), BLU-RAY or other optical storage device, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage device, or any other medium which can be used to store the desired information in a non-transitory manner.

As described above, the storage device 1018 may store an operating system 1020 for controlling the operation of the computer 1000. According to one embodiment, operationsThe system includes a LINUX operating system. According to another embodiment, the operating system comprises a software program from Microsoft corporation of Redmond, washingtonThe SERVER operating system. According to a further embodiment, the operating system may comprise a UNIX operating system or one of its variants. It should be appreciated that other operating systems may be used. The storage devices 1018 may store other systems or applications and data for use by the computer 1000.

In one embodiment, the storage device 1018 or other computer-readable storage medium is encoded with computer-executable instructions that, when loaded into the computer 1000, transform the computer from a general-purpose computing system into a special-purpose computer capable of implementing the embodiments described herein. As described above, these computer-executable instructions transform the computer 1000 by specifying how the CPU 1004 transitions between states. According to one embodiment, the computer 1000 may access a computer-readable storage medium storing computer-executable instructions that, when executed by the computer 1000, perform the various processes described above with respect to fig. 1-9. Computer 1000 may also include a computer-readable storage medium having stored thereon instructions for performing any of the other computer-implemented operations described herein.

The computer 1000 may also include one or more input/output controllers 1016 for receiving and processing input from a number of input devices, such as a keyboard, mouse, touchpad, touch screen, electronic stylus, or other type of input device. Similarly, an input/output controller 1016 may provide output to a display, such as a computer monitor, flat panel display, digital projector, printer, or other type of output device. It should be understood that computer 1000 may not include all of the components shown in fig. 10, may include other components not explicitly shown in fig. 10, or may utilize an architecture entirely different from that shown in fig. 10.

As described herein, the computer 1000 may include a data nodeOne or more of a control node, a firewall node, an edge router, and/or a key value store. The computer 1000 may include one or more hardware processors 1004 (processors) configured to execute one or more stored instructions. Processor 1004 may include one or more cores. In addition, computer 1000 may include one or more network interfaces (e.g., NIC 1012) configured to provide communication between computer 1000 and other devices over a network (e.g., networks 108 and 1024). The network interface may include devices configured to couple to Personal Area Networks (PANs), wired and wireless Local Area Networks (LANs), wired and wireless Wide Area Networks (WANs), and the like. For example, the network interface may include a network interface with Ethernet, wi-Fi ^TM Etc. compatible devices.

Program 1022 may include any type of program or process to perform the techniques described in this disclosure to load balance encrypted traffic based on the SPI value of the packet header, and to use the SPI value to indicate QoS and migrate encrypted connections to different hosts.

In summary, techniques for load balancing encrypted traffic based on a Security Parameter Index (SPI) value of a packet header and a five-tuple value set of the packet header are described herein. Further, techniques for including quality of service (QoS) type information in an SPI value field of a packet header are described herein. The QoS type information may indicate a particular traffic class according to which the data packet is processed. In addition, techniques for pre-configuring a back-end host such that encrypted traffic may be migrated from another back-end host to the back-end host without causing temporary service disruption are also described herein.

Although the application has been described with respect to particular examples, it should be understood that the scope of the application is not limited to these particular examples. For example, while many examples are described with respect to the IPsec protocol, it should be understood that the described techniques are applicable to other protocols. Since other modifications and changes varied to fit particular operating requirements and environments will be apparent to those skilled in the art, the application is not deemed limited to the examples selected for disclosure purposes and is intended to cover all such variations and modifications as do not constitute a departure from the true spirit and scope of the application.

Although the application has been described in terms of particular structural features and/or acts, it is to be understood that the claims are not necessarily limited to the specific features or acts described. Rather, the specific features and acts are merely illustrative of some embodiments that fall within the scope of the claims of the present application.

Claims (23)

1. A method, comprising:

receiving, from a client device and at a network device of a network, a request to establish an encrypted tunnel through the network such that data plane traffic flows between the client device and a service via the encrypted tunnel;

Determining that the data plane traffic belongs to a particular traffic class of a set of traffic classes, the particular traffic class being associated with a particular quality of service (QoS) performance metric;

generating a Security Parameter Index (SPI) value to be used by the client device for the data plane traffic, the SPI value corresponding to the specific traffic class;

transmitting an indication of the SPI value to the client device;

receiving, at a load balancing node associated with the network, data packets of the data plane traffic including the SPI value; and

the data packet is transmitted over the network based at least in part on the data packet including the SPI value such that the data packet is processed according to the particular QoS performance metric.

2. The method of claim 1, wherein generating the SPI value comprises:

generating a first bit combination representing the particular QoS performance metric for the data packet to be processed based at least in part on the particular traffic class;

generating a second bit combination representing a security association; and

masking the first bit combination and the second bit combination such that the first bit combination includes a first portion of the SPI value and the second bit combination includes a second portion of the SPI value.

3. The method of claim 2, wherein the first bit combination is represented by a first hexadecimal number and the second bit combination is represented by a plurality of hexadecimal numbers.

4. A method as recited in any of claims 1-3, wherein the first portion of the SPI value is a first identifier corresponding to the particular traffic class and the second portion of the SPI value is a second identifier corresponding to a security association of the network.

5. A method as recited in any of claims 1-4, wherein the data packet is a first data packet, the SPI value is a first SPI value, the specific traffic class is a first traffic class, and the specific QoS performance metric is a first QoS performance metric, the method further comprising:

receiving, at the load balancing node, a second data packet including a second SPI value corresponding to a second traffic class, the second traffic class being associated with a second QoS performance metric; and

the second data packet is transmitted over the network based at least in part on the second data packet including the second SPI value such that the second data packet is processed according to the second QoS performance metric.

6. The method of any of claims 1-5, wherein transmitting the data packet over the network comprises transmitting the data packet over the network using an equal cost multi-path (ECMP) routing algorithm based at least in part on the SPI value and a quintuple of the data packet.

7. A method as recited in any of claims 1-6, wherein generating the SPI values comprises generating a plurality of SPI values to be used by the client device for the data plane traffic, each of the plurality of SPI values corresponding to a respective traffic class, each respective traffic class being associated with a respective QoS performance metric.

8. A system, comprising:

one or more processors; and

one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:

receiving, from a client device, a request to establish an encrypted tunnel over a network such that data plane traffic flows between the client device and a service via the encrypted tunnel;

determining that the data plane traffic belongs to a particular traffic class of a set of traffic classes, the particular traffic class being associated with a particular quality of service (QoS) performance metric;

Generating a Security Parameter Index (SPI) value to be used by the client device for the data plane traffic, the SPI value corresponding to the specific traffic class;

transmitting an indication of the SPI value to the client device;

receiving data packets of the data plane traffic from the client device that include the SPI value; and

the data packet is transmitted over the network based at least in part on the data packet including the SPI value such that the data packet is processed according to the particular QoS performance metric.

9. A system as in claim 8, wherein generating the SPI value comprises:

generating a first bit combination representing the particular QoS performance metric for the data packet to be processed based at least in part on the particular traffic class;

generating a second bit combination representing a security association; and

masking the first bit combination and the second bit combination such that the first bit combination includes a first portion of the SPI value and the second bit combination includes a second portion of the SPI value.

10. The system of claim 9, wherein the first bit combination is represented by a first hexadecimal number and the second bit combination is represented by a plurality of hexadecimal numbers.

11. A system as recited in any of claims 8-10, wherein the first portion of the SPI value is a first identifier corresponding to the particular traffic class and the second portion of the SPI value is a second identifier corresponding to a security association of the network.

12. The system of any of claims 8 to 11, wherein the data packet is a first data packet, the SPI value is a first SPI value, the particular traffic class is a first traffic class, and the particular QoS performance metric is a first QoS performance metric, the operations further comprising:

receiving a second data packet, the second data packet including a second SPI value corresponding to a second traffic class, the second traffic class being associated with a second QoS performance metric; and

the second data packet is transmitted over the network based at least in part on the second data packet including the second SPI value such that the second data packet is processed according to the second QoS performance metric.

13. The system of any of claims 8-12, wherein transmitting the data packet over the network comprises transmitting the data packet over the network using an equal cost multi-path (ECMP) routing algorithm based at least in part on the SPI value and a quintuple of the data packet.

14. A system as recited in any of claims 8-13, wherein generating the SPI values comprises generating a plurality of SPI values to be used by the client device for the data plane traffic, each of the plurality of SPI values corresponding to a respective traffic class, each respective traffic class being associated with a respective QoS performance metric.

15. A non-transitory computer-readable medium storing instructions that, when executed by one or more computing devices, cause the computing devices to perform operations comprising:

receiving, from a client device, a request to establish an encrypted tunnel over a network such that data plane traffic flows between the client device and a service via the encrypted tunnel;

determining that the data plane traffic belongs to a particular traffic class of a set of traffic classes, the particular traffic class being associated with a particular quality of service (QoS) performance metric;

generating a Security Parameter Index (SPI) value to be used by the client device for the data plane traffic, wherein a first portion of the SPI value corresponds to the particular traffic class;

transmitting an indication of the SPI value to the client device;

receiving data packets of the data plane traffic from the client device that include the SPI value; and

The data packet is transmitted over the network based at least in part on the data packet including the SPI value such that the data packet is processed according to the particular QoS performance metric.

16. The non-transitory computer-readable medium of claim 15, wherein generating the SPI value comprises:

generating a first bit combination representing the particular QoS performance metric for the data packet to be processed based at least in part on the particular traffic class;

generating a second bit combination representing a security association; and

masking the first bit combination and the second bit combination such that the first bit combination includes a first portion of the SPI value and the second bit combination includes a second portion of the SPI value.

17. The non-transitory computer readable medium of claim 16, wherein the first bit combination is represented by a first hexadecimal number and the second bit combination is represented by a plurality of hexadecimal numbers.

18. The non-transitory computer readable medium of any of claims 15-17, wherein the second portion of the SPI value is a second identifier corresponding to a security association of the network.

19. The non-transitory computer readable medium of any one of claims 15 to 18, wherein the data packet is a first data packet, the SPI value is a first SPI value, the particular traffic class is a first traffic class, and the particular QoS performance metric is a first QoS performance metric, the operations further comprising:

Receiving a second data packet comprising a second SPI value, a first portion of the second SPI value corresponding to a second traffic class, the second traffic class being associated with a second QoS performance metric; and

the second data packet is transmitted over the network based at least in part on the second data packet including the second SPI value such that the second data packet is processed according to the second QoS performance metric.

20. The non-transitory computer readable medium of any of claims 15-19, wherein generating the SPI values comprises generating a plurality of SPI values to be used by the client device for the data plane traffic, each of the plurality of SPI values corresponding to a respective traffic class, each respective traffic class associated with a respective QoS performance metric.

21. An apparatus, comprising:

means for receiving, from a client device and at a network device of a network, a request to establish an encrypted tunnel through the network such that data plane traffic flows between the client device and a service via the encrypted tunnel;

means for determining that the data plane traffic belongs to a particular traffic class of a set of traffic classes, the particular traffic class associated with a particular quality of service (QoS) performance metric;

Means for generating a Security Parameter Index (SPI) value to be used by the client device for the data plane traffic, the SPI value corresponding to the specific traffic class;

means for sending an indication of the SPI value to the client device;

means for receiving data packets of the data plane traffic including the SPI value at a load balancing node associated with the network; and

means for: the data packet is transmitted over the network based at least in part on the data packet including the SPI value such that the data packet is processed according to the particular QoS performance metric.

22. The apparatus of claim 21, further comprising means for performing the method of any one of claims 2 to 7.

23. A computer program, computer program product or computer readable medium comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method according to any of claims 1 to 7.