CN107925681A

CN107925681A - The system and method that the load balance in network packet core system is defined for distributed software

Info

Publication number: CN107925681A
Application number: CN201680050430.1A
Authority: CN
Inventors: 波克·克努恩·伦道夫·郑; 哈桑·西普拉; 安库尔·贾殷
Original assignee: Google LLC
Current assignee: Google LLC
Priority date: 2015-09-23
Filing date: 2016-09-20
Publication date: 2018-04-17
Anticipated expiration: 2036-09-20
Also published as: CN107925920B; EP3726815A1; GB2556745A; GB201801423D0; GB2556745B; KR20180035234A; DE112016004283T5; GB2556746B; EP3726815B1; KR101998355B1; DE112016004286T5; CN107925681B; CN108886825B; EP3318047A4; US20170086115A1; EP3318084B1; JP6513878B2; JP6564934B2; EP3318047B1; US20200092207A1

Abstract

The packet being used for by least one processing in multiple SDN network function modules of local SDN infrastructure example is received for defining the stateless load balancer in network (SDN) block core system in local SDN infrastructure example of the balanced loaded system and method including distribution SDN block core systems in distributed software.The packet can be directed to the stateful load balancer resided on the first computing device among the multiple computing device in the local SDN infrastructure examples by the stateless load balancer.The stateful load balancer can determine that the stateful packet to be applied to the packet handles task and determines for identified stateful packet to be handled task application to the second computing device in the local SDN infrastructure examples of the packet.The packet and then can be forwarded to second computing device by the stateful load balancer.

Description

The system that the load balance in network packet core system is defined for distributed software And method

Cross reference to related applications

This application claims entitled " the DISTRIBUTED SOFTWARE DEFINED submitted for 23rd in September in 2015 The U.S. Provisional Application No. 62/222,406 of WIRELESS PACKET CORE SYSTEM "；Submitted on October 16th, 2015 Entitled " SYSTEMS AND METHODS FOR MOBILITY MANAGEMENT IN A DISTRIBUTED SOFTWARE The U.S. Provisional Application No. 62/242,668 of DEFINED NETWORK PACKET CORE SYSTEM "；And in 2015 10 Entitled " the SYSTEMS AND METHODS FOR LOAD BALANCING IN A DISTRIBUTED that the moon is submitted on the 16th The U.S. Provisional Application No. 62/242,677 of SOFTWARE DEFINED NETWORK PACKET CORE SYSTEM " it is preferential Power, its content are incorporated herein by reference.

Technical field

The disclosure relates generally to the field of cordless communication network.

Background technology

More users will be switched to and (either more usually use) mobile network to access internet or lead to other users Letter.Since many mobile equipment are equipped with the communication capacity associated with various radio access technologies, thus user has base In the flexibility that such as availability, cost, network performance or its combination switch between different radio access networks.

The content of the invention

According at least one aspect of the disclosure, a kind of distributed software defines network (SDN) block core system can Local SDN infrastructure example including multiple interconnections so that each local SDN infrastructure example includes geographically connecing The cluster of the nearly computing device arranged and be communicably coupled to corresponding multiple radio access nodes.Each local SDN bases Facility example can include representing by computer equipment, stateless load balancer and multiple stateful load balancing modules Multiple SDN network function modules of the example of the executable network function of one or more.Stateless load balancer can be by It is configured to receive the packet being used for by least one processing in network function and packet is directed to local SDN One in computing device in infrastructure example.Each stateful load balancing module can reside in corresponding calculate and set In standby and (1) can be configured to determine to be applied to the stateful of the packet received at corresponding computing device Packet handles task, and (2) are determined for identified stateful packet processing task application to be arrived packet Packet is forwarded to identified computing device by computing device and (3) in local SDN infrastructure example.Each originally Ground SDN infrastructure example can also include the computing device being configured in management instruction local SND infrastructure examples The load manager of the information of load and mode of operation with multiple SDN network function modules.

According at least one aspect of the disclosure, one kind is in distributed software defines network (SDN) block core system Balanced loaded method includes：Loaded by the stateless in the local SDN infrastructure example of distributed SDN block cores system Balancer receives the packet being used for by least one processing in multiple SDN network function modules.Multiple network function moulds Block is executable on multiple computing devices of local SDN infrastructure example.Method further includes：Stateless load balancer Packet is directed to and is resided on the first computing device among multiple computing devices in local SDN infrastructure example Stateful load balancer.Method further includes：Stateful load balancer determines the stateful number to be applied to packet According to packet-processing task；Determine for identified stateful packet to be handled task application to the local of packet The second computing device in SDN infrastructure examples；And packet is forwarded to the second computing device.

Brief description of the drawings

Fig. 1 shows that diagram defines the general introduction of the communication environment of network (SDN) block core system using distributed software Block diagram.

Fig. 2 is the block diagram of the diagram SDN element associated with distribution SDN block core systems.

Fig. 3 is that the load for the processing load that diagram is used in the distributed SDN block cores system shown in balance chart 2 is put down The block diagram of balance system.

Embodiment

The mobile data services of increased number of mobile subscriber and incrementss are needed by Mobile Network Operator in network base Continuous and expensive investment in Infrastructure.In mobile subscriber can more frequently be accessed by various radio access networks Hold and quotient data is provided.Mobile equipment, which can be equipped with, to be used to access and different radio access technologies (RAT) such as third generations (3G) mobile communication technology, Long Term Evolution (LTE) mobile communication technology, WiFi, proprietary High Altitude Platform or other be wirelessly electrically accessed The ability for the different radio access networks that technology is associated.

When accessing internet, mobile subscriber can depend on corresponding user's context and different connection sexual norms it Between switch.For example, mobile subscriber can be in by WiFi, outdoors by honeycomb access network or in retail space, fly Access internet is serviced by visitor WiFi in airport or other public places.User usually has authenticates to difference by oneself Access network, different bills arrangement and different safeties or strategy different modes.Moreover, different connects The roaming entered between network causes connectivity to be interrupted.These factors cause the user experience of fragmentation.Mobile subscriber can like carrying For the uniform service of the combination of the isomery radio access schemes with the Seamless Mobile between various RAT.

Improve one in the mode of user experience and be to provide and enable isomery radio access network cohesion to single Hierarchical system framework in service-domain.In system architecture as design, different factors will be considered, including system can expand Moved in malleability (for example, on new RAT or on the increase in network capabilities), system complexity, infrastructure cost, RAT Dynamic mobility between property and RAT.Interworking Scheme as realization can be complicated and be also possible to hinder system architecture on new The scalability of RAT.Distributed and level block core system architecture can allow for the improvement on these factors (relative to existing There is block core system).Especially, level block core system can be arranged so that substantially global and opposite The network function of ground less latency sensitive (for example, compared with other network functions) is implemented as software defined network based on cloud Network (SDN) function, and the network function of substantial part and latency sensitive is implemented in and is in close proximity to connecing for its service In the region pod (pod) (or computational domain) for entering network (either corresponding radio access node).Region pod can include having Realize the cluster of the computing device of network function block on it and can be configured to service corresponding geographic area.Each Region pod can include multiple examples of one or more network function blocks to allow increased capacity and parallel processing.

In such distributed and level deployment, how one in problem to be solved be in various region pods, phase Route test face and data surface business between the computing device and/or corresponding network function block answered.Chain of command business includes using Carry out the message of the foundation of user conversation of the signal notice for example between radio access node and block core system and for leading to Cross the routing iinformation of packet core system guiding data surface business.Data surface business be included in user equipment and internet (or Another user equipment) between exchange and the packet transmitted by block core system tunnel.Route test face and data The processing load that face business includes to various region pods, corresponding computing device and/or corresponding network calculations block is balanced. In the disclosure, level load balance system and balancing method of loads are arranged to balanced distribution formula software defined network (SDN) the processing load in block core system.

Fig. 1 shows that diagram defines the general of the communication environment 10 of network (SDN) block core system using distributed software The block diagram stated.Communication environment 10 can include service mobile equipment 20 in its vicinity multiple radio access nodes (AP) 30, Multiple local SDN infrastructure examples 110 (being also known as region pod 110 herein), center SDN infrastructure example 120, In one or more Metropolitan Area Network (MAN)s 12, the first transition network 15, one or more second transition networks 18 and one or more data The heart 90.Each local SDN infrastructure example 110 can include multiple computer equipments 115, it, which is configured to perform, is used to take The network function for the mobile equipment 20 being engaged near the SDN infrastructure example 110.

Mobile equipment 20 include but not limited to mobile phone, smart phone, tablet computer, laptop computer, intelligent watch, It is wearable items, audio-frequence player device (such as MP3 player), video playback apparatus, game station with communication capacity, complete Ball alignment system (GPS) equipment, automobile computer or any other client device with communication capacity.Mobile equipment 20 At corresponding access point 30 mobile access network can be attached to via Radio Link.In some embodiments, movement is set Standby 20 can be configured to via distributed SDN block cores system request and access associated with content-provider networks Content.For example, the application (such as browser, media player, game application/platform, the electronics postal that are run in mobile equipment 20 Part application, enterprise's application, social media application, cloud application etc.) it can ask to be used to show from 90 available content of data center On the screen of mobile equipment 20.

Radio access node (AP) 30 can be configured to communicate with mobile equipment 20 via Radio Link.AP 30 can Be be associated with least one radio access technologies (RAT) such as 3G, LTE, LET-A, Wi-Fi or other RAT it is separated Radio access network (such as Wi-Fi hotspot network, Universal Mobile Communication System (UMTS) network, Long Term Evolution (LTE) network Or any other mobile communications network) a part.For example, AP 30 can include node B, enode b (e node B), Wi-Fi hotspot access point or other types of radio access node.AP 30 can be distributed on the one or more states of definition, state In family or multiple national big geographic areas.Each AP 30 can be configured to service the shifting in corresponding geographic proximity Dynamic equipment 20.Each AP 30 can be configured to by 20 direct communication of mobile equipment in air interface and its neighborhood and Corresponding local SDN infrastructure example 110 can be coupled to by city ISP network 12.When mobile equipment 20 is from one When geographical location is moved to another geographical location, corresponding wireless connection can be based on mobile equipment 20 for different AP 30 proximity is transmitted (or transfer) to another AP 30 from an AP 30.

Multiple local SDN infrastructure examples 110 represent to be in close proximity to radio access network or its AP 30 is disposed And service its Distributed Calculation or network processes resource pool set.For example, each SDN infrastructure example 110 can It is associated and can be coupled to the AP 30 for servicing the mobile equipment 20 in the urban area with corresponding urban area.Often A local SDN infrastructure example 110 can include corresponding multiple (such as more than ten, tens or other quantity) The rack of computer equipment 115, such as computer server or server, it is configured to execution and is implemented as SDN elements With the network of the service data service associated with the AP 30 that (or service) is coupled by local SDN infrastructure example 110 Operation or function.In some embodiments, the computer being associated with corresponding local SDN infrastructure example 110 is set Standby 115 can be configured to perform delay-sensitive network function.For example, control plane signaling, data surface grappling and mobility operation It can be implemented as (or running on it) the SDN elements that can perform by corresponding computer equipment 115.

In some embodiments, each local SDN infrastructure example 110 can include being configured to balance accordingly The load balance entity of load between computer equipment 115 or the SDN elements run on it.In some embodiments In, each local SDN infrastructure example 110 can be configured to local cache and one or more content suppliers phase Associated internet data, cloud data, inline network data or its combination.For example, 110 energy of each local SDN infrastructure example It is enough to include one or more cache servers.Local SDN infrastructure example 110 can be configured in response to from shifting The request of dynamic equipment 20, when being server based or locally cached requested internet data, interconnects from local cache service Network data.Internet data can include webpage, the number associated with online service (Email, social media etc.) According to, the data associated with game, video stream data (Netflix or YouTube) etc..In some embodiments, originally Ground SDN infrastructure example 110 can be configured to local cache (and service) internet data, inline network data, Cloud data or its combination.

Center SDN infrastructure example 120 can include have run on it and be configured to perform be not (or It is at least less) SDN of the network function of latency sensitive or the network function comprising multiple local SDN infrastructure examples 110 One or more computer equipments of element.The SDN elements run in center SDN infrastructure example 120 can include example Such as subscribe to policy module, the global mould for managing the mobility between local SDN infrastructure example 110 (or roaming) Block or its combination.Mandate, certification and book keeping operation (AAA) server, own subscription profile thesaurus can be included by subscribing to policy module (SPR), strategy and charging control module or other modules for managing subscription strategy.It is attached to point by mobile equipment 20 During cloth SDN block core systems, local SDN infrastructure example 110 can from center SDN infrastructure example 120 Associated subscription and the certification or mandate of the mobile equipment 20 (or its user) of policy module request.Center SDN infrastructure Example 120 can also include being used for the global load-balancing mould for balancing the load between multiple local SDN infrastructure examples 110 Block.Center SDN infrastructure example 120 can be disposed in cloud, such as in host data center 91.

Each local SDN infrastructure example 110 can be coupled to the AP near it by corresponding Metropolitan Area Network (MAN) 12 30.Metropolitan Area Network (MAN) 12 can from a local SDN infrastructure example 110 to another local SDN infrastructure example 110 without Together.In some embodiments, local SDN infrastructure example can be coupled to accordingly by one or more Metropolitan Area Network (MAN)s 12 AP 30.For given local SDN infrastructure example 110, corresponding Metropolitan Area Network (MAN) 12 can be service and local SDN One or more communication networks of the associated urban area of infrastructure example 110.In some embodiments, Metropolitan Area Network (MAN) 12 One or more of can be possessed by corresponding third party's network provider or manage and can not be distributed SDN point A part for group core network.In some embodiments, it is real can to service multiple local SDN infrastructure for a Metropolitan Area Network (MAN) 12 Example 110.In some embodiments, each AP 30 can be coupled to geographical nearest local SDN infrastructure example 110 (and by its service).In some embodiments, the entity in AP 30 and local SDN infrastructure example 110 is (such as Computer equipment 115 or the SDN elements run on it) between coupling can pass through corresponding transmission tunnel on Metropolitan Area Network (MAN) 12 Realize in road 35.For example, transmission tunnel 35 (such as ethernet ip (EoIP) tunnel) can be by providing to local SDN infrastructure The internuncial Metropolitan Area Network (MAN)s 12 of IP of example 110 use.In some embodiments, transmission tunnel 35 can include ethernet ip tunnel Road, common user datagram protocol (UDP) encapsulation (GUE) tunnel, generic route encapsulation (GRE) tunnel, 802.11GUE tunnels, GPRS Tunnel Protocol (GTP) tunnel, Internet protocol (IP) security (IPSec) tunnel, other types of tunnel or its combination.

In some embodiments, multiple local SDN infrastructure examples 110 can be mutual by the first transition network 15 Connection.First transition network 15 allows the communication between separated local SDN infrastructure example 110 for example local to handle Movement (or roaming) and signaling between SDN infrastructure example 110.In some embodiments, the first transition network 15 It can be configured to local SDN infrastructure example 110 being coupled to center infrastructures example 120.First transition network 15 It can include fiber data network, ISP's (ISP) network or other communication networks.First transition network 15 Can be by single entity or separated entity management (or possessing) with multiple local SDN infrastructure examples 110.In some realities Apply in mode, the first transition network 15 discretely can possess or manage with local SDN infrastructure example 110.That is, first Transition network 15 can not be a part for distributed SDN block cores system but be used for local SDN infrastructure example 110 external networks communicatively coupled with one another.

Communication environment 10 can include multiple data centers 90.Each data center 90 can include being used to store and carry For to content provider data, web page, cloud data or the computing device of the access of its combination.For example, data center 90 can It is configured to receive request of data from local SDN infrastructure example 110, and in response, there is provided requested data.Number Trustship web page and corresponding content, video stream application such as YouTube or Netflix, society can be configured to according to center 90 Hand over media application and content, game application, enterprise's application or any other cloud application or service.In some embodiments, in Heart SDN infrastructure example 120 can be implemented in host data center 91.

Each local SDN infrastructure example 110 can be communicatively coupled by one or more second transition networks 18 To data center 90.Second transition network 18 can be coupled to local SDN infrastructure example by the first transition network 15 110.In some embodiments, the second transition network 18 can be coupled directly to local SDN infrastructure example 110. In some embodiments, the second transition network 18 can be configured to center SDN infrastructure example 120 being coupled to this ground Infrastructure example 110.In some embodiments, the second transition network 18 can be optional.In such embodiment, Data center 90 can be coupled directly to the first transition network 15.In some embodiments, the first transition network 15 and Two transition networks 18 can be a part for single communication network.

In some embodiments, distributed SDN block cores system can be considered as local SDN infrastructure example 110 and the combination of center SDN infrastructure example 120.In some embodiments, distributed SDN block cores system can also Enough include the first transition network 15.In some embodiments, distributed SDN block cores system can also include the second switching Network 18.The framework of distributed SDN block cores system (as described in Figure 1 on) allows data service locally to be route And disconnection, for example, at local SDN infrastructure example 110.It is being in close proximity to arrangement local SDN infrastructure at AP 30 Example 110 and handling data service therein allows quick packet to handle and therefore improved performance and reduce Network resource consumption.

Distributed SDN block cores system can be across large geographical area (such as one or more states or one or more Country) it is widely distributed and be configured to track and manage the shifting for the substantial amounts of mobile equipment 20 being distributed in large geographical area Dynamic property.Mobility tracks and management can be by the mobility work(based on SDN that is implemented in distributed SDN block cores system It can perform.The scalability of function and optimization as the Software Implementation permission of mobility functions.However, with mobility work( Delay that can be associated and scalability can depend on the corresponding Software Implementation used.Network work(is used for example, working as Can separated mobility of virtualization (NFV) realization for the different RAT in cloud (such as in data center 90 or 91) During function (mobility management entity (MME) such as LTE), for give the scalability of RAT usually by replicate with All mobility functions that the RAT is associated are realized.Replicating the highly complex function associated with substantial amounts of maintenance state can Cause the poorly efficient use of electric power and computing resource, especially if needing to increase in the subset of local SDN infrastructure example Mobility process resource when.Moreover, the synchronization of corresponding state and therefore further increase will be needed by replicating mobility functions The complexity of block core system.Moreover, realize that the mobility functions in cloud can result in high mobility processing delay.Especially Ground, although local SDN infrastructure example 110 can be (big with user or corresponding mobile equipment 20 close to population center Type set is associated) arrangement, but data center (such as data center 90 and 91) tends to be positioned in long-range geographical position In putting.Therefore, in the case that sexual function movable within is implemented in cloud, the processing of each mobility event, which includes, passes through number According to the signaling on the long communication link between center 91 and one or more locals SDN infrastructure example 110.

In the disclosure, level load-balance model (LB models) is used between the different entities of communication environment 10 Route test face (C planes) and data surface (D planes) business, and set in SDN infrastructure example 110 with corresponding computer Carried out between the SDN elements run on the standby 115 or SDN in computer equipment 115 and distribution SDN block core systems Load balance.Level LB models can be configured to apply the LB of coarseness between various SDN infrastructure examples 110, and And in each SDN infrastructure example between the SDN elements in computer equipment 115 and the SDN infrastructure example 110 Fine-grained load balance is applied in 110.LB models can also allow when the failure of whole SDN infrastructure example 110 or manage Elasticity during the cut-out service of reason ground.

Fig. 2 is the block diagram of the diagram SDN element associated with distribution SDN block cores system 200.Distributed SDN points Group core system 200 includes center SDN infrastructure example 220 based on cloud and multiple local SDN infrastructure examples 210 (being also known as region pod 210).Center SDN infrastructure example 220 based on cloud include subscriber database (subscriber DB) 222, Global SDN mobility controls device 224, global load-balancing module 226 and policy database (tactful DB) 228.Each region Pod 210 includes multiple WiFi front ends (WFE) module 212a, multiple LTE front-end modules 212b, certification, authorization and accounting (AAA) generation Manage module 215, policy module 216, fire wall/network address translation (FW/NAT) module 217, Lawful intercept module 213, multiple D planar applications example 218, mobility manager 214 and pod LB managers 219.Each WFE modules 212 are communicably coupled to simultaneously And service one or more WiFi AP 230a.Each LFE modules 212 are communicably coupled to and service one or more e sections Point B 230b.

The different functions of distributed block core system 200 have the characteristic for needing different Distribution Strategies.Some nets Network function or module such as subscriber database 222, policy database 224, overall situation LB modules 226 and overall situation SDN mobility controls Device 22 processed is substantially global and the low cost provided in large centralised data center can be utilized to calculate and storage Device.Therefore, such function (or module) is arranged to reside in center SDN infrastructure example 220 based on cloud. However, packet-processing function is latency sensitive and can benefit from and be as closely as possible to the specific AP 230 of RAT and dispose.And And local mobile event (that is, the mobility event occurred in single region pod 210) can be by residing in region pod Local mobile function in 210 rather than more efficiently handled by mobility functions based on cloud.By using local shifting Dynamic sexual function, can be arranged region pod 210 so that most of (such as 85%, 90%, 95% or other percentages) is used Family mobility event is treated as local mobile event.For example, each region pod 210 can be arranged to service corresponding city City region and the mobile equipment 20 associated with residing in the user in the urban area, which can be attached to, services the metropolitan district The region pod 210 in domain and corresponding family pod can be associated as with the region pod 210.In distributed SDN block cores In system 200, the network function of delay-sensitive network function and local property be arranged to reside in region pod 210 rather than In cloud.

Subscriber database 222 is able to maintain that the information related with the subscriber of distributed block core system 200, such as User identifier, device identification, user/device authorization information, user/device authentication information, user/equipment accounting information or its group Close.Policy database 224 be able to maintain that instruction to be applied to the communication associated with each subscriber or relevant device (or Person's session) tactful instruction.Both subscriber database 222 and policy database 224 can be configured to receive from various The inquiry of region pod 210 and it is responded.

Global LB modules 226 are configured to that region pod 210 will be routed to the request of chain of command or data surface treatment.Entirely Office's LB modules 226 are further configured to processing/computational load between equilibrium region pod 210.Global SDN mobility controls device 228 It is configured to mobility event in tracking pod and (that is, wherein moves equipment 20 in two AP being associated with separated region pod 210 The event of 230 internetwork roaming) and notify mobility event as various region pods.

Each region pod 210 is configured to service multiple WiFi AP 230a and e node B 230b (usually quilts herein The specific AP 230 of referred to as RAT).In general, can be configured to service associated with least one RAT for each region pod 210 Corresponding multiple specific AP 230 of RAT.The specific AP 230 of RAT can include Wi-Fi AP (or Wi-Fi Hotspot AP) 230a, with Enode b (e node B) 230b that LTE is associated, the node B (node B) associated with 3G network, other be wirelessly electrically accessed Point or its combination.That is, although Fig. 2 shows Wi-Fi AP 230a and e node B 230b, in general, distribution SDN Block core system 200 can service any combinations of AP 230.Although moreover, Fig. 2 show including WFE modules 212a and The region pod 210 of LFE modules 212b, but in general, each region pod 210 can include it is associated with least one RAT more A specific FE modules 212 of RAT.The specific FE modules 212 of RAT can include WFE modules 212a, LFE module, the third generation (3G) FE moulds Block (being used for service node B), with other RAT FE modules being associated or its combine.

Mobile equipment 20 can be attached to distributed SDN block cores system via one in the specific AP 230 of RAT 200.For example, mobile equipment 20 can based on the signal strength for example on each RAT for being supported by region pod 210, communication into This or user are subscribed to be attached to the specific AP 230 of the given RAT associated with given RAT.Associated with given RAT is every A specific AP 230 of RAT can be communicably coupled to the RAT associated with the RAT by Metropolitan Area Network (MAN) 12 (shown in Fig. 1) special Determine FE modules 212.For example, the WiFi AP 230a serviced by region pod 210 can be communicably coupled in the region pod 210 WFE modules 212a.Similarly, the e node B 230b serviced by region pod 210 can be communicably coupled to the LFE of the LMD 210 Module 212b.In general, each LMD 210 at least two LMD 210 in distribution SDN block core systems includes The specific FE modules 212 of the RAT associated with one or more RAT.

In some embodiments, the specific FE modules 212 of each RAT can be configured to pass via one or more covering The defeated AP 230 specific with RAT of tunnel 235 communicates.For example, each WFE modules 212a can be configured to via ethernet ip (EoIP) tunnel communicates with WiFi AP 230a.Using EoIP tunnels, WiFi AP 230a can make data such as UE 802.3 Frame independently channel transfer to WFE modules 212a.In some embodiments, the specific AP 230 of RAT are coupled to corresponding RAT The covering transmission tunnel 235 of specific FE modules 212 can include general UDP encapsulation (GUE) tunnel, generic route encapsulation (GRE) Tunnel, Ethernet gre tunneling, 802.11GUE tunnels, General Packet Radio Service (GPRS) tunnel protocol (GTP) tunnel, net Border protocol security (IPSec) tunnel or other communication tunnels.The specific FE modules 212 of RAT are configured to move in pod in processing RAT Sexual behavior part.That is, it is configured to handle with each RAT particular modules 212 that the corresponding RAT in given area pod 210 is associated It is associated with the RAT and by the mobility event between specific AP 230 of RAT that given area pod 210 services.In some realities Apply in mode, each specific FE modules 212 of RAT can be configured to maintain to be serviced by the specific FE modules 212 of the RAT each Mobile equipment 20 is mapped to the list of layer -2 (L2) binding for the specific AP 230 of RAT for servicing the movement equipment 20.Especially, L2 Binding can include device identification (device id) and to service AP marks (service AP-ID) right.In some embodiments, equipment ID can include media access control (MAC) address of mobile equipment 20 and service that AP-ID can include and corresponding RAT is special Determine the associated Tunnel End Point Identifiers (TEID) of AP 230.In some embodiments, the specific FE modules 212 of each RAT can Including for handling the RAT C planar applications of mobility event in pod in RAT and passing through the specific FE modules of the RAT for forwarding The RAT D planar applications of the packet of 212 tunnel transmissions.The RAT D being associated with the specific FE modules 212 of corresponding RAT are put down Face application, which can use, to be used to forward the corresponding L2 forwardings by the packet of specific 212 tunnel transmissions of FE modules of the RAT Database (FDB).

Each region pod 210 includes mobility manager 214, it is configured between processing RAT or mobile sexual behavior between pod Part (is also known as layer -3 (L3) mobility event).L3 mobility can be anchored at mobility manager 214.Mobility pipe Reason device 214 is further configured to serve as the Internet protocol of the mobile equipment 20 for being serviced by distributed SDN block cores system 200 (IP) anchor point.Especially, each mobility manager 214 is able to maintain that L3UE identifiers (such as IP address) and corresponding service Binding between the identifier (such as IP address) of the specific FE modules 212 of RAT.Each mobility manager 214 can include：One A or multiple mobility gateway C planar applications, it is configured to processing L3 mobility events and updates by mobility manager The 214 forwarding rules maintained；And one or more mobility gateway D planar applications, it is configured to act as use in movement Property manager 214 at maintain L3 forwarding rule Data Relay Entity.The mobility net associated with mobility manager 214 Close D planar applications and can use and be used for the corresponding of the packet that forwarding passes through 214 tunnel transmission of mobility manager L3FDB.For each mobile equipment 20 serviced by distributed SDN block cores system 200, L3FDB can include reflecting It is mapped to the corresponding IP address of the IP address of the corresponding service specific FE modules 212 of RAT.

Each region pod 210 includes multiple D planar applications examples 218, it is configured to processing will be by the region pod 210 The forwarding of the packet of tunnel transmission.Although D planar applications example 218 is shown as the specific FE of separated RAT in Fig. 1 212 and mobility manager 214, but D planar applications example 218 can include it is associated with the specific FE modules 212 of RAT RAT D planar applications 218 and the mobility gateway D planar applications associated with mobility manager 214.

Each region pod 210 includes certification, authorization and accounting (AAA) proxy module 215, it is configured to from subscriber data Storehouse 222 is obtained user's subscriber data and user or corresponding mobile equipment 20 is authenticated using such data, authorized Or charging.Region pod 210 further includes policy module 216, it is configured to obtain user tactics information from policy database 224 And subscriber policy is caused to be applied to the communication associated to relative users or corresponding mobile equipment 20.In some embodiments In, regions module 210 can include AAA proxy modules 215 or multiple examples of policy module 216.

Each regions module 210 includes FW/NAT modules 217, its be configured to by firewall security process apply for example to The packet associated with sub-network or given user group.In some embodiments, region pod 210 can include FW/NAT Multiple examples of module 217.Moreover, each regions module 210 includes multiple Lawful intercept modules 213, it is configured to intercept Packet and retrieve corresponding header or even content information.

Each region pod 210 further includes pod LB managers 219, it is configured to maintain instruction to be used in the region pod 210 Various computer equipments and various SDN elements health status and processing load information.Pod LB managers 219 can also be by Be configured to will indicate corresponding region pod 210 overall health and processing load information be delivered to reside in it is based on cloud Global LB modules 226 in center SDN infrastructure example 220.

Redundancy in the distributed structure/architecture of distributed SDN block cores system 200 and the example of many SDN elements can Allow the improved robustness and elasticity of distribution SDN block cores system 200.That is, if region pod 210 or computing device 115 shutdown or inoperable, then other region pods 210 or other computing devices 115 can be configured to carry inoperable The load of entity.Moreover, compared with the single instance SDN elements that may cause to handle bottleneck, the example of many SDN elements Redundancy in (the specific FE modules 212 of such as RAT, LI modules 213 or D planes example 218) can allow to be distributed on the contrary Corresponding processing load.In order to maximally utilize such framework, level and expansible load balancing mechanism be used for balance across Region pod 210 and/or the processing load across the computing device 115 in each region pod 210.

Fig. 3 is the negative of the processing load that diagram is used in the distributed SDN block cores system 200 shown in balance chart 2 Carry the block diagram of balance (LB) system 300 and corresponding LB mechanisms.LB systems 300 include overall situation SDN LB modules 340, it is resided in Front-end module 345 is inquired about in cloud and including LB.LB systems 300 also include stateless LB modules in each region pod 210 320th, pod LB managers 330 and multiple stateful LB modules 315.Illustrate as discussed, LB mechanisms can be considered as two Layer LB processes.

Referring to figs. 2 and 3, specific AP 230 of such as RAT of peer device 301 or with distributed SDN block cores system 200 associated another network elements can be configured to lead to based on the region pod 21 by distributed SDN block cores system 200 The service IP address of announcement, finds the network function example of distribution SDN block cores system 200.For example, 301 energy of peer device Enough it is attempt to the e for the mobility control device (or its mobility functions) that discovery is run in distributed SDN block cores system 200 Node B 230b or Wi-Fi AP 230a.

In some embodiments, each region pod 210 can be configured to externally expose (or notice) some clothes Business IP address.In some embodiments, the IP address for the notice of given area pod 210 need not reflect the region pod 210 Computing device 115 quantity.Quantity by making notice IP address is unconnected to the quantity of computing device 115, Mei Gequ The inside health and mode of operation (such as maintenance event) of corresponding computing device 115 in domain pod 210 can be from peer devices 301 maskings.For example, if the computing device 115 in corresponding region pod 210 is due to the event (for example, software upgrading) of scheduling Or non-scheduled event (for example, hardware fault) cut-out services, then peer device 302 is used for being connected to the region pod 210 IP address need not change.In some embodiments, area is usually compared for the quantity of the IP address of each region pod 210 notice The quantity of computing device 115 in domain pod 210 is much smaller (for example, an IP address).

In some embodiments, global LB modules 340 are configured to permit peer device 301 to position specific region pod 210.Especially, peer device 301 can pass through the phase of the particular network functions on distributed SDN block cores system 200 The LB inquiry front-end modules 345 answered inquire about global LB modules 340.In response to such inquiry, global LB modules 340 can select For servicing the specific region pod 210 of peer device 301 and being indicated to be sent to peer device 301.LB inquires about front end mould Block 345 can be configured to the type based on peer device 301 and expose different agreements.For example, LB inquires about 345 energy of front-end module Enough is to just know that the exposure DNS interfaces of peer device 301 that link address how is selected using dns resolution.Set for other equities Standby 301, LB inquiries front-end module 345 can be that peer device 301 exposes HTTP interface to inquire about overall situation LB for link address Module 340.

Global LB modules 340 being capable of the position based on peer device 301, the current health of one or more regions pod 210 (or operation) and loading condition or its combination, selection region pod 210.In some embodiments, global LB modules 340 can be by It is configured to maintain instruction from each potential subnet block (for example, for each/24IPv4 subnets) on radio access network To the information of the delay of each region pod 210.Global LB modules can also be configured to for example from corresponding pod LB managers 330 Receive the information for the active volume (or relatively available capacity) for indicating each region pod 210.For example, the opposite of given area pod can Can be the numeral between 0 and 100 with capacity, wherein, 100 mean that region pod 210 has all told and 0 means area Domain pod 210 is busy or inoperable and is unable to service request.In some embodiments, when request is from 301 quilt of peer device During reception, global LB modules 340 can select the region pod 210 with minimum delay.If two region pods 210 have similar Delay, then overall situation LB modules 340 can select the region pod 210 with maximum available.In some embodiments, it is global LB modules 340 can be configured to the score based on the weighted sum for being defined as delay metric and pod capacity measurement, selection region Pod 210.

, can be in selected region pod 210 originally once peer device 301 is directed into selected region pod 210 Ground performs more fine-grained load balance.Local load balance at the pod of region can include two load balance levels. At one level, processing request (or its packet) is received by the stateless LB modules 320 of selected region pod 210.It is ill-mannered State LB modules 320 can be configured to one or more fields (such as 5 tuples of IP) of packet that retrieval receives and Use one that the packet received is directed in the available computing device 115 in region pod 210 by retrieved field. In some embodiments, stateless LB modules 320 can based on the network function supported by each computing device 115, one Or the mode of operation of multiple computing devices, the load condition of one or more computing devices 115 or its combination, selection region pod Computing device 115 in 210.In some embodiments, stateless LB modules 320 can be configured to be incited somebody to action according to poll pattern The processing request received from peer device 301 is directed to various computing devices 115.

In some embodiments, whether stateless LB modules 320 can be configured to determine the packet that receives It is associated with stateless or stateful packet processing task.If stateless LB modules 320 determine the data point received Group is associated with stateless data processing task, then stateless LB modules 320 can be configured to directly draw packet SDN elements 311 are led, it is configured to perform the corresponding stateless network function in selected computing device 115.So And if stateless LB modules 320 determine the packet that receives, stateless associated with stateful data processing task Packet can be directed to the stateful LB modules 315 in selected computing device 115 by LB modules 320.In general, The first water of local load balance can be performed with high relative velocity (for example, compared with second level of local load balance) It is flat, because the first level of local load balance can be performed and because so not based on the IP link fields of packet Complicated parsing comprising packet.

Many packet-processing tasks in distributed SDN block cores system 200 be it is stateful and require tracking across One or more states of multiple packets.Will (or with mobile equipment 20 associated) associated with given communication session simultaneously And the processing for the packet being associated with corresponding stateful data processing task is assigned to multiple calculating in region pod 220 Equipment 115 will cause non-expandable local heterogeneous equilibrium mechanism.Especially, such appointment causes multiple computing devices 115 In each computing device 115 maintain packet state and the synchronization between multiple computing devices 115 will be included.One In a little embodiments, the particular computing device 115 in region pod 210 be designated with handle it is associated with given communication session (or Person is associated with mobile equipment 20) and the packet that is associated with corresponding stateful data processing task and be referred to as " host " for such packet.For example, for LFE modules 212b, example host can be to maintain use In the computing device 115 of the state of specific General Packet Radio Service (GPRS) tunnel protocol (GTP) session.In some implementations In mode, each computing device 115 in each region pod 210 includes being used to incoming data packet being directed to " correct " host The corresponding stateful LB modules 315 of machine.In some embodiments, the only subset bag of the computing device 115 in region pod 210 Include corresponding stateful LB modules 315.Stateful LB modules 315 in any region pod 210, which are formed, draws incoming data packet Lead corresponding host stateful LB layers.When receiving packet, the stateful LB of selected computing device 115 Module 315 can be configured to determine the computing device for handling packet according to stateful packet processing task 115。

Stateful LB modules 315 can be configured to parsing complex packet gauge outfit, and (such as type-length-value (TLV) is tied Structure).In some embodiments, parsing packet gauge outfit can include stateful 315 ciphertext data packet headers of LB modules Such as application-specific, data session or the further feature being associated with definite instruction with corresponding stateful data processing task Application identifier, Session ID or other gauge outfit field parameters.Once it is determined that stateful packet processing task, there is a shape State LB modules 315 can determine for the calculating of identified stateful packet processing task application to packet to be set Standby 115.In some embodiments, stateful LB modules 315 are able to maintain that and use for assigning computing device 115 to locate Manage one or more data structures (such as look-up table) of the packet received.In some embodiments, stateful LB Module 315 is able to maintain that the logic mark that Data Packets field value is mapped to instruction SDN elements 311 (or corresponding network function) Know first look-up table of (logic ID) and searched for SDN elements 311 to be mapped to the second of corresponding trustship computing device 115 Table.In some embodiments, stateful LB modules 315, which are able to maintain that, allows to indicate to be applied to the stateful of packet The Data Packets field value of network function is directly mapped to SDN element 311 of the trustship for performing stateful network function The single look-up table of computing device 115.In some embodiments, it is used for if multiple computing devices 115 are confirmed as trustship Perform to be applied to for by the SDN elements 311 of the network function of the packet for the session being initiated, then stateful LB modules 315 can multiple identified 115 middle selection operations of computing device (or health) and with minimum treat load Computing device 115.Once computing device is designated as the given communication meeting for including status data packet-processing task All packets of the session (or mobile equipment 20) are directed into during the host of words (either for mobile equipment 20) The host of appointment.

As illustrated examples, for the number encapsulated according to General Packet Radio Service (GPRS) tunnel protocol (GTP) For packet, stateful LB modules 315 can be configured to decapsulate packet to mark from GTP user planes (GTP-U) Head retrieval Tunnel End Point Identifier (TEID).Stateful LB modules 315 can be then by TEID and 115 (such as trustship of computing device The computing device 115 of the specific FE modules 212 of RAT for handling packet) it is associated.In some embodiments, there is shape The packet process associated with service computing device 115 can be that agreement is specific by state LB modules 315.

For example, in the case that the first computing device 115 is necessary to switch off service wherein, applied at each region pod 210 Local load balancing mechanism allow the responsibility (or network function) of the first computing device 115 being moved to the second computing device 115.For example, if the first computing device 115 shuts down (due to certain failure or in order to safeguard), stateless LB modules 320 It can stop data service being directed to the first computing device 115.Such data service can conversely be directed into one or A number of other computing devices 115, it is configured to perform the network function previously performed by the first computing device 115.Assign To such change in the responsibility (or task) of separated computing device 115 inside the pod of region and not to exterior equity 301 is transparent.

Each region pod 210 includes pod LB managers 330, it is configured to collection for various in the region pod 210 The health of computing device 115 and SDN elements 311 (or operation) and load information.Health (or operation) information includes instruction Each SDN elements 311 whether normal operating either whether computing device 115 operable or the information shut down.Load information bag Include that such as computer processing unit (CPU) utilizes, memory utilizes, network link utilizes, the queue associated with SDN elements State, the processing time being associated with various SDN elements, other factors or its information combined.Pod LB managers 330 can be by It is configured to make such information can be used for SDN elements 311, stateless LB modules 320, stateful LB modules 315 or its combination, uses In use in C planes signaling and load balance.For example, when new session is established, stateless LB modules 320 or stateful Request (or packet) for establishing such session can be directed to minimum treat load by LB modules 315 The computing device 115 or SDN elements 311 of healthy (either operable).If the first SDN elements 311 will request the 2nd SDN members The service of part 311, then the first SDN elements 311 can use by pod LB managers 330 provided health and load information determine The health and minimum load example of 2nd SDN elements 311.

Pod LB managers 330 can be configured to the accumulation health and load information of region pod 210 sending signals to the overall situation LB modules 340 are used to use when making pod trade-off decision.Instruction area can be included by sending signals to the information of global LB modules 340 Each meter in the mode of operation (such as operate or shut down) of domain pod, the relatively available processing capacity of region pod, region pod 210 Calculate the relatively available capacity of equipment 115 or the information of its combination.In some embodiments, pod LB managers 330 can by with It is set to and the detailed health and load information of region pod 210 is sent signals into global LB modules 340, it is every in such as region pod 210 The health and load information of a computing device 115 and each SDN elements 311.Global LB modules 340 can be configured to from various Region pod 210 gather health and load information and use processing load among gathered information balance region pod 210 or It is either inoperable or (or reduced with the availability reduced that person selects one or more regions pods 210 to carry out takeover load Active volume) the load of another region pod or the part of load.In some embodiments, for being positioned in by separating The urban area that services of region pod 210 between (or in its boundary) mobile equipment 20 for, global 340 energy of LB modules Enough it is configured to health and load information based on region pod 210, the guiding data associated with such mobile equipment 20 point Group.

Can be by remote for the signaling transport protocol mentioned above between the different DSN entities of region pod 210 The journey invocation of procedure (RPC) is realized.That is, do not require for signaling particular protocol layer (GTP, PMIPv6 etc.) or be used for In the case of any extra play (such as IPSec) of information authentication, region pod 210 can be signaled as the RPC of verification Different SDN entities between message.

Can be with the Fundamental Digital Circuit or computer software to be implemented on tangible medium, firmware or hardware realization sheet Theme described in this description and the embodiment of feature operation, including structure disclosed in this specification and its structure etc. Jljl or one or more combination.The embodiment of theme described in this specification can be implemented as by reality One or more computer programs on present tangible medium, that is, be coded for being performed by data processing equipment or being controlled One or more moulds of computer program instructions on one or more computer-readable storage mediums of the operation of data processing equipment Block.Computer-readable storage medium can be or be included in computer readable storage devices, computer-readable storage substrate, random Or the combination of serial access memory array or equipment or one or more of which.Computer-readable storage medium can also be or Person is included in one or more separated physical units or medium (for example, multiple CD, disk or other storage devices). Computer-readable storage medium can be tangible and non-transitory.

Operation described in this specification can be implemented as being stored in one or more computer-readable storages As the operation performed by data processing equipment in equipment or from the data that other sources receive.Process and logic flow can also pass through Dedicated logic circuit such as FPGA (field programmable gate array) or ASIC (application-specific integrated circuit) is performed, and device can also Enough dedicated logic circuit is implemented as example, FPGA (field programmable gate array) or ASIC (application-specific integrated circuit).

Although this specification includes many particular implementation details, it is not construed as to any invention or can To advocate the limitation of what scope, and it is specific for the description of the feature of the particular implementation of specific invention.Separated Described some features can also combine and be implemented in single embodiment in this specification in the context of embodiment In.On the contrary, the various features described in the context of single embodiment can also be adapted to discretely or with any Sub-portfolio be implemented in multiple embodiments.Moreover, although feature can be described above as acting on simultaneously in some combinations And be even so initially claimed, but in some cases, the one of combination claimed can be cut off from combination A or multiple features, and combination claimed can be related to the modification of sub-portfolio or sub-portfolio.

Can being interpreted as including property for quoting for "or" so that using the described any term of "or" can To indicate single, more than one and whole any one in described term.Label " first ", " second ", " the 3rd " etc. are no It must mean that instruction sorts and is typically only utilized for distinguishing between same or similar term or element.

Therefore, the particular implementation of this theme is had been described above.Other embodiment is in the scope of the following claims. In some cases, action described in claim can be performed in a different order and still realize desired knot Fruit.In addition, discribed process realizes desired result without requiring shown certain order or sequential order in attached drawing. In certain embodiments, multitask or parallel processing can be used.

Claims

1. a kind of distributed software defines network (SDN) block core system, including：

The local SDN infrastructure example of multiple interconnections, each local SDN infrastructure example are included geographically close to cloth Put and be communicably coupled to the cluster of the computing device of corresponding multiple radio access nodes, each local SDN infrastructure Example includes：

Multiple SDN network function modules, its expression can perform by one or more of computer equipment computer equipment Network function example；

Stateless load balancer, it is configured to receive the data point being used for by least one processing in the network function Group and the packet is directed to one in the computing device in the local SDN infrastructure examples；

Multiple stateful load balancing modules, each stateful load balancing module reside in corresponding computing device and by It is configured to：

Determine to appoint to be applied to the stateful packet processing of the packet received at corresponding computing device Business；

Determine for identified stateful packet to be handled task application to the local SDN bases of the packet Computing device in Infrastructure example；And

The packet is forwarded to identified computing device；And

Load manager, it is configured to management and indicates the computing device in the local SDN infrastructure examples and institute State the load of multiple SDN network function modules and the information of mode of operation.

2. distribution SDN block core systems according to claim 1, wherein, the load manager is configured to adopt Collection indicates that the computing resource in corresponding local SDN infrastructure examples uses, memory uses and network link uses In at least one load information.

3. distribution SDN block core systems according to claim 1, wherein, the load manager is configured to adopt Collection indicates the information of the mode of operation of the SDN network function module.

4. distribution SDN block core systems according to claim 1, wherein, the load manager be configured to by Indicate that the cumulative operation of corresponding local SDN infrastructure examples and the information of load condition are provided to the distribution The centralized load balancer based on cloud of SDN block core systems.

5. distribution SDN block core systems according to claim 1, wherein, the stateless load balancer by with One or more header fields based on packet are set to, the packet is assigned to corresponding computing device.

6. distribution SDN block core systems according to claim 1, wherein, each stateful load balancing module quilt It is configured to maintain to indicate at the stateful packet associated with each computing device of the local SDN infrastructure examples The information of reason task.

7. distribution SDN block core systems according to claim 6, wherein, each stateful load balancing module quilt It is configured to：

Parse one or more fields of the packet received；

The described information maintained based on one or more of fields and at the stateful load balancing module, determines to use In the computing device for handling the stateful packet processing task associated with the packet received；And

The packet is forwarded to identified computing device.

8. distribution SDN block core systems according to claim 1, further include global load balancer based on cloud, It is configured to：

Network processes request is received from access network element；And

Load and mode of operation based on one or more locals SDN infrastructure examples, the network processes are asked to assign To local SDN infrastructure example.

9. distribution SDN block core systems according to claim 8, wherein, the global load-balancing based on cloud Device is configured to be based on delay metric, and network processes request is assigned to local SDN infrastructure example.

10. distribution SDN block core systems according to claim 8, wherein, the global load-balancing based on cloud Device is configured to measure based on capacity, and network processes request is assigned to local SDN infrastructure example.

11. a kind of method that distributed software defines the load balance in network (SDN) block core system, including：

Received by the stateless load balancer in the local SDN infrastructure example of the distribution SDN block core systems For the packet handled by least one SDN network function module in multiple SDN network function modules, the multiple net Network function module can perform on multiple computing devices of local SDN infrastructure example；

The packet is directed to by the stateless load balancer and is resided in the local SDN infrastructure examples The multiple computing device among the first computing device on stateful load balancer；

Determined to handle task to be applied to the stateful packet of the packet by the stateful load balancer；

Determine to be used to identified stateful packet handling task application described in by the stateful load balancer The second computing device in the local SDN infrastructure examples of packet；And

The packet is forwarded to second computing device by the stateful load balancer.

12. according to the method for claim 11, further include and adopted by the load manager of the local SDN infrastructure examples Collection indicates that the computing resource in corresponding local SDN infrastructure examples uses, memory uses and network link uses In at least one load information.

13. according to the method for claim 11, further include and adopted by the load manager of the local SDN infrastructure examples Collection indicates the information of the mode of operation of the SDN network function module.

Will by the load manager of the local SDN infrastructure examples 14. according to the method for claim 11, further including Indicate that the cumulative operation of corresponding local SDN infrastructure examples and the information of load condition are provided to distributed SDN points The centralized load balancer based on cloud of group core system.

15. according to the method for claim 11, wherein, the packet is directed to stateful load balancer includes One or more header fields based on the packet, guide the packet.

Instruction and described are maintained by the stateful load balancer 16. according to the method for claim 11, further including The information for the stateful packet processing task that each computing device of ground SDN infrastructure examples is associated.

17. according to the method for claim 16, further include：

One or more fields of the packet received by the stateful load balancer parsing；

The described information maintained based on one or more of fields and at the stateful load balancer, determines to be used to locate Manage second computing device of the stateful packet processing task associated with the packet received；And

The packet is forwarded to second computing device.

18. according to the method for claim 11, further include：

By the centralized load balancer based on cloud of distributed SDN block cores system network is received from access network element Reason request；And

By load of the centralized load balancer based on cloud based on one or more locals SDN infrastructure examples and behaviour Make state, network processes request is assigned to local SDN infrastructure example.

19. according to the method for claim 18, further include and delay degree is based on by the global load balancer based on cloud Amount, local SDN infrastructure example is assigned to by network processes request.

20. according to the method for claim 18, further include and voluminosity is based on by the global load balancer based on cloud Amount, local SDN infrastructure example is assigned to by network processes request.