CN117280673A - Wireless access network intelligent controller (RIC) Software Development Kit (SDK) - Google Patents

Abstract

To provide a low latency near RT RIC, some embodiments separate the functionality of the RIC into several different components that operate (e.g., execute on a VM or cluster) on different machines operating on the same host computer or different host computers. Some embodiments also provide a high-speed interface between these machines. Some or all of these interfaces operate in a non-blocking, lock-free manner to ensure that critical near RT RIC operations (e.g., data path procedures) are not delayed by one or more component stalls caused by multiple requests. Furthermore, each of these RIC components also has an internal architecture designed to operate in a non-blocking manner such that none of the components' processes can block the operation of the other process of the component. All of these low latency features allow the near RT RIC to be used as a high speed IO between the E2 node and the xApp.

Description

Wireless access network intelligent controller (RIC) Software Development Kit (SDK)

Background

In a telecommunications network, a Radio Access Network (RAN) performs more and more functions with each iteration of the telecommunications standard. That is, in order to support the advantages of 5G over previous standards, the 5G RAN performs various additional functions.

These RAN functions are located between the user equipment and the core network and are therefore often performed at base stations (e.g., cell towers) where computational power may be limited.

Disclosure of Invention

Some embodiments provide novel RAN Intelligent Controllers (RIC) for telecommunications networks. For example, to provide a low latency near RT RIC, some embodiments separate the RIC functionality into several different components that operate on different machines operating on the same host computer or different host computers (e.g., executing on VMs or clusters (Pod)). Some embodiments also provide a high-speed interface between these machines. Some or all of these interfaces operate in a non-blocking, lock-free manner to ensure that critical near RT RIC operations (e.g., data path processes) are not delayed by one or more component stalls caused by multiple requests. Furthermore, each of these RIC components also has an internal architecture designed to operate in a non-blocking manner, such that none of the processes of the component blocks the operation of the other process of the component. All of these low latency features allow the near RT RIC to be used as a high speed IO between the base station node (i.e., E2 node) and the control plane application (e.g., xApp).

In some embodiments, the near RT RIC includes a data path cluster, a service cluster, and an SDL (shared data layer) cluster. A portion of the low latency architecture of RIC may be attributed to the use of different clusters to implement data IO, service, and SDL operations, such that each cluster may be provided with different resource allocation and management operations based on the respective requirements of the operations performed by each of these clusters. Moreover, in some embodiments, the RIC provides low latency messaging between its various clusters.

In some embodiments, the service clusters execute applications (e.g., xApp) load, registration, FCAPS (failure, configuration, billing, performance, security) and other services. It also provides services (such as metric collection, policy provisioning, and configuration) to other RIC components. The SDL cluster implements a shared data layer of the near RT RIC. In some embodiments, the SDL cluster also executes one or more service containers to perform one or more pre-processing or post-processing services on the data stored in the SDL.

The data path cluster performs data message forwarding between the base station components of the telecommunication network and the control and edge applications of this network. In some embodiments, some or all of the datapath services of the datapath cluster are embedded in the datapath threads and control threads of the datapath cluster. In other embodiments, the data path services are embedded in a data IO thread, a plurality of Data Processing Threads (DPTs), and a control thread.

In some embodiments, the control thread is an interface with the SDL cluster and the service cluster of the datapath thread, while in other embodiments it is an interface with only the service cluster of the datapath thread (since the datapath thread can communicate directly with the SDL cluster). The control threads in any of these methods perform the slower, control-related operations of the datapath, while one or more datapath threads perform the faster IO operations of the datapath. In some embodiments, the control thread interfaces with the service cluster to receive configuration data for configuring its own operations as well as the operations of the datapath threads.

Embodiments that separate the datapath threads into a datapath thread and a plurality of data processing threads further optimize the datapath by pushing the more computationally intensive operations of the datapath thread into the plurality of datapath processing threads, which then allows the less computationally intensive operations to run in the datapath thread. Both optimizations are intended to ensure a fast data path IO (without experiencing unnecessary latency) so that the near RT RIC can be used as a high speed interface between the base station node and the control and edge applications.

The foregoing summary is intended to serve as a brief introduction to some embodiments of the invention. This is not meant to be an introduction or overview of all-inventive subject matter disclosed in this document. The following detailed description and the accompanying drawings referred to in the detailed description will further describe the embodiments described in the summary of the invention and other embodiments. Accordingly, a full appreciation of the summary, the detailed description, and the accompanying drawings is required in order to understand all embodiments described in this document. Furthermore, the claimed subject matter is not limited to the details of the description of the invention, the detailed description and the accompanying drawings, but rather is defined by the appended claims, as the claimed subject matter may be embodied in other specific forms without departing from the spirit of the subject matter.

Drawings

The novel features of the invention are set forth in the appended claims. However, for purposes of explanation, several embodiments of the invention are set forth in the following figures.

Fig. 1 illustrates an example of an O-RAN architecture in accordance with some embodiments.

FIG. 2 illustrates an in-depth view of components of a non-real-time RIC and a near-real-time RIC, according to some embodiments.

Fig. 3 illustrates a deeper view of a MAC control facilitator of some embodiments.

Fig. 4 illustrates a deeper view of a user-level tracker of some embodiments.

Fig. 5 illustrates another view of the O-RAN architecture of some embodiments, with a deeper view of the near real-time RIC.

Fig. 6 illustrates deployment of RIC SDKs on machines executing control plane applications in some embodiments.

FIG. 7 illustrates that some embodiments deploy several RICs to execute on several host computers to implement a distributed near RT RIC that includes the RIC components shown in FIGS. 5 and 6.

FIG. 8 illustrates RICs executing on one host computer and two machines on which two control plane applications execute.

Fig. 9 illustrates two RIC executing on two host computers and two machines on which two control plane applications and two RIC SDKs execute.

FIG. 10 illustrates RICs executing on a first host computer to connect two control plane applications executing on two machines operating on two other host computers.

FIG. 11 illustrates RICs executing on a first host computer to connect two control plane applications executing on two machines, one machine operating on the first host computer and the other machine operating on the other host computer.

FIG. 12 illustrates an example of different standard-specified APIs supported by a distributed near RT RIC platform of some embodiments.

Fig. 13 illustrates an embodiment in which the SDL cache is part of each RIC SDK executing on the same machine as its control plane application.

FIG. 14 illustrates an example of control or edge applications having pass-through access to the hardware accelerator of their host computers to perform some or all of their computations.

Fig. 15 illustrates a process performed in some embodiments in response to an O-RAN component instructing a CP or edge application to perform operations requiring the application to use the hardware accelerator of its host computer.

Fig. 16 illustrates an application that performs operations based on data from an E2 node.

FIG. 17 illustrates another example of a control or edge application having pass-through access to the hardware accelerator of its host computer to perform some (or all) of its computations.

Fig. 18 illustrates yet another example of CP or edge applications having pass-through access to the hardware accelerator of their host computer to perform some or all of their computations.

Fig. 19 illustrates a process by which some embodiments deploy O-RAN applications with direct, pass-through access to the hardware accelerators of their host computers.

FIG. 20 illustrates an example of a CP or edge application having pass-through access to a virtual hardware accelerator defined by a hypervisor executing on its host computer.

FIG. 21 illustrates an example of a near RT RIC with several components operating on several different machines.

FIGS. 22 and 23 illustrate different examples of components for deploying the near RT RIC of FIG. 21.

Fig. 24 and 25 illustrate other examples of near RT RIC.

FIG. 26 illustrates an example of a RIC data path cluster.

FIG. 27 illustrates a process that the datapath thread executes to process a subscription request from xApp in some embodiments.

FIG. 28 illustrates a process that the data IO thread and DPT execute to process data messages that one or more xApps from the E2 node should receive in some embodiments.

FIG. 29 illustrates a process performed by a data IO thread and DPT to process a data message from xApp that should be sent to an E2 node in some embodiments.

FIG. 30 illustrates an example of a process used by a data IO thread to assign an E2 node to a DPT in some embodiments.

Fig. 31 illustrates a distributed near RT RIC implemented by an active RIC and a standby RIC.

FIG. 32 illustrates interfaces between a near RT RIC and an E2 node and between a near RT RIC and an xApp cluster in some embodiments.

Fig. 33 illustrates E2AP message handling of a near RT RIC data path cluster.

Fig. 34 illustrates an example RIC with an SDL cluster of some embodiments.

Fig. 35 conceptually illustrates an electronic system with which some embodiments of the invention are implemented.

Detailed Description

In the following detailed description of the present invention, numerous details, examples, and embodiments of the present invention are set forth and described. It will be apparent, however, to one skilled in the art that the invention is not limited to the illustrated embodiments and that the invention may be practiced without some of the specific details and examples that are discussed.

Today, there is a push to implement a Radio Access Network (RAN) of a telecommunication network, e.g. a cellular network, as an O-RAN, i.e. a standard for allowing interoperability of RAN elements and interfaces. Fig. 1 illustrates an example of an O-RAN architecture 100 in accordance with some embodiments. The O-RAN architecture 100 includes a service management and orchestration framework (SMO) 110 with a non-real time RIC 105, a near real time RAN Intelligent Controller (RIC) 115, an open control plane central unit (O-CU-CP) 120, an open user plane central unit (O-CU-UP) 125, an open distributed unit (O-DU) 130, an open radio unit (O-RU) 135, and an O-cloud 140.O-CU-CP 120, O-CU-UP 125, and O-DU 130 may be collectively referred to hereinafter as managed functions 120-130.

As defined in the standard, in some embodiments SMO 110 includes an integrated architecture that allows SMO to connect to and manage RIC 115, managed functions 120-130, and O-cloud 140 via open interface 150. Unlike these elements, in some embodiments, O-RU 135 is not managed by SMO 110, but rather by O-DU 130, as indicated by dashed line 160. In some embodiments, O-RU 135 processes and transmits the radio frequency to O-DU 130.

In some embodiments, managed functions 120-130 are logical nodes that each host a set of protocols. For example, in accordance with the O-RAN standard, in some embodiments, O-CU-CP 120 includes protocols such as control plane portions of Radio Resource Control (RRC) and Packet Data Convergence Protocol (PDCP), while O-CU-UP 125 includes protocols such as Service Data Adaptation Protocol (SDAP) and user plane portions of Packet Data Convergence Protocol (PDCP).

Both RIC are each adapted to specific control loop and latency requirements. Near real-time RIC 115 provides programming control of open centralized units (O-CUs) and open distributed units (O-DUs) at time periods of 10 milliseconds to 1 second. On the other hand, non-real-time RIC (non-RT RIC) 105 provides higher layer policies that may be implemented in the RAN via a near RT RIC or via a direct connection to the RAN node. The non-RT RIC is used for control loops exceeding 1 second. Each RIC 105 or 115 serves as a platform on which RAN control applications execute. These applications may be developed by third party vendors other than the RIC vendor. These applications are referred to as "xApps" (for near RT RIC 115) and "rApps" (for non-RT RIC).

In some embodiments, near real-time RIC 115 is a logical aggregation of several functions using data collection and communication through interface 155 in order to control managed functions 120-130. In some embodiments, the non-real-time RIC 105 uses machine learning and model training to manage and optimize the managed functions 120-130. In some of these embodiments, near RT RIC also uses machine learning.

In some embodiments, O-cloud 140 is responsible for creating and hosting virtual network functions (VNs) for use by near RT RICs 115 and managed functions 120-130. In some embodiments, the DUs are responsible for per-slot decisions for user scheduling and include a RAN scheduler that performs MAC control assistance and user-level tracking. To increase the computational power available in the cloud (i.e., compared to base stations that typically perform RAN functions), RIC is implemented in one or more public and/or private cloud data centers and an improved cloudized RAN scheduler is implemented in the cloud, offloading these MAC control assistance and user-level tracking functions from DUs to RIC. In some embodiments, interface 155 enables the RAN to provide inputs to functions at the RIC, and at least in some embodiments, receive outputs that have been calculated by those functions at the RIC.

FIG. 2 illustrates an in-depth view of the components of both the non-real-time RIC 201 and the near-real-time RIC 202. Each of RIC 201 and 202 includes a respective set of analysis functions 210 and 212 and a respective set of optimization functions 214 and 216, each shown in dashed lines to indicate that they are existing components. In addition to these existing components, the near real-time optimization function 216 also includes two new components, namely a MAC control facilitator 220 and a user-level tracker 222, shown by solid lines to visually distinguish them from the existing components. In some embodiments, these components are part of a larger MIMO component (e.g., along with MU-MIMO UE pairs and precoders).

In some embodiments, the MAC control facilitator 220 may include various functions such as (1) User Equipment (UE) -specific beamforming weight calculation based on UL SRS channel signal reception, (2) UE Radio Frequency (RF) condition prediction, and (3) multi-user multiple input multiple output (MU-MIMO) pairing recommendation based on the MAC scheduler of the UE-specific beam. For each of these functions, some embodiments expose a reporting interface (which provides input data for the function from the DU to the RIC) and a control interface (which provides output data for the function from the RIC to the DU).

In some embodiments, the user level tracker 222 generates L1/L2/L3 level information related to user configuration and traffic performance. This trace data may be used as input to various control algorithms including MAC schedulers, parameter settings, etc. The user-level tracker 222 may include tracking operations that may (i) track user behavior in a cell, (ii) track user RF conditions, (iii) track user data traffic performance in different layers (MAC, radio Link Control (RLC), packet Data Convergence Protocol (PDCP)), and (iv) track user RF resource consumption.

Fig. 3 illustrates a deeper view of a MAC control facilitator 300 of some embodiments. As shown, the MAC control facilitator 300 includes a UE-specific beamforming weight calculator (BFWC) 310, a UE RF condition predictor 320, and a MU-MIMO pairing recommender 330. In some embodiments, the UE-specific BFWC 310 is based on UL SRS channel signal reception. In some embodiments, the MU-MIMO pairing recommender 330 is used for a MAC scheduler based on the UE-specific beam.

As shown, each of the components 310-330 of the MAC control facilitator 300 includes an uplink and a downlink. For the UE-specific BWC function, some embodiments expose a reporting interface for an uplink sounding reference signal (UL SRS) channel response matrix as an input to the weight calculation function and a control interface for a UE-specific beamforming weight matrix. For the UE RF condition predictor function, some embodiments expose a reporting interface for Downlink (DL) channel condition reporting as input to RF condition prediction and a control interface for predicted DL channel conditions (e.g., including DL SINR, PMI, and rank) for the next scheduling window. For the MU-MIMO pairing proposal function, some embodiments expose a reporting interface for a UE-specific beamforming weight matrix as input to the pairing proposal function and a control interface for UE pairing proposal and SINR impact assessment.

Fig. 4 illustrates a deeper view of a user-level tracker 400 of some embodiments. In some embodiments, the tracker 400 includes a plurality of uplinks 410 and a plurality of downlinks 415 for performing tracking operations. These operations produce L1/L2/L3 level information related to user configuration and traffic performance. This trace data may be used as input to various control algorithms including MAC schedulers, parameter settings, etc. These tracking operations may (1) track user behavior in the cell, (2) track user RF conditions, (3) track user data traffic performance in different layers (MAC, RLC, PDCP), and (4) track user RF resource consumption.

For these trace operations, some embodiments expose reporting interfaces for DUs and/or CUs to provide various metrics to user-level trace operations. These metrics may include selected RRC messages, MAC/RLC/PDCP traffic volume and performance, RF conditions, and RF resource consumption. In some embodiments, messages to the RIC over these interfaces are triggered based on user behavior and/or periodic reporting (e.g., for traffic performance and RF conditions/resource consumption).

The tracking operation tracks the various user data indicated above, and this information may be provided back to the RAN or other control algorithm (e.g., other algorithm operating at the RIC). For example, these algorithms may perform analysis of user data performance from user level trace operations, determine certain performance shortfalls, and modify the manner in which the RAN handles user traffic. In some embodiments, examples of control algorithms that may benefit from user-level tracking include (1) traffic steering, (2) quality of service (QoS) scheduling optimization, (3) user configuration adjustment, and (4) user behavior anomaly detection.

For all of the operations described in fig. 3-4 (i.e., MAC scheduler functions and user-level trace operations), the increased computing power available to RIC in the cloud enables more complex computations without excessive latency. For example, some or all of these operations may be performed at the RIC using machine learning (e.g., using a network of machine training, etc.).

Fig. 5 illustrates another view of the O-RAN architecture of some embodiments, with a deeper view of the near real-time RIC. Architecture 500 includes SMO 505 with non-real-time RIC 510, distributed near-real-time RIC 515, and E2 nodes 520 (e.g., O-DU and/or O-CU nodes). Distributed near real-time RIC 515 includes messaging infrastructure 540, a set of services (e.g., 550, 552, 554, and 556), shared data layer 560, database 570, and a set of termination interfaces (e.g., 580, 582, and 584). As shown, a set of embedded applications (e.g., 530, 532, and 534) use this distributed near RT RIC 515. As described further below, in some embodiments, distributed near RT RIC 515 is implemented by multiple RIC executing on multiple host computers.

As shown, the set of services includes a conflict mitigation service 550, an application subscription management service 552, a management service 554, and a security service 556. In addition, the set of termination interfaces includes an O1 termination interface 580 that connects SMO 505 to near real-time RIC 515, an A1 termination interface 582 that connects non-real-time RIC 510 to near real-time RIC 515, and an E2 termination interface 584 that connects E2 node 520 to near real-time RIC 515. In some embodiments, each application represents various functions of near RT RIC 515 using data sent from E2 node 520. For example, application 530 may correspond to UE-specific BFWC 310 of MAC control facilitator 300, application 532 may correspond to UE RF condition predictor 320 of MAC control facilitator 300, and so on.

In some embodiments, the goal of framework 500 is to offload computationally intensive near real-time functions and provide the results back to the O-DU (e.g., via an E2 interface with E2 node 520). In some embodiments, the results may be used to assist or enhance real-time decisions in the MAC layer. Three example use cases of the MAC control framework will be described below, each of which is specific to a different component of the MAC control facilitator (e.g., a UE-specific BFWC, a UE RF condition predictor, and a MU-MIMO pairing recommender), and one example of use of the user-level tracker.

The first example use case is specific to UE-specific beamforming weight calculation based on the UL SRS signal receiving component of the MAC control assistance framework (e.g., component 310 of the MAC control assistance 300). In some embodiments of this use case, the input metrics may include a number of options based on UL SRS, such as raw SRS received data and SRS channel response matrix from channel estimation.

In some embodiments, an algorithm for generating the output metrics evaluates optimal beamforming weights to the user. Some embodiments use conventional signal processing algorithms based on channel models. Alternatively or in combination, machine learning based algorithms are used that utilize raw data input, which require feedback from DUs in the E2 node 520.

In some embodiments, the output metrics generated by the algorithm include a beamforming weight (BFW) matrix for the user. In some embodiments, BFW may also be mapped to a beam index in a pre-designed beam set. In some embodiments, the DU uses the matrix to control MIMO antenna array gain/phasing in RUs (e.g., O-RU 135 in architecture 100) for user data transmission and reception.

The second example is specific to a UE RF condition predictor component of the MAC control assistance framework (e.g., component 320 of the MAC control assistance 300). For this second use case, according to some embodiments, the input metrics include at least the channel reports from the UE, such as wideband or subband CQI/PMI/RI for DL, or SRS for UL. The input metrics of some embodiments may also optionally include support information such as UE distance, UE location, etc.

In some embodiments, the application algorithm for this second use case aims to predict the RF condition of the UE based on observations. Some embodiments utilize conventional signal processing algorithms based on channel and mobility models. Alternatively or in combination, some embodiments also use machine learning based algorithms that use data input and potentially other factors, such as site layout (which requires feedback from DUs).

In some embodiments, the output metrics of this use case include the predicted channel conditions of the user for the next scheduling window, as well as the predicted downlink and uplink SINRs, precoding matrix (e.g., if applicable), and SU-MIMO layer. In some embodiments, these output metrics are used by DUs for user link adaptation on PDCCH/PDSCH/PUSCH transmissions.

The third example is specific to a MU-MIMO pairing recommender for a MAC scheduler component (e.g., component 330 of the MAC control facilitator 300). In some embodiments, the input metrics for this example use case include at least a UE-specific BFW matrix and a UE RF condition estimate. In addition to the UE-specific BFW matrix and UE RF condition estimation, some embodiments may include support metrics such as user data requirements as input metrics.

In some embodiments, the application algorithm for this use case is intended to identify users that may be paired for MU-MIMO operation. For example, some embodiments of the third example use conventional signal processing algorithms based on information theory and cross-channel covariance estimation. Alternatively or in combination, some embodiments use a machine learning based algorithm that uses data input, which also requires feedback from the DU.

In some embodiments, the output metrics of this third example may include UE pairing recommendations and impact evaluations on SINR and SU-MIMO layers. Further, in some embodiments, the DU uses the output metrics to select users for RF scheduling and determines transmission efficiency.

An example use case of a user level tracker may include QoS scheduling optimization, which aims to adjust the user scheduling priority of RF resources to optimize quality of service. The input for some embodiments of this use case may include a quality of service target from the user subscription. In some embodiments, user level tracking includes (1) tracking user RF conditions, (2) tracking user data traffic performance in different layers (e.g., MAC/RLC/PDCP), and (3) tracking user RF resource consumption.

In some embodiments, application algorithms are based on QoS objectives and observed user traffic performance and may be used to determine resource under-allocation for users. In some embodiments, the algorithm format may be logic-based or machine-learning-based. In some embodiments, the output may include a recommendation to the MAC scheduler to adjust traffic priority or link adaptation in order to improve performance.

On each machine (e.g., each VM or cluster) executing a control plane application, some embodiments configure the RIC SDK to serve as an interface between the control plane application on the machine and a set of one or more elements of the RAN. In some embodiments, the RIC SDK provides a set of connectivity APIs (e.g., a framework) through which an application may communicate with distributed near real-time (RT) RIC implemented by two or more near real-time RIC. In some embodiments, examples of such applications include xApp, as well as other control plane and edge applications. In the O-RAN, the xApp performs control plane, monitoring and data processing operations. The following discussion with respect to fig. 6 and 8-20 relates to control plane applications (e.g., 615, 815, 820, 915, 920, etc.). In some embodiments, these control plane applications are xapps in an O-RAN system.

Fig. 6 illustrates deployment of RIC SDK 605 on a machine 610 executing a control plane application 615 in some embodiments. As shown, one or more machines 610 execute on each of several host computers 607 in one or more data centers. In some embodiments, RIC SDK 605 on each machine 610 includes a set of network connectivity procedures that establish network connections to a set of RAN elements (e.g., E2 nodes 520, shared data layer 560, management services 554, SMO 505, etc.) for control plane applications. The RIC SDK procedure allows the control plane application on its computer to forgo performing network connectivity operations. In some embodiments, a set of network connectivity procedures for each RIC SDK for each machine establishes and maintains network connections between the machine and a set of RAN elements used by a control plane application of the machine, and handles data packet transmissions to and from the set of RAN elements by the control plane application.

The control plane application on each machine communicates with a set of RAN elements through the high level APIs 620, and the RIC SDK converts the high level APIs 620 to low level APIs 625. In some embodiments, at least a subset of the low-level API calls 625 are specified by a standard specification body. Moreover, in some embodiments, the high-level API 620 is made in a high-level programming language (e.g., c++), while the low-level API calls include low-level calls that establish and maintain network connections and pass data packets through these connections.

In some embodiments, a set of RAN elements to which the RIC SDK is connected to control plane applications on its machines includes RAN elements produced and/or developed by different RAN providers and/or developers. In some embodiments, these RAN elements include CUs 630 and DUs 635 of the RAN. Moreover, this SDK communicates with CUs and DUs through low-level, standard-specified E2 interfaces, while control plane applications on the machine communicate with CUs and DUs through RIC SDKs using high-level API calls. In some embodiments, the high-level API call specifies E2 interface operations at the high-level application layer that do not include low-level transport or network operations.

In conjunction or alternatively, the set of RAN elements to which the RIC SDK connects with the control plane application 615 on its machine 610 includes the network elements of the RIC. Again, in some embodiments, these network elements include RAN elements produced and/or developed by different RAN providers and/or developers. In some embodiments, these RIC elements include a Shared Data Layer (SDL) 560, data path input/output (I/O) elements, and in some embodiments, application and management services 552 and 554. FIG. 7 illustrates that some embodiments deploy several near RT RICs 705 to execute on several host computers 710 to implement a distributed near RT RIC 700 that includes the RIC components shown in FIGS. 5 and 6. In some embodiments, one RIC 705 executes on each host computer that also executes the control plane application 615. In other embodiments, the control plane application 615 may execute on a host computer that does not execute RIC. For example, in some embodiments, one or more control plane applications execute on one or more host computers having Graphics Processing Units (GPUs), while RIC are not executing on such host computers because they do not require the processing power of the GPUs.

Through distributed near RT RIC, the RIC SDK also connects its control plane applications to other control plane applications executing on other machines. In other words, in some embodiments, the RIC SDK and the distributed near RT RIC serve as a communication interface between control plane applications. In some embodiments, different control plane applications are developed by different application developers that communicate with each other through a distributed near RT RIC using a common set of RIC APIs. In some of these embodiments, the distributed near RT RIC adds one or more parameters to an API call as it forwards the API call from one control application to another.

Figures 8-11 illustrate several examples of RIC architectures in which the RIC SDK and distributed near RT RIC establish a communication interface between control plane applications. These architectures are mutually exclusive in some embodiments, while in other embodiments two or more of these architectures are used in combination. Fig. 8 illustrates a RIC 800 executing on one host computer 805 and two machines 810 and 812, with two control plane applications 815 and 820 executing on the two machines 810 and 812. With RIC SDKs 802 and 804 executing on machines 810 and 812, RIC 800 receives API calls from CP application 815 and forwards the API calls to CP application 820, and passes responses to those API calls from second CP application 820 to first CP application 815. It also passes API calls from the second CP application 820 to the first CP application 815 and passes responses from the first CP application 815 to the second CP application 820.

Fig. 9 illustrates two RIC 900 and 901 and two machines 910 and 912 executing on two host computers 905 and 907, two control plane applications 915 and 920 and two RIC SDKs 902 and 904 executing on the two machines 910 and 912. As shown, API calls from the first CP application 915 to the second CP application 920 are forwarded through the first RIC SDK 902, the first RIC 900, the second RIC 901, and the second RIC SDK 904. Responses from the second CP application to the first CP application 915 to these API calls are routed in reverse paths from the second RIC SDK 904, the second RIC 901, the first RIC 900, and the first RIC SDK 902.

The API calls from the second CP application 920 to the first CP application 915 are forwarded through the second RIC SDK 904, the second RIC 901, the first RIC 900 and the first RIC SDK 902, while the responses from the first CP application 915 to the second CP application 920 to these API calls are forwarded through the first RIC SDK 902, the first RIC 900, the second RIC 901 and the second RIC SDK 904.

FIG. 10 illustrates a RIC 1000 executing on a first host computer 1005 to connect two control plane applications 1015 and 1020 executing on two machines 1010 and 1012 operating on two other host computers 1006 and 1007. With the RIC SDKs 1002 and 1004 executing on the machines 1010 and 1012, the RIC 1000 receives API calls from the CP applications 1015 and forwards the API calls to the CP applications 1020, and passes responses to the API calls from the second CP applications 1020 to the first CP applications 1015. It also passes API calls from the second CP application 1020 to the first CP application 1015 and passes responses from the first CP application 1015 to the second CP application 1020.

FIG. 11 illustrates a RIC 1100 executing on a first host computer 1105 to connect to two control plane applications 1115 and 1120 executing on two machines 1110 and 1112, one of which operates on the host computer 1105 and the other of which operates on the host computer 1106. The RIC 1100 receives API calls from the CP applications 1115 through RIC SDKs 1102 and 1104 executing on machines 1110 and 1112, forwards API calls to the CP applications 1120, and passes responses to those API calls from the second CP application 1120 to the first CP application 1115. Through these SDKs 1102 and 1104, the ric 1100 also passes API calls from the second CP application 1120 to the first CP application 1115 and passes responses from the first CP application 1115 to the second CP application 1120.

FIG. 12 illustrates an example of a different standard specified API supported by a distributed near RT RIC platform of some embodiments. As shown, in some embodiments, distributed near RT RIC platform 1200 uses E2, O1, and A1 interfaces specified by an O-RAN standard specification body. It communicates with E2O-RAN nodes (such as O-CU-CP 1202, O-CU-UP 1204, and O-DU 1206) using E2 APIs. It also communicates with the non-real time RIC platform 1208 using the A1 API and with SMO 1210 using the O1 API.

For each of these E2, A1, and O1 APIs, the RIC SDK 1215 provides a high-level peer-to-peer API for the control plane application 1220, which uses the RIC SDK and distributed near RT RIC platform for the control plane application 1220 to communicate with the E2 communication nodes 1202-1206, the non-real-time RIC platform 1208, and the SMO 1210. FIG. 12 designates these high level peer APIs for the E2, O1, and A1 interfaces with an apostrophe as E2' API calls, O1' API calls, and A1' API calls. These high-level peer APIs are not specified by the standard body, but rather RIC SDKs and/or distributed near RT RIC translate to APIs of standard-specified API calls.

FIG. 12 also shows several RIC internal APIs for allowing control plane applications 1220 to communicate with each other, and with one or more elements of a distributed near RT RIC (e.g., shared Data Layer (SDL) 560, data path input/output (I/O) elements, and application and management services 552 and 554) through the RIC SDK and the distributed near RT RIC.

An enablement API is an API that is used in some embodiments to allow control plane applications 1220 to communicate with each other. As described above with reference to fig. 8-11, in some embodiments, these APIs are delivered through a distributed near RT RIC. In other embodiments, these APIs allow the RIC SDKs of the control plane applications to communicate directly with each other without going through any other component of the distributed near RT RIC. To this end, fig. 12 includes a dashed double-headed arrow between the RIC SDKs 1215 of two control plane applications 1220 to indicate that in some embodiments the RIC SDKs 1215 of these applications communicate directly with each other.

In some embodiments, the enablement APIs include a registration API, a service discovery API, and an inter-application communication API. The registration APIs are used by applications 1220 (e.g., xApps) to introduce themselves to other applications 1220 by providing their network identifiers (e.g., their network addresses and available L4 ports) and providing their functionality (e.g., performing channel prediction). The service discovery API allows control plane applications 1220 (e.g., xapps) to query a service directory (e.g., a distributed near RT RIC service directory) for other control plane applications (e.g., other xapps) that provide a particular service. Inter-application communication APIs allow control plane applications to communicate with each other to pass data and/or request certain operations.

Some embodiments deploy an SDL cache on the same host computer as the control plane application and use this cache to process at least a subset of the SDL storage access requests of the control plane application. In some embodiments, the control plane application and the SDL cache operate on machines executing on a host computer. In other embodiments, the SDL caches operate outside of the machine on the same host computer but on which the control plane application is executing. In some of these embodiments, multiple control plane applications executing on the same host computer use a common SDL cache on the host computer.

In some embodiments, the SDL cache is part of a RIC executing on the same host computer as the control plane application. In other embodiments, the SDL cache is part of a RIC SDK executing on the same machine as the control plane application. In any of these embodiments, the synchronization process of the RIC or RIC SDK synchronizes data stored in the SDL cache with data stored in the SDL storage.

In some embodiments, the SDL storage operates on a different host computer than the host computer on which the control plane application is executing, while in other embodiments at least a portion of the SDL storage operates on the same host computer as the host computer on which the control plane application is executing. Also, in some embodiments, when the RIC SDK is unable to process the SDL access request through the SDL cache, the RIC or RIC SDK forwards the SDL access request from the control plane application to the SDL storage. For example, when the SDL cache does not store data requested by the control plane application, the RIC or RIC SDK cannot process the SDL access request through the SDL cache.

Fig. 13 illustrates an embodiment in which the SDL cache 1302 is part of each RIC SDK 1300 executing on the same machine 1305 as its control plane application 1310. As shown, the RIC SDK 1300 includes a query manager 132 and a synchronization service 1327, the query manager 132 processing SDL requests from the CP applications 1310, the synchronization service 1327 synchronizing data stored in the SDL cache with data stored in the SDL storage 1350 of the SDL 1355 of the distributed near RT RIC 1330. In this example, the SDL storage 1350 operates on a different host computer than the host computer on which the control plane application 1310 is executing. However, in other embodiments, at least a portion of the SDL storage 1350 operates on the same host computer on which the control plane application 1310 is executing.

When the control plane application 1310 uses a high-level API call to read or write data to the SDL storage device, the query manager 1325 of the RIC SDK 1300 first determines whether the data record being read or written is stored in the SDL cache 1302. If so, the query governor 1325 reads from or writes to this record. When this operation is a write operation, the synchronization service 1327 writes new data to the SDL storage 1350 in real-time or in bulk. On the other hand, when the query manager 1325 of the RIC SDK 1300 determines that the data record being read or written is not stored in the SDL cache 1302, it passes the API call to the SDL layer of the distributed near RT RIC to perform the requested read or write operation. In some embodiments, when passing this API call, the RIC SDK 1300 modifies the format of the call and/or modifies the parameters provided with the call.

Some embodiments provide various methods for offloading operations in an O-RAN (open radio access network) onto a Control Plane (CP) or edge application executing on a host computer having a hardware accelerator in a Software Defined Data Center (SDDC). For example, a CP or edge application operating on a machine executing on a host computer with a hardware accelerator, the method of some embodiments receives data from an O-RAN E2 unit on which it must perform an operation. The method uses a driver of the machine to communicate directly with the hardware accelerator to instruct the hardware accelerator to perform a set of computations associated with the operation. This driver allows communication with the hardware accelerator to bypass a set of intermediate drivers executing between the driver of the host and the hardware accelerator. With this driver, the application in some embodiments receives the calculation results, which it then provides to one or more O-RAN components (e.g., to the E2 unit, another E2 unit, or another xApp that provided the data).

Fig. 14-20 illustrate several different embodiments for offloading O-RAN operations to a CP or edge application with through access to the hardware accelerator of its host computer. Examples of such hardware accelerators include Graphics Processing Units (GPUs), field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), and structured ASICs.

Fig. 14 illustrates an example of a CP or edge application 1402 with pass-through access to the hardware accelerator 1450 of its host computer 1410 to perform some or all of its computations. As shown, each application 1402 executes on a cluster 1404, the cluster 1404 having an accelerator driver 1412, the accelerator driver 1412 having direct, through access to an accelerator 1450 of its host computer 1410. Each cluster 1404 operates within a VM 1406 (i.e., executes on the VM 1406), which in turn executes on the hypervisor 1408 of the host computer.

In some embodiments, a cluster is a small deployable computing unit that can be created and managed in Kubernetes. A cluster includes a set of one or more containers with shared storage and network resources, and specifications of how to run the containers. In some embodiments, the contents of the clusters are always co-located and co-scheduled and run in a shared context. The cluster models application specific logical host computers; it contains one or more application containers that communicate with each other. In some embodiments, the shared context of the cluster is a set of operating system namespaces (e.g., linux cgroups). In the context of a cluster, the individual applications may apply further sub-isolation.

The accelerator drivers 1412 of each cluster have direct access 1450 to the hardware accelerator, and this access bypasses the hardware accelerator drivers 1414 and 1416 of the VM 1406 and the hypervisor 1408. In some embodiments, the hypervisor 1408 executes on an operating system (not shown) of the host computer 1410. In these embodiments, direct access to the hardware accelerator 1450 by the accelerator driver 1412 of each cluster also bypasses the hardware accelerator driver of the operating system.

To communicate with the hardware accelerator, in some embodiments, each application 1402 communicates through the RIC SDK 1430 executing on its cluster. For example, in some embodiments, each application 1402 uses the high-level API of the RIC SDK 1430 to communicate with the hardware accelerator 1450. The RIC SDK 1430 then converts the high level APIs into low level APIs required to communicate with the machine's driver 1412, which in turn relays the communication to the hardware accelerator 1450. The low-level API is provided by a first company associated with the sale of the hardware accelerator 1450, while the RIC SDK 1430 is provided by a second company associated with the distribution of the RIC SDK 1430. In some embodiments, the low-level APIs used by RIC SDK 1430 are APIs specified in API library 1432 associated with hardware accelerator 1450.

Fig. 15 illustrates a process 1500 implementing the method of some embodiments. Process 1500 is performed in response to the O-RAN component instructing the CP or edge application to perform operations requiring the application to use the hardware accelerator of its host computer. This process 1500 will be described below with reference to fig. 16, which illustrates that the application 1402 performs operations based on data received from the E2 node 1650.

As shown in fig. 15, process 1500 begins when an application 1402 receives data (at 1505) from an O-RAN E2 unit 1650 executing on a host computer 1610. In some embodiments, application 1402 subscribes to data from E2 unit 1650, and the data received at 1505 is responsive to this subscription. In some embodiments, this subscription is made through a distributed near RT RIC 1640. In some embodiments, the host computers 1410 and 1610 of the application 1402 and the E2 unit 1650 operate in one SDDC. In other embodiments, the two host computers 1410 and 1610 operate in two different physical locations. For example, host computer 1410 operates in a first location and host computer 1610 operates in a second location near the cell site of the O-RAN. In some embodiments, the second location is free of a computer with a hardware accelerator that performs complex operations including the received operations.

Application 1402 receives data from E2 unit 1650 (at 1505) through (1) distributed near RT RIC 1680 formed by near RT RICs 1640 and 1645 executing on host computers 1410 and 1610, and (2) RIC SDK 1430 executing on its cluster 1404. The application 1402 then uses (at 1510) the hardware accelerator 1450 to perform a set of computations associated with the operation.

To communicate with the hardware accelerator 1450, the application 1402 uses a high-level API provided by the RIC SDK 1430. The RIC SDK 1430 then converts the high level APIs into low level APIs specified in the API library 1432 associated with the hardware accelerator 1450. These low-level APIs are then transferred by the clustered driver 1412 to the hardware accelerator 1450 through its direct, pass-through access to the accelerator 1450, bypassing the drivers 1414 and 1416 of the VM 1406 and the hypervisor 1408. Through this driver 1412, the API specified in the API library 1432, and the RIC SDK 1430, the application 1402 also receives the results of operations (e.g., calculations) performed by the hardware accelerator 1450.

The application 1402 provides (at 1515) the results of its operations to one or more O-RAN components, such as the E2 unit 1650 or SDL store that provides data for the startup process 1500. This result is provided by RIC SDK 1430 and distributed near RT RIC 1680. In other embodiments, the application 1402 provides (via the RIC SDK 1430) the results of its operations to one or more other applications (applications other than the E2 unit that provided the data on which the application performed the operations) operating on another O-RAN E2 unit or on the same host computer or on a machine executing on another host computer as the application performing the operations using the hardware accelerator 1450. After 1515 process 1500 ends.

Other embodiments use pass-through access for O-RAN control or edge applications in other deployment settings. For example, fig. 17 illustrates another example of a CP or edge application 1702, the CP or edge application 1702 having a pass-through access to its host computer 1710 hardware accelerator 1750 to perform some (or all) of its computations. In this example, each application 1702 (1) executes on a cluster 1704, the cluster 1704 executes on a VM 1706, and (2) uses the accelerator driver 1712 of this VM 1706, the accelerator driver 1712 having direct, through access to the accelerator 1750 of its host computer 1710. VM 1706 executes on top of hypervisor 1708, hypervisor 1708 operating on host computer 1710. The accelerator driver 1712 of the VM bypasses the hardware accelerator driver 1716 of the hypervisor 1708. In some embodiments, the hypervisor 1708 executes on an operating system (not shown) of the host computer 1710. In these embodiments, direct access by accelerator driver 1712 of the VM to hardware accelerator 1750 bypasses the hardware accelerator driver of the operating system.

To use the hardware accelerator 1750, in some embodiments, each application 1702 uses the high-level APIs of the RIC SDK 1730 (executing on its cluster 1704) to communicate with the hardware accelerator 1750. The RIC SDK 1730 converts the high-level APIs into low-level APIs required to communicate with the driver 1712 of the VM, which driver 1712 in turn relays the communications to the hardware accelerator 1750. In some embodiments, the low-level APIs used by RIC SDK 1730 are APIs specified in API library 1732 associated with hardware accelerator 1750. This API library 1732 is part of the driver interface of VM 1706.

Fig. 18 illustrates yet another example of a CP or edge application 1802, the CP or edge application 1802 having a pass-through access to a hardware accelerator 1850 of its host computer 1810 to perform some or all of its computations. In this example, each application 1802 (1) executes on a VM 1804, the VM 1804 executing on a hypervisor 1806 operating on a host computer 1810, and (2) uses an accelerator driver 1812 of its VM 1804, the accelerator driver 1812 having direct, through access to an accelerator 1850 of its host computer 1810.

The accelerator driver 1812 of the VM bypasses the hardware accelerator driver 1816 of the hypervisor 1806. In some embodiments, the hypervisor 1806 executes on top of an operating system (not shown) of the host computer 1810. In these embodiments, direct access by the accelerator driver 1812 of the VM to the hardware accelerator 1850 bypasses the hardware accelerator driver of the operating system.

To use the hardware accelerator 1850, in some embodiments, each application 1802 uses the high-level APIs of the RIC SDK 1830 (executing on its cluster 1804) to communicate with the hardware accelerator 1850. The RIC SDK 1830 converts the high level APIs into low level APIs required to communicate with the driver 1812 of the VM, which driver 1812 in turn relays the communication to the hardware accelerator 1850. In some embodiments, the low-level APIs used by the RIC SDK 1830 are APIs specified in an API library 1832 associated with the hardware accelerator 1850. This API library 1832 is part of the driver interface of the VM 1806.

One of ordinary skill will recognize that in other embodiments, pass-through access to O-RAN control or edge applications is used in other deployment settings. For example, applications in other embodiments operate on containers, rather than clusters. These embodiments then use the hardware accelerator drivers of their clusters or VMs to make pass-through access to the hardware accelerator for control or edge applications. In some of these embodiments, the control or edge application communicates with the hardware accelerator through its associated RIC SDK and with other O-RAN components (to receive data and provide its results of data processing) through its associated RIC SDK and the distributed near RT RIC connecting the O-RAN components and the application. In some embodiments, the control or edge application in these embodiments performs a process similar to process 1500 of FIG. 15.

The above-described direct, through access to the hardware accelerator is very beneficial for the O-RAN. RIC decouples intelligence that was often embedded in RAN software (CUs and DUs) and moves it to the cloud. One of the benefits of doing so is to use more advanced computations in the cloud for xApp and edge operations (e.g., for ML, deep learning, reinforcement learning for control algorithms, etc.). DUs near cell sites typically cannot run advanced calculations because placing GPUs at each cell base station is not economically viable because the network capital expenditure will be very high.

By using a hardware accelerator (GPU, FPGA, eASIC, ASIC) in SDDC, some embodiments run complex control algorithms in the cloud. Examples of such xapps include massive MIMO beamforming and multi-user (MU) MIMO user pairing as described above. In general, any xApp whose computation may benefit from massive parallelization will obtain the benefits of a GPU or other accelerator. The use of ASICs facilitates channel decoding/encoding (turbo encoding, LDPC encoding, etc.). In some embodiments, the RIC is typically located on the same worker VM as the xApp. However, in other embodiments, the RIC is executed on a different host computer, such that more xapps that require GPUs and other hardware accelerators may run on a host with GPUs and/or other hardware accelerators.

Fig. 19 illustrates a process by which some embodiments deploy O-RAN applications with direct, pass-through access to the hardware accelerators of their host computers. To install an application on a host computer, process 1900 selects (at 1905) a set of one or more installation files that include a description for configuring the pass-through access of the application to the hardware accelerator of the host computer. In some embodiments, the set of files includes one description file specifying direct, through access by an application to the hardware accelerator of its computer.

The process 1900 configures (at 1910) a program executing on the host computer based on the description related to the pass-through access using the set of installation files to pass calls from a particular hardware accelerator driver associated with the application to the hardware accelerator without passing through a set of intermediate one or more drivers of the hardware accelerator executing on the host computer between the particular hardware accelerator driver and the hardware accelerator. This configuration allows the application to bypass the set of intermediate drivers when instructing the hardware accelerator to perform the operation of the application and receiving the operation result from the hardware accelerator.

In some embodiments, the program configured at 1910 is the operating system of the host computer, while in other embodiments it is a hypervisor executing on the host computer. In other embodiments, the program is a Virtual Machine (VM) and the application operates on a cluster or container executing on the VM. Process 1900 completes (at 1915) the installation of the application by processing the remaining set of installation files selected at 1905, and then ends. In other embodiments, process 1900 performs the configuration of the program as its last operation rather than as its first operation at 1910. In other embodiments, it performs this configuration as one of its intermediate installation operations.

Before performing the selection and configuration, the deployment process of some embodiments identifies a host computer of the plurality of host computers as the computer on which the application should be installed. The process in some embodiments identifies a host computer by determining that the application requires a hardware accelerator, identifying a set of host computers each including the hardware accelerator, and selecting a host computer from the set of host computers. The process selects a host computer by: (1) Determining that an application will need to communicate with a set of one or more other applications executing on a selected host computer, and (2) selecting the host computer when a set of other applications are concurrently executing on the host computer. Such installation of the application with a set of other applications on the selected host computer reduces communication latency between the application and the set of other applications.

Some embodiments communicate the hardware accelerator driver of the O-RAN control or edge application with a virtualized hardware accelerator provided by an intermediate virtualized application (e.g., a hypervisor) executing on the same host computer as the application. For example, the method of some embodiments deploys a virtualized application on a host computer for sharing resources of the host computer among several machines executing on the host computer. This computer has a first set of one or more physical hardware accelerators.

The method deploys several applications on several machines to perform several O-RAN related operations for a set of O-RAN components. By virtualizing the application, the method defines a second set of two or more virtual hardware accelerators mapped by the virtualized application to the first set of physical hardware accelerators. The method assigns different virtual hardware accelerators for different applications. The method also configures the application to perform its operations using the virtual hardware accelerator assigned to it.

In some embodiments, the deployed machine is a cluster and the application is deployed to execute on the cluster. At least two clusters execute on one VM executing on top of the virtualized application. This VM includes a hardware accelerator driver configured to communicate with two different virtual hardware accelerators for two applications executing on two clusters. In other embodiments, multiple clusters execute on one VM executing on top of a virtualized application, and each cluster has a hardware accelerator driver configured to communicate with a virtual hardware accelerator assigned to the driver.

Fig. 20 illustrates an example of a CP or edge application 2002 having through access to virtual hardware accelerators 2052 and 2054 defined by a hypervisor 2008 executing on their host computer 2010 in order to perform some or all of their computations. As shown, each application 2002 executes on a cluster 2004 having an accelerator driver 2012, the accelerator driver 2012 having direct, through access to a virtual accelerator 2052 or 2054. Each cluster 2004 operates within a VM 2006 (i.e., executes on the VM 2006), which in turn executes on top of the hypervisor 2008 of the host computer 2010.

The accelerator driver 2012 of each cluster has direct access to the virtual accelerator 2052 or 2054, and this access bypasses the accelerator drivers 2014 and 2016 of the VM 2006 and the hypervisor 2008. In some embodiments, hypervisor 2008 is executed on top of an operating system (not shown) of host computer 2010. In these embodiments, direct access by the accelerator driver 2012 of each cluster to the virtual accelerator 2052 or 2054 also bypasses the hardware accelerator driver of the operating system.

As shown, virtual accelerators 2052 and 2054 communicate with hardware accelerator 2050 through accelerator manager 2060 of hypervisor 2008. The accelerator manager 2060 allows virtual accelerators 2052 and 2054 (and their associated applications 2002) to share one hardware accelerator 2050 while operating with this accelerator 2050 as if the accelerator 2050 were dedicated to their respective applications and clusters 2002 and 2004. Examples of such hardware accelerators 2050 include Graphics Processing Units (GPUs), field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), and structured ASICs.

To communicate with its virtual accelerator 2052 or 2054, in some embodiments, each application 2002 communicates through the RIC SDK 2030 executing on its cluster 2004. For example, in some embodiments, each application 2002 uses the high-level API of RIC SDK 2030 to communicate with its virtual accelerator 2052 or 2054. The RIC SDK 2030 then converts the high-level APIs into the low-level APIs required to communicate with the driver 2012 of each machine, which driver 2012 in turn relays the communications to the virtual accelerator 2052 or 2054. The virtual accelerator 2052 or 2054 then relays communications to the hardware accelerator 2050 through the accelerator manager 2060.

As mentioned above with reference to fig. 14, in some embodiments, the low-level API is provided by a first company associated with the sale of the hardware accelerator 2050, while the RIC SDK 2030 is provided by a second company associated with the distribution of the RIC SDK 2030. In some embodiments, the low-level APIs used by RIC SDK 2030 are APIs specified in API library 2032 associated with hardware accelerator 2050. Each application 2002 receives the result of the operation of the hardware accelerator 2050 through the accelerator manager 2060, its virtual accelerator 2052 or 2054, its driver 2012, and its RIC SDK 2030.

To provide a low latency near RT RIC, some embodiments separate the functionality of the RIC into several different components that operate on different machines (e.g., executing on VMs or clusters) that operate on the same host computer or different host computers. Some embodiments also provide a high-speed interface between these machines. Some or all of these interfaces operate in a non-blocking, lock-free manner to ensure that critical near RT RIC operations (e.g., data path procedures) are not delayed by multiple requests causing one or more components to stall. Furthermore, each of these RIC components also has an internal architecture designed to operate in a non-blocking manner, such that none of the processes of the component blocks the operation of the other process of the component. All of these low latency functions allow the near RT RIC to be used as a high speed IO between the E2 node and the xApp.

FIG. 21 illustrates an example of a near RT RIC 2100 with several components operating on several different machines. In this example, the near RT RIC is divided into three clusters, namely, data path cluster 2105, service cluster 2110, and SDL cluster 2115. In some embodiments, this RIC (1) handles E2AP messages between E2 node 2118 and xApp 2120, (2) manages connections with E2 node 2118, (3) handles subscriptions of the E2 node by xApp, and (4) handles xApp liveness operations. RIC 2100 provides reliable low latency messaging between its various components, between the E2 node and the xApp, and between the E2 node/xApp and RIC components. Part of the RIC low latency architecture is attributable to the use of different clusters to implement data IO, service, and SDL operations, such that different resource allocation and management operations may be provided to each of these clusters based on their respective needs for performing operations in several different clusters.

Each of the three RIC clusters 2105, 2110, and 2115 communicates with one or more xApp clusters 2120. In some embodiments, each cluster (2105, 2110, 2115, or 2120) is allocated hardware resources (e.g., CPU, memory, disk storage, network IO, etc.) according to the unique requirements of the cluster (i.e., according to the data path, service, and storage operations performed by each cluster). Moreover, in some embodiments, each cluster has its own high availability and lifecycle update configuration that matches the unique needs of each cluster.

In some embodiments, service cluster 2110 performs xApp loading, registration, FCAPS (failure, configuration, billing, performance, security) and other services. For example, in some embodiments, service cluster 2110 provides management service 554 of near RT RIC, and performs O1 termination 580 and A1 termination 582 for SMO and its associated non-RT RIC. In some embodiments, each of these components 554, 580, and 582 operate on separate containers in service cluster 2110, while in other embodiments, two or more of these components operate on one container in service cluster 2110.

As mentioned above, in some embodiments, the A1 interface is located between a near RT RIC and a non-RT RIC. Through this interface, near RT RIC relays relevant network information reported by E2 nodes (e.g., CUs and DUs), and non-RT RIC provides control commands (e.g., use case operations for controlling non-RT granularity) for E2 nodes. The O1 interface is located between the near RT RIC and SMO and is used in some embodiments for discovery, configuration, resource management and auto scaling, lifecycle management, and fault tolerance.

In some embodiments, RIC management services 554 include services provided by near RT RIC to xApp and other RIC components. Examples of services provided to an xApp include xApp services registry/directory (xapps may be used to identify other xapps associated with a distributed near RT RIC and operations performed by these other xapps) and FCAP operations, such as metrics collection, policy provisioning, and configuration. In some embodiments, an xApp may query a service registry/directory to identify other xapps or other xapps that execute a particular service, and may register to receive notifications about xapps and their capabilities when xapps are added to the directory.

Examples of FCAP operations performed by service cluster 2110 for an xApp include fault operations involving collecting metrics that monitor CPU and memory utilization for analysis to alert, configuration operations for configuring or reconfiguring the xApp, billing operations for collecting data needed for billing, and performance operations for collecting metrics from the xApp for analysis to quantify xApp performance.

For other RIC components (e.g., data path cluster 2105 and SDL cluster 2115), service cluster 2110 also performs services such as metric collection, policy provisioning, and configuration. The service cluster 2110 may be considered a local controller that performs operations under the direction of a central controller (which is a SMO). Through SMO, service cluster 2110 will receive configuration and policies to distribute to the xApp and other RIC components. Moreover, the service cluster 2110 provides metrics, logs, and trace data collected from xApp and/or RIC components (e.g., data path clusters and SDL clusters) to SMO. In some embodiments, a service cluster may be extended (e.g., replicated) and backed up independently of other clusters. In some embodiments, the service cluster has a data cache, which is a cache for a time-series database of SMOs. In this cache, the service cluster stores statistics, logs, tracking data, and other metrics that it collects from the xApp and one or more RIC components before uploading those data to the SMO database.

SDL cluster 2115 implements SDL 560 and its associated database 570. As described further below, in some embodiments, the SDL cluster 2115 also executes one or more service containers to perform one or more pre-processing or post-processing services on data stored in the SDL. Like the service clusters, the SDL clusters in some embodiments may be expanded (e.g., replicated) and backed up independently of other clusters.

Data path cluster 2105 includes several important near RT RIC components. These are E2 termination 584, conflict mitigation 550, application subscription management 552, and RIC SDK interface 2150. As described further below, in some embodiments, some or all of these data path services are embedded in the data path threads and control threads of the data path clusters. In other embodiments, the data path services are embedded in a data IO thread, a plurality of data processing threads, and a control thread.

A thread is a component of a process executing on a computer. The process may be part of an application or a larger application. A thread is a series of programmed instructions that can be managed independently of other threads of a process. Multiple threads of a given process may execute concurrently (e.g., by using the multi-threaded functionality of a multi-core processor) while sharing memory allocated to the process. Multithreading is a programming and execution model that allows multiple threads to exist in the context of a process. These threads share the resources of the process but can execute independently.

In some embodiments, the control thread is an interface with the service cluster and SDL cluster for the datapath thread, while in other embodiments it is an interface with only the service cluster for the datapath thread (since the datapath thread can communicate directly with the SDL cluster). The control threads in any of these methods perform the slower, control-related operations of the datapath, while one or more datapath threads perform the faster IO operations of the datapath. In some embodiments, the control thread interfaces with the service cluster to receive configuration data for configuring its own operations as well as the operations of the datapath threads.

Embodiments that separate the datapath threads into a datapath thread and a plurality of data processing threads further optimize the datapath by pushing the more computationally intensive operations of the datapath thread into the plurality of datapath processing threads, which then allows the less computationally intensive operations to run in the datapath thread. Both optimizations are intended to ensure a fast data path IO (a data path IO that does not experience undesirable latency) so that a near RT RIC can be used as a high speed interface between the E2 node 2118 and the xApps 2120.

As mentioned above, clusters 2105, 2110 and 2115 communicate over a high speed inter-cluster interface. In some embodiments, the cluster-to-cluster connection is established through SCTP (stream control transmission protocol) or through an even higher speed shared memory (shmem) connection. In some embodiments, a shared memory connection is employed between only a pair of clusters executing on the same host computer. Examples of such pairs of clusters include (1) data path clusters and SDL clusters, (2) data path clusters and service clusters, (3) service clusters and SDL clusters, (4) xApp clusters and data path clusters, (5) xApp clusters and SDL clusters, and so forth. In some embodiments, the shared memory is lock-free and access thereto is non-blocking. Other embodiments use a slower interface (e.g., gRPC) between the service cluster 2110 and the other clusters 2105, #115, and 2120, as the service cluster is not as critical as the other clusters.

In some embodiments, different clusters of near RT RIC (e.g., 2105, 2110, and 2115) may be executed on the same host computer or may be executed on two or more host computers. In other embodiments, one or more of the clusters (e.g., service cluster 2110) always operate on a separate host computer from the other clusters (e.g., data path cluster 2105 and SDL cluster 2115). Moreover, in some embodiments, clusters 2105, 2110, and 2115 operate on one host computer 2205 along with one or more xApp clusters 2220a, while other xApp clusters 2220b operate on other host computers 2210, as shown in fig. 22. In other embodiments, two of clusters 2105, 2110, and 2115 operate on one host computer with one or more xApp clusters, while another one of clusters 2105, 2110, and 2115 operates on another host computer with one or more xApp clusters.

For example, FIG. 23 illustrates data path cluster 2105 and service cluster 2110 executing on host computer 2300 with two xApp clusters 2320a, while SDL cluster 2115 executing on host computer 2310 with two xApp clusters 2320 b. In some embodiments, as mentioned above, clusters of hardware accelerators are required to be placed on host computers with such hardware resources, while clusters of these accelerators are not required to be placed on host computers without these accelerators. In some embodiments, the SDL cluster and the xApp cluster use hardware accelerators, while the data path cluster and the service cluster do not. Various examples of clusters that may benefit from hardware accelerators and bypass paths to these accelerators are described above and further below.

Moreover, while several near RT RICs are described above and below as being implemented with clusters, near RT RICs in other embodiments employ VMs to implement RIC components. Moreover, even in embodiments where different RIC components are implemented with clusters, some or all of the clusters operate on VMs, such as lightweight VMs (e.g., photon VMs provided by VMware corporation).

In addition to using a fast communication interface between clusters, in some embodiments, some or all of the clusters use non-blocking, lock-free communication protocols and architectures. For example, data path cluster 2105 uses non-blocking, lock-free communications between threads and processes that make up this cluster. Data path cluster 2105 also uses non-blocking, lock-free communications when communicating with service cluster 2110, SDL cluster 2115, or xApp cluster 2120. The non-blocking communication ensures that a first component sending a request to a second component does not stop operation of the second component when the second component is processing too many requests. In such a case, the second component would instruct the first component to resend its request at a later time. The datapath cluster employs lock-free communication because it uses single-threaded processing that does not employ thread switching. Thus, there is no need to lock any portion of memory to ensure that another process thread does not modify it during the transition period.

In some embodiments, the communication interface between RIC SDK interface 2150 of data path cluster 2105 and RIC SDK 2112 of xApp cluster 2120 is also novel. In some embodiments, this interface parses the header of the E2AP message received from the E2 node, storing some or all of the parsed components in a new encapsulation header that encapsulates the E2SM payload of the E2AP message and some or all of the original E2AP header. In doing so, in some embodiments, the SDK interface 2150 performs certain optimizations, such as efficiently performing data packaging to reduce message size overhead for communications from one cluster to another (e.g., reducing the size of E2 global ID values, etc.). These optimizations improve the efficiency of near RT RIC datapath cluster and xApp cluster communications.

The near RT RIC in other embodiments has one or more other clusters. For example, fig. 24 shows a near RT RIC 2400, where near RT RIC 2400 includes a lifecycle management (LCM) cluster 2405 in addition to clusters 2105, 2110, and 2115. LCM clusters are specialized service clusters responsible for upgrading other clusters 2105, 2110 and 2115 of near RT RIC 2400. Separating lifecycle management from service clusters 2110 allows service clusters 2110 to be upgraded more easily.

In some embodiments, LCM cluster 2405 uses different upgrade methods to upgrade different clusters. For example, in some embodiments, the LCM cluster replicates an SDL data store and seamlessly transitions from an active data store to another standby data store in order to perform an uninterrupted upgrade of the SDL. On the other hand, to upgrade the data path cluster, the LCM cluster process is more complex because it configures the active and standby data path clusters as dual-host connections for each E2 node and each xApp, and configures the active data path clusters to replicate the state of the standby data paths.

Fig. 25 presents a more detailed view of a near RT RIC 2500 of some embodiments. As shown, near RT RIC 2500 includes a data path cluster 2505, a service cluster 2510, and an SDL cluster 2515.SDL cluster 2515 includes SDL agent 2526 and shared SDL storage 2528 for RIC 2500, while service cluster 2510 includes service agent 2524 and O1 and A1 termination interfaces 2535 and 2537. The datapath cluster 2505 includes a datapath thread 2507 and a control thread 2509.

Data path thread 2507 provides a fast data path IO of the near RT RIC between E2 node 2518 and xApp 2520. In some embodiments, the data plane capabilities of the RIC may be extended by implementing the RIC datapath IO with one control thread and multiple datapath threads that share the load of datapath processing of the datapath cluster. Several such embodiments are described further below. The control thread 2509 performs several control operations associated with the data path of the RIC. Near RT RIC 2500 separates the control thread from the datapath thread because data IO operations need to be fast and should not be slowed down by control operations that can operate at slower rates. In some embodiments, the control thread and the data path thread are two threads in a single process (i.e., running in the same shared memory address space).

In some embodiments, each of these threads uses a non-blocking, lock-free interface to communicate with other components in this architecture (e.g., with the RIC SDK, service cluster agent, SDL agent, and/or E2 node) to the extent that they communicate with these other components. Moreover, in some embodiments, both threads use minimal OS system calls and run as an infinite loop. As described further below, the datapath thread and the control thread exchange data through two circles 2522 (referred to as cbuf), one of which handles messages from the datapath thread to the control thread and the other of which handles messages from the control thread to the datapath thread.

Control thread 2509 serves as a control interface to E2 node 2518, SMO 2530 (via service cluster agent 2524), xApp (e.g., via SCTP), and SDL layer (via SDL agent 2526). In some embodiments, the control thread is a main thread in communication with these external entities; however, as described further below, in some embodiments, the datapath threads also communicate with SDL 2515 through SDL agent 2526.

In some embodiments, the control thread 2509 handles all control functions. This thread sends various control parameters to other functions and in some embodiments enforces admission control. In other embodiments, the datapath thread 2507 enforces admission control and SMO specifies admission control through the service cluster. In some embodiments, control thread 2509 has control channel communication with the RIC SDK of the xApp cluster through SCTP. In other embodiments, the control thread communicates with the RIC SDK of the xApp cluster through the gRPC. Also, in some embodiments, when the xApp cluster and the datapath cluster execute on the same host computer, the control thread communicates with the RIC SDK through shared memory (shmem).

The control thread 2509 also provides a transport mechanism to transport statistics, logs, and trace data generated by the datapath thread 2507. In some embodiments, some or all of this data is transmitted to SDL cluster 2515 through SDL agent 2526 and/or to SMO 2530 through service agent 2524. In some embodiments, control thread 2509 cooperates security keys with the E2 node peers and passes these keys to the datapath thread, which uses them to perform its encryption/decryption operations.

Datapath thread 2507 provides high-speed IO between the E2 node and the xApp. In some embodiments, this thread handles RIC SDK interface and E2 termination operations, as well as conflict mitigation and xApp subscription operations. This thread performs asn.1 decoding on the E2AP message to extract the message data. In some embodiments, the datapath thread does not decode the E2SM payload of these messages. Datapath thread 2507 validates E2 and xApp messages and sequences. In some embodiments, the message types include E2 node setup and service updates, E2 node indication reports, xApp initiated subscriptions to E2 node data, and xApp initiated control requests.

In some embodiments, datapath thread 2507 runs an E2 state machine to create and maintain states (e.g., state of E2 nodes, subscriptions to E2 nodes, etc.) on behalf of the xApp. Also, in some embodiments, the datapath thread performs a table lookup to send a message to the xApp requesting the data. This thread also handles control requests from the xApp to the E2 node and forwards responses to those requests from the E2 node back to the xApp.

The datapath thread communicates with the xApp through SCTP when the xApp is located on another host computer, or through shared memory when the xApp is located on the same host computer. In some embodiments, the xApp message has CRC bits to detect corruption. These messages also carry time stamps and may be compressed in some embodiments. The datapath thread 2507 performs data replication for multiple subscriptions. The datapath thread 2507 also performs datapath security operations, such as by signing, encrypting, and decrypting data messages.

As mentioned above and described further below, in some embodiments, the datapath thread 2507 communicates with the control thread 2509 via a pair of loops 2522. In some embodiments, the message frequency between two threads may be tuned (e.g., configurable) from sub-millisecond to second for each ring pair. The datapath thread 2507 receives configuration data updates and state changes through the control thread. The datapath thread 2507 generates statistics, logs, and traces, and provides the generated statistics, logs, and trace data to the control thread for storage in the SDL and/or to the SMO.

In the case where multiple xapps attempt to set the same parameters to the same E2 node at the same time, the datapath thread 2507 also performs conflict management operations. For example, the conflict management operation ensures that two xapps do not attempt to change cellular network settings (e.g., the direction of the antenna) in different ways within a short period of time. In some embodiments, the conflict management of datapath threads employs different methods to resolve different types of conflicts, e.g., (1) for one set of requests, for a duration, a second request to modify a parameter is denied after receiving the conflicting earlier first request, (2) for another set of requests, when another higher priority xApp issues a conflicting request for the same parameter, it denies a request for the parameter from one xApp, (3) for another set of requests for another set of parameters, it accepts only requests issued by xapps that are allowed to issue such requests for a specific period of time. The policies to handle these conflicts are provided by SMO 2530 through agent 2524 of the service cluster.

In fig. 25, each xApp cluster 2520 may execute one or more xapps 2532 and interface with data path cluster 2505, SDL cluster 2515, and service cluster 2510 through RIC SDKs 2534 executing on the xApp clusters. Each RIC SDK provides a high-level interface for the xApp to communicate with RIC and E2 nodes. This high-level interface hides the details of the underlying implementation. The RIC SDK communicates with the RIC instance over a fast data IO communication channel (such as shared memory or SCTP).

The RIC SDK also uses control communication channels with the service cluster 2510 and control threads 2509 to perform xApp control operations such as xApp loading, registration, capability, subscription, FCAPS, etc. In some embodiments, when the xApp cluster (and its SDK) and data path cluster are operating on the same host computer, the control channel communication between the SDK and the control thread 2509 is through shared memory, and through SCTP when they are operating on different computers. Moreover, in some embodiments, control channel communication between the xApp cluster (and its SDKs) and the service clusters is through shared memory when the SDKs and service clusters operate on the same host computer, and through the gRPC when they operate on different computers. Other embodiments use SCTP for communication between the SDK and the service cluster when the xApp cluster (and its SDK) and the service cluster are operating on different host computers.

Some embodiments use protobuf when the RIC SDK communicates with the service cluster through the gRPC. Moreover, in some embodiments in which the RIC SDK communicates with the datapath cluster through a shared memory, the shared memory communication uses protobuf. The RIC SDK has APIs for data functions, e.g., E2 messages to and from E2 nodes. These APIs also include control function messaging such as load xApp (name, version, function), message subscription, keep-alive messaging, and communication with the A1 and O1 interfaces of SMO through the service cluster (e.g., communication for storing statistics, logs, and tracking data in time series databases on SMO or service clusters such as promethaus and ELK).

Some embodiments assign a datapath thread and a control thread to one processor core, an SDL to another processor core (to isolate it from data and control threads), and a service cluster to yet another processor core. When one or more xapps execute on the same host computer as the RIC, these xapps are assigned to cores that are different from the RIC cluster, where multiple xapps may be assigned to the same core or each core may be assigned to each xApp as desired.

To further improve the performance of RIC and xApp, other embodiments perform other hardware assignment optimizations, such as specific memory allocation (e.g., larger RAM allocation) and specific IO allocation. Examples of special IO allocations for some of the clusters include (1) SRIOV allocations for an xApp cluster on one host computer to communicate with a data path cluster on another host computer, (2) SRIOV allocations for a data path cluster to communicate with E2 nodes, (3) SRIOV allocations for an xApp cluster on one host computer to communicate with a service cluster on another host computer, and (4) gRPC or SCTP communications on the SRIOV allocations, where gRPC communications have lower bandwidth allocations and lower priority than SCTP communications.

In some embodiments, one RIC and several xapps are bundled together to operate on different clusters that operate on one VM. In some embodiments, multiple instances of RIC may also be deployed with different sets of xapps. Moreover, in some embodiments, xapps that need to talk to each other are bundled on the same VM.

As mentioned above, some embodiments do not implement the RIC datapath as one datapath thread, but rather as one dataio thread and multiple datapath processing threads. In some embodiments, each data path processing thread (DPT) is responsible for performing data path processing on a different set of E2 nodes, where each E2 node is assigned to only one data path processing thread. In some embodiments, the data IO thread identifies the DPT associated with the E2 message or the xApp message by hashing the E2 node identifier contained in the message and using the hashed value (obtained by hashing) as an index to a lookup table that provides a DPT identifier of the DPT that needs to process the data message.

Fig. 26 illustrates an example of a RIC datapath cluster 2600 with one data IO thread 2605, one control thread 2610, and multiple DPTs 2615. The DPT shares the data path processing load of data path cluster 2600. As shown, there are a pair of cbuf rings 2620 between each DPT 2615 and data IO thread 2605, between each DPT 2615 and control thread 2610, and between data IO thread 2605 and control thread 2610. Each ring 2620 in the cbuf pair communicates data messages in one direction from one of the two threads associated with the ring to the other thread, with one ring handling communications in one direction (e.g., from the first thread to the second thread) and the other ring handling communications in the other direction (e.g., from the second thread to the first thread).

Separating the data IO thread 2605 from the multiple DPTs 2615 optimizes the data IO of the data path cluster 2600 by pushing more computationally intensive operations into the DPTs, which then allows less computationally intensive IO operations to run in the data IO thread 2605. This optimization ensures a fast data path IO (IO that does not experience unnecessary latency) so that the RIC can be used as a high speed interface between the E2 node and xApp. Moreover, each E2 node is only responsible for one DPT thread 2615, which DPT thread 2615 is typically responsible for several E2 nodes. Because each E2 node is handled by one particular DPT, no two DPTs may attempt to change one or more records associated with one E2 node. Thus, data path cluster 2600 does not need to lock any records of the E2 node because there is an explicit division of responsibility for communication with the E2 node.

The data IO thread 2605 performs (1) managing connections to the E2 node and the xApp cluster, (2) transmitting data messages to and from the E2 node and the xApp cluster over these connections, (3) performing security operations, (4) communicating with the control loops of the control threads 2610 and DPT 2615, and (5) generating statistics, logs, and tracking data about the messages it processes.

Each DPT thread 2615 performs the following operations (1) message decoding and encoding operations (e.g., message encryption and decryption operations), (2) message verification operations, (3) sequence verification operations, (4) maintenance state machines to keep track of the E2 node and the state of the xApp requests and subscriptions, (5) performs conflict management, (6) communicates with the control loops of the control threads 2610 and DPT 2615, and (7) generates statistics, logs, and tracking data about the messages it processes.

FIG. 27 illustrates a process 2700 that a datapath thread executes in some embodiments to process a subscription request from an xApp. As shown, the process begins when the DPT receives (at 2705) an xApp subscription request from a data IO thread. For a particular set of data tuples maintained by a particular E2 node (e.g., a particular set of operating parameters or other parameters), the subscription request is directed to the particular E2 node.

At 2710, process 2700 determines whether it has subscribed to receive a particular set of data tuples with a particular E2 node. This may be the case if the DPT previously sent one or more subscription requests to a particular E2 node, either individually or together, requesting a particular set of data tuples or a larger set of data tuples comprising the particular set of data tuples.

When process 2700 determines (at 2710) that it has subscribed to a particular E2 node for receipt of a particular set of data tuples, it adds (at 2715) a new record or updates a previously specified record for the xApp in the subscription list of this E2 node and specifies in this record the particular set of data tuples that the xApp should receive. After 2715, the process ends.

On the other hand, when process 2700 determines (at 2710) that it has not subscribed to a particular E2 node for receiving a particular set of data tuples, it must either send a first subscription to the particular E2 node (if it has not yet subscribed actively to this node) or an updated subscription to the node (if it has an active subscription but does not include all of the data tuples in the particular set of data tuples specified in the request received in 2705).

Thus, in this case, process 2700 adds (at 2720) a new record or updates a previously specified record for the xApp in the subscription list of this E2 node, and specifies in this record the particular set of data tuples that the xApp should receive. Next, it sends an updated subscription request to the particular E2 node using the previously assigned RIC request ID. This updated subscription specifies all of the data tuples in the requested set of particular data tuples when previous subscriptions to the particular E2 node did not previously request these tuples, or specifies some of these data tuples when other data tuples in the particular set were previously requested by one or more previous subscriptions to the particular E2 node. After 2725, process 2700 ends.

FIG. 28 illustrates a process 2800 performed by data IO thread 2605 and DPT 2615 in some embodiments to process data messages from E2 nodes that one or more xApps should receive. As shown, process 2800 begins when a data path cluster receives a data message over an SCTP connection with an E2 node (at 2805). At 2810, the data IO thread 2605 generates a hash value from the ID of the E2 node. It then uses the hash value as an index to a lookup table (at 2815) to identify the DPT assigned to process the message associated with the E2 node.

At 2820, the data IO thread passes the received data message to the identified DPT (i.e., the DPT identified at 2815) along cbuf ring 2620 for passing messages from the data IO thread to the identified DPT. Next, at 2825, the DPT uses its data structure record (e.g., a record maintained by its state machine) to identify a set of one or more xApps that should fetch the E2 message. In some embodiments, the identified set of xapps are xapps that have subscribed to receive data (e.g., all or a subset of the data) from the E2 node.

At 2830, the DPT specifies a data message for the data IO thread 2605 to send to the identified set of xApp. This data message is in the encapsulation format described below with reference to table 1. The DPT then passes (at 2835) the data message along cbuf ring 2620 to data IO thread 2605, cbuf ring 2620 being used to pass the message from DPT 2615 to data IO thread 2605. Next, at 2840, data IO thread 2605 retrieves the data message from cbuf ring 2620, identifies the xApp that needs to receive the data message, and then sends the data message to each identified xApp. After 2840, the process ends.

Fig. 29 illustrates a process 2900 that data IO thread 2605 and DPT 2615 execute in some embodiments to process a data message from an xApp that should be sent to an E2 node. As shown, process 2900 begins when a data message is received (at 2905) by a data path cluster through an SCTP connection or shared memory communication with an xApp RIC SDK. This message is in the encapsulation format described below with reference to table 1. This message includes an E2 node identifier identifying the E2 node that should receive this message.

At 2910, the data IO thread 2605 generates a hash value from the ID of the E2 node. It then uses the hash value (at 2915) as an index to a lookup table to identify the DPT assigned to process the message associated with the E2 node. At 2920, the data IO thread passes the received data message to the identified DPT (i.e., the DPT identified at 2915) along cbuf ring 2620 for passing messages from the data IO thread to the identified DPT.

Next, at 2925, the DPT uses its data structure record (e.g., a record maintained by its state machine) to identify the E2 node that should receive the message. In some embodiments, the data message is a subscription request and the identified E2 node is the E2 node to which the xApp wants to subscribe. At 2930, the DPT specifies a data message for the data IO thread 2605 to send to the identified E2 node. This data message is in the E2AP message format required by the standard. The DPT then passes (at 2935) the data message along cbuf ring 2620 to data IO thread 2605, cbuf ring 2620 being used to pass the message from DPT 2615 to data IO thread 2605. Next, at 2940, the data IO thread 2605 retrieves the data message from the cbuf ring 2620, identifies E2 nodes that need to receive the data message, and then sends the data message to each identified E2 node. After 2940, the process ends.

In some embodiments, DPT 2615 may determine that a new subscription message need not be sent to the E2 node it identified at 2925. For example, before receiving (at 2905) a subscription request from a first xApp for a set of data tuples from an E2 node, the data path cluster previously sent a subscription request to the same E2 node for the second xApp for the same set of data tuples or for a larger set of data tuples including the data tuple of the first xApp request. In this case, DPT 2615 simply adds the first xApp to the subscription list of E2 nodes so that it can provide the value subsequently received from the E2 nodes to the first xApp. In some embodiments, DPT 2615 also supplies the values stored in the SDL that were previously received from the E2 node to the first xApp or instructs the xApp to obtain these values from the SDL.

In some cases, the first xApp requests additional data tuples from the E2 node that the second xApp did not previously request. In such cases, DPT 2615 will prepare an updated subscription message for the data IO thread to send to the E2 node requesting the newly requested data tuple of the first xApp. The DPT will also prepare such a message when the second xApp requests additional data tuples from the E2 node after its initial subscription.

In some embodiments, service cluster 2510 configures data path cluster 2600 to instantiate N DPTs at its startup, where N is an integer greater than one. For data path cluster 2600 of the near RT RIC, in some embodiments, number N is calculated based on the expected number of E2 nodes and xapps communicating with the E2 nodes through the near RT RIC. In some embodiments, data IO thread 2605 of data path cluster 2600 then assigns E2 nodes to DPTs based on the order in which subscription requests were received by it and the load on the DPT at the time of those requests.

Fig. 30 illustrates an example of a process 3000 used by a data IO thread 2605 to assign an E2 node to a DPT in some embodiments. As shown, process 3000 begins when (at 3005) data IO thread 2605 receives a first subscription request for a particular E2 node from an xApp associated with a near RT RIC of the data IO thread. The first subscription request for a particular E2 node means that the data IO thread did not previously receive other subscription requests for this particular E2 node.

Next, at 3010, the data IO thread 2605 generates an N-bit hash value from the global E2 node ID of the particular E2 node, where N is an integer (e.g., is 6 or 8). This N-bit value is used to identify a particular E2 node in the hash LUT (look-up table), as described further below. At 3015, process 3000 selects a particular DPT for a particular E2 node based on the current load on each DPT of data path cluster 2600 (e.g., by selecting the DPT with the least amount of load). In some embodiments, the current load is based only on the number of E2 nodes assigned to each DPT, while in other embodiments, the current load is based on the number of E2 nodes and the number of xApp subscriptions to those nodes. In still other embodiments, the current load is calculated in other ways.

At 3020, process 3000 then creates a record in the LUT and associates in this record the N-bit hash value with the identifier of the particular DPT selected for the particular E2 node at 3015. In some embodiments, the N-bit hash value is an index in the LUT identifying records specifying the ID of a particular E2 node. At 3020, process 3000 also designates the status of this record as active.

At a later time, if the data IO thread encounters a situation where all xApps have canceled their subscription to a particular E2 node, then process 3000 maintains the LUT record created at 3020, but changes the state of this record to inactive. The data IO thread maintains this inactive state until the next xApp submits a subscription request for that particular E2 node, at which point the state of this record is changed to active again. This state value is used as a mechanism to ensure that the data IO thread does not have to constantly revisit the E2 node assignment to the DPT.

Fig. 31 illustrates a distributed near RT RIC implemented by an active RIC 3102 and a standby RIC 3104. As shown, E2 node 3118 and xApp cluster 3120 communicate with active RIC 3102 until one or more components of this RIC fail. When an active RIC fails, standby RIC 3104 becomes the active RIC, and the E2 node and the xApp cluster continue to communicate through RIC 3104 (which is now the active RIC).

Both RIC have the same components, namely, data path cluster 3105, service cluster 3110, and SDL cluster 3115. The datapath cluster is shown as including a control thread 3109 and datapath threads 3107. Some embodiments employ one data IO thread and multiple DPTs as mentioned above, rather than one data path thread 3107. In some embodiments, active RIC 3102 is implemented by a first set of one or more computers, and standby RIC 3104 is implemented by a second, different set of one or more computers.

As shown, each E2 node 3118 has a dual-homed SCTP connection with data path threads 3107 of active RIC 3102 and standby RIC 3104. Similarly, each xApp cluster 3120 has dual-host SCTP connections with data path threads 3107 of active RIC 3102 and standby RIC 3104. Dual host connectivity is a feature provided by SCTP. When the first component is connected to the active/standby component pair through the dual-homed connection, the first component may automatically switch to use the standby component when the active component fails. Thus, using a dual-homed SCTP connection, each E2 node or xApp cluster may switch to the datapath thread 3107 of the standby RIC 3104 when the active RIC 3102 or its datapath cluster fails.

As shown, the RIC SDK interface 3122 of the data path thread 3107 of active RIC 3102 forwards messages it receives from the xApp RIC SDK and messages it sends to the xApp RIC SDK to the RIC SDK interface 3122 of the data path thread 3107 of standby RIC 3104. In some embodiments this is done so that the datapath thread 3107 of the standby RIC can update its state machine to match the state of the datapath thread 3107 of the active RIC. Moreover, as shown, the synchronization agent 3127 of the active RIC 3102 and the standby RIC 3104 synchronizes the SDL storage 3126 of the standby RIC 3104 with the SDL storage 3126 of the active RIC 3102. All components of active RIC 3102 and standby RIC 3104 are consistently managed by SMO 3130.

Fig. 32 illustrates interfaces between near RT RIC 3200 and E2 node 3205 and between near RT RIC 3200 and xApp cluster 3210 in some embodiments. In some embodiments, the near RT RIC is one of the near RT RIC described above. As mentioned above, and as shown in fig. 32, in some embodiments, near RT RIC employs SCTP interfaces 3220 and 3222 with both E2 node 3205 and xApp cluster 3210. When the xApp cluster and the near RT RIC are executing on the same host computer, some embodiments use shared memory as an interface between the near RT RIC and the xApp cluster, as shown. These interfaces keep messages exchanged quickly and minimize coding and decoding overhead (e.g., performing one ASN decoding and one ASN encoding) on all paths, thereby minimizing latency. In some embodiments, the interface 3220 between the E2 node and the near RT RIC will follow the E2AP specification, and all message exchanges will conform to the E2AP specification.

Moreover, in some embodiments, interface 3222 between the near RT RIC and the xApp cluster uses a novel encapsulation header as will be described below with reference to table 1. Interface 3222 handles a mix of different types of messages. In some embodiments, examples of such messages include (1) an entire E2AP message (e.g., E2 setup request) from an E2 node, (2) some fields of an E2AP header and entire E2SM content (i.e., entire E2AP message payload) (3) internal messages between a near RT RIC and an xApp cluster (e.g., messages from the near RT RIC that an earlier message of an xApp caused an error), and (4) messages from an xApp to the near RT RIC or messages of the E2 node. In some embodiments, the E2 content may not be encoded by ASN1 (e.g., a portion of the subscription request may not be encoded).

In some embodiments, near RT RIC 3200 may be configured to decode only the E2AP message, or the entire E2AP header along with its E2SM payload, before it sends the message to the xApp, as the case may be. In some cases, the near RT RIC sends the entire E2AP header, while in other cases it sends only a portion of this header. In RIC E2AP message handling of some embodiments, all fields are in network byte order, and near RT RIC 3200 will work as much as possible in that order. To display the fields, some embodiments may translate the data into host order. In some embodiments, the near RT RIC 3200 will not look at the E2SM payload, while in other embodiments it will look (e.g., to avoid repeated subscription errors).

In some embodiments, the RAN function ID is specific to the E2 node. The xApp will not subscribe to RAN functions across E2 nodes, as each subscription will be to a separate E2 node. Also, in some embodiments, the RIC request ID space is local to the E2 node. In some embodiments, the RIC request ID number space is a temporary component as well as a persistent component. For example, the RIC request ID used to indicate reporting will persist, while the RIC request ID used from the subscription may be reused.

Table 1 below shows exemplary message formats used in some embodiments for communication between RIC and RIC SDKs. This is the format of the encapsulation header that encapsulates all messages to and from the RIC and RIC SDKs. In some embodiments, the encapsulation header stores the RIC SDK and data needed by the RIC to efficiently process the data message. In the example shown in Table 1, the first 16 bytes associated with msg_type, msg_serial_num, msg_len, msg_flags, and ctrl_len are part of the encapsulation header along with the ctrl_info field. The payload of the encapsulated packet may include any data. In the example shown in table 1, the payload includes the original E2AP packet along with its E2SM payload.

All messages between the RIC and RIC SDKs are encapsulated with the header shown in table 1. The control information and payload are optional. Some messages may have control information but no payload field, other messages may have payload but no control information, and some messages may have both control and payload fields. In some embodiments, the RIC SDK may be configured to capture these messages and reformat them for presentation to the xApp. The format of the message is the original byte stream. In some embodiments, the message CRC field is not used, while in other embodiments the message CRC field is used.

TABLE 1

Near RT RIC 3200 handles E2 node and xApp connections, disconnects, resets, and crashes as follows. RIC in some embodiments handles connections, disconnects, and crashes similarly for E2 nodes. In particular, when the connection with the E2 node is broken and restored for any of these reasons, the E2 node will send the connection setup again just as it started for the first time, and the near RT RIC will clear all its states related to the E2 node and restart. In some embodiments, when an E2 node connection is broken and restored, the near RT RIC notifies all xapps, whether they have previously subscribed to that particular E2 node or not, because that E2 node may advertise new functionality in which previously unsubscribed xapps may be of interest. When an xApp connects, disconnects, or crashes, the near RT RIC again performs the same operation, i.e. it resets all states of the xApp in the near RT RIC and deletes its subscription from all E2 nodes.

Fig. 33 illustrates E2AP message handling of a data path cluster of a near RT RIC 3200. In the following discussion regarding E2AP message handling and other message handling, this data path cluster is simply referred to as near RT RIC 3200 or RIC 3200 for purposes of brevity. As shown, near RT RIC 3200 initially decodes E2AP messages received from E2 nodes. In some embodiments, this decoding involves decoding of the E2AP header only, while in other embodiments, this decoding involves decoding of the E2AP header and the E2SM payload. For some or all of this decoding (e.g., E2 SM), in some embodiments, the near RT RIC uses its hardware accelerator (e.g., GPU accelerator) accessed through a bypass path, as described above

After decoding the E2AP message, the near RT RIC creates or updates its internal data structure to interpret the received data message, and then creates a flat packed message to the xApp in the format described above with reference to table 1. Since the near RT RIC and RIC SDKs operate on different containers and reside on different clusters in some embodiments, they do not pass any data structures to each other, but rather in some embodiments format their data exchanges into encapsulated messages with a specific byte sequence. After encapsulating the data message, the near RT RIC forwards the data message to the xApp cluster for forwarding to the appropriate xApp by the RIC SDK on this cluster.

The internal data structure created or updated by the near RT RIC when processing E2AP messages is used to process response messages from the xApp to the E2AP messages and to process subsequent E2AP messages. In some embodiments, examples of data stored in the internal data structure of the near RT RIC include (1) a subscription list of xapps that are interested in data from a particular E2 node, (2) a particular data tuple from each E2 node that is interested in each xApp, (3) a record identifying network addresses and other location data related to the E2 node and xApp, (4) an identifier (e.g., RIC request ID) that is assigned and assigned.

When the xApp sends a message, its RIC SDK processes the message and forwards it to the RIC along the shared memory or SCTP interface, as described above. The near RT RIC then parses the message and stores the parsed components. Based on these components and on one or more data tuples for the associated E2 node message(s) it stores in its internal data structure, the RIC creates an E2AP response, then encodes this response and forwards it to the E2 node it is directed to.

For example, after a first xApp sends a subscription request to receive M data tuples from an E2 node, the data path cluster of the near RT RIC creates a state to record the desired subscription of the first xApp, requests the E2 node to subscribe to the M data tuples, and forwards the M data tuples to the xApp each time they are received and after that they are received. In some embodiments, the data path cluster of the near RT RIC may be configured to forward M data tuples to its associated SDL each time they are received from the E2 node.

After the first xApp subscribes to receive M data tuples from the E2 node, the second xApp may subscribe to receive N different data tuples from the E2 node, where N is greater than M. The near RT RIC then sends an updated subscription request to the E2 node. This update now requests N data tuples. Whenever the near RT RIC receives N data tuples, it sends M data tuples to the first xApp and all N data tuples to the second xApp.

Another example involves the near RT RIC removing and caching the RIC request ID from the E2AP message from the E2 node in response to the subscription request. After removing this ID, the RIC provides the xApp with a portion of the E2AP message and its E2SM payload (if applicable). Subsequently, when xApp wants to delete a subscription, the RIC retrieves the RIC request ID from its state and inserts it into the E2AP message to the E2 node to request deletion of the subscription.

In some embodiments, the E2 setup, response message, and failure message of the near RT RIC are handled as follows. The near RT RIC initially receives a setup request message from the E2 node. In response, the near RT RIC will decode the message and build an internal data structure. The RIC will also cache the original ASN1 payload. In some embodiments, the near RT RIC accepts all added RAN function identifiers. In some embodiments, the near RT RIC sends a setup message to the xApp (i.e., as a message with an ASN1 encoded E2SM payload) after decoding the E2AP header but not others. In some embodiments, the setup message sent to xApp by the near RT RIC has a control length (ctrl_len) 0 and its ASN1 encoded payload.

When an xApp connects later, the near RT RIC will send all setup requests from the E2 node to the xApp, so it has a manifest of the connected E2 node. In some embodiments, the near RT RIC sends these messages one at a time. Also, as mentioned above, in some embodiments, the near RT RIC constructs and sends an E2 setup response to the E2 node. In some embodiments, the near RT RIC sends a failure message when a request format error is set up (e.g., it is a copy of the RAN function list, or removes records that are not added to the list).

After receiving the reset from the E2 node, the near RT RIC performs the following actions after decoding the message. It sends a message about this reset to all xapps subscribed to this E2 node. In some embodiments, this is an internal message without any ASN1 content. The near RT RIC sends a subscription delete message to the E2 node for all previous subscriptions it sent to. It also sends control, insert and policy delete to this E2 node. It clears all outstanding requests and sends a reset response to the E2 node.

In some embodiments, the near RT RIC also employs service update, acknowledgement, and failure messages. This message updates the supported RAN function list with additions, modifications and deletions. The near RT RIC informs all xapps about the new service configuration of the E2 node. In some embodiments, the near RT RIC sends a message to the xApp after applying the configuration, so it will reflect the final state of the configuration. In other embodiments, the near RT RIC sends messages as is for the xApp to calculate the delta between the previous state and the new state of the supported RAN functions. In this latter approach, the near RT RIC does not require ASN1 encoding of the resulting delta.

In some embodiments, the handling of E2AP subscriptions is as follows. xApp formats and ASN1 encodes the subscribed E2SM portion. Table 2 below details the control portion of the subscription message (i.e., the portion stored in the control field of the message from xApp to near RT RIC). The payload will be ASN1 encoded E2SM content. Multiple subscription message types are defined in some embodiments to disambiguate the selectable information. Also, in some embodiments, a message flag is used to specify the exact format. In some embodiments, each subscription message specifies an E2 node global ID and a RAN function ID.

In some embodiments, the E2 node sends a 113 byte identifier and the RIC compresses it to a 40 byte ID. When a subscribe message is sent to the E2 node, the RIC converts the 40 byte ID to a 113 byte ID. The subscribe message control field will be in a fixed format as much as possible. In some embodiments, the RIC caches all subscription requests and compares requests from multiple xapps to avoid issuing duplicate subscription messages. However, when a second subsequent xApp requests additional information from the same E2 node after the first initial xApp requests some information from the E2 node, the RIC resends the subscription (with the same RIC request ID in some embodiments), but this time requires additional information. In some embodiments, when a subscription request is sent to the E2 node, the RIC sends out the entire payload received from the xApp as an E2AP message payload.

TABLE 2

Near RT RIC handles E2AP RIC subscription responses by storing the E2 node global ID and RIC request ID (generated by RIC) as control information and sending an exact ASN1 encoded message back from the E2 node to the xApp.

An E2AP RIC subscription delete request, response or failure message is sent from the xApp to the near RT RIC, with the message field sent as control information (i.e., as part of ctrl_info). The near RT RIC creates and sends an encoded ASN1 message to the E2 node. The delete request does not specify the E2 node global ID. Thus, this information is provided by the RIC for the xApp. In some embodiments, the response message is sent as a packed byte (not ASN1 encoded) from the near RT RIC to the xApp.

The E2AP indication report of the E2 node is handled as follows. The message is decoded by the near RT RIC to determine the RIC request ID field. This helps determine which xApp(s) subscribe to the indication. In some embodiments, the near RT RIC sends a message encoded as ASN1 to the xAPP(s). In some embodiments, the near RT RIC also sends a reduced E2 global ID along with the message as control information.

In some embodiments, the near RT RIC processes the E2AP control request as follows. The xApp sends this request in the form of packed byte information. Near RT RIC does not specify an E2 global ID in the message because this information is specified by xApp. The near RT RIC format of this message is shown in table 3.

TABLE 3 Table 3

The near RT RIC handles the E2AP control response or failure message as follows. The near RT RIC decodes the message to obtain the RIC request ID. It then sends an ASN1 encoded message to the xApp and adds the global E2 node ID as control information in front.

In some embodiments, the SDL data store is a memory database running in its own set of one or more clusters. It is assigned its own computing and memory resources. As mentioned above, multiple near RT RIC instances define a distributed near RT RIC. In some embodiments, each near RT RIC instance has its own SDL instance storing system scope information for the RIC instance. Examples of such information include a list of connected E2 nodes (i.e., base station nodes), xapps, subscriptions from each xApp, and key cell data returned by the E2 nodes. Additionally, in some embodiments, each SDL instance provides services for preprocessing incoming data or post-processing data retrieved from its storage by running custom algorithms internally as the data arrives and by interfacing with a hardware accelerator (e.g., GPU).

The data IO cluster and xApp cluster of the RIC instance are connected to the SDL cluster of the RIC instance. In some embodiments, each SDL instance operates only with the data IO cluster and the service cluster of its own RIC instance. Also, in some embodiments, the SDL cluster is managed by SMO and configured via a service cluster. The data flows to and from the SDL include (1) data IO to and from the SDL data store, (2) xApp from the SDL data store, (3) xApp to the SDL data store, (4) data IO from SDL data access (e.g., retrieval of E2 node information, subscription information, etc.), and (5) service clusters communicate to and from the SDL to provide and retrieve configuration information.

Fig. 34 illustrates an RIC instance 3400 with an SDL cluster 3410 of some embodiments. As shown, SDL 3410 includes a configuration agent 3412, an SDL data path agent 3414, a RIC SDK agent 3417, SDL preprocessors 3416 and post-processors 3418, and one or more SDL data stores 3420.SDL configuration agent 3412 interfaces with service cluster agent 3430. This agent 3412 configures and manages the other components of the SDL cluster. The SDL cluster is managed by SMO and configured via service agent 3430 of service cluster # 090.

SDL data path agent 3414 is a data path interface that the SDL cluster exposes to control threads and data path threads of data path cluster 3450. The SDL data path agent 3414 handles communications from these entities for SDL and performs reads and writes to the SDL data store 3420 for these data path entities. In some embodiments, the SDL data path agent may be configured to communicate with the data path cluster 3450 using either SCTP or a shared memory vault.

In some embodiments, RIC SDK agent 3417 is an agent whose SDL cluster is exposed to the RIC SDK of the xApp cluster. The RIC SDK agent 3417 handles communication from the RIC SDK to the SDL, and performs reading and writing to the SDL data store 3420 for the RIC SDK. In some embodiments, the RIC SDK agent 3417 may be configured to communicate with the RIC SDK using either SCTP or a shared memory library. This agent 3417 also performs a cache synchronization operation to synchronize the SDL cache of the RIC SDK with the SDL data store(s) 3420. Moreover, the connection manager in some embodiments leverages the epoll connection handling of RIC instances (e.g., the epoll connection handling used by the data IO clusters in some embodiments) where it needs to extend to tens of connections from the xApp.

The SDL agents 3414 and 3417 handle event subscriptions and notifications from the RIC SDKs and data path clusters. This is separate from E2AP subscription management but is conceptually similar. For example, through such subscription services of the SDL, the xApp specifies its interest in certain data via the reported frequency and/or key. The RIC SDK agent 3417 then provides periodic updates to this xApp based on its subscription. In some embodiments, it also provides security services by encrypting and decrypting data.

In some embodiments, the data pre-and post-processors 3416 and 3418 are part of the SDL 3410 to flexibly run certain value-added algorithms on the data. In some embodiments, both of these processors 3416 and 3418 operate in one container, while in other embodiments, each of them operate on separate containers in the SDL cluster 3410. In some embodiments, each of these processors also interfaces with an external accelerator (e.g., GPU 3480) to perform their operations. The data preprocessor 3416 operates inline when data is stored in the SDL data store 3420.

In some embodiments, the data post processor 3418 operates inline when data is read from the SDL data store 3420. Alternatively or in combination, in some embodiments, the post-processor 3418 can be configured to run on data stored in the SDL data store 3420 in the background (e.g., retrieve data from this data store, perform some operations on this data, and store results back to the data store). In some embodiments, the data processor encodes and/or decodes the E2SM payload of the E2AP message. This is advantageous because it allows the datapath cluster to pass the ASN1 string to the SDL for decoding and storage. As mentioned above, in some embodiments, the RIC SDK may also be configured to provide E2SM encoding/decoding services.

Another example of a post-processor operation performed by the data processor 3418 in some embodiments is a machine-trained operation. In some embodiments, the post-processor 3418 collects various data tuples stored by various xapps and/or various clusters (e.g., data path clusters) and communicates the data tuples through a machine-trained network (e.g., a neural network trained through machine learning). In some embodiments, to perform its machine-trained network, the post-processor 3418 uses one or more hardware accelerators (e.g., one or more GPUs) of the host computer of the SDL to perform its operations. The post-processor 3418 accesses the hardware accelerator by the bypass method described above with reference to fig. 14-20.

Post processor 3418 passes the results of its operations to one or more xapps, or it may store the results in SDL data store 3420 for retrieval by one or more xapps. Examples of results obtained by post-processing SDL data with a machine-trained network include anomaly detection (e.g., identifying an E2 node that is behaving abnormally, e.g., suddenly receiving too many connected cell sites).

Using the machine-trained network in SDL cluster 3410 to handle the different xApp outputs stored in SDL data store 3420 is advantageous because this data store 3420 stores several xApp outputs and many data tuples provided by E2 nodes based on xApp subscriptions. Individual xapps can often only see the data tuples they subscribe to, as well as their own calculation results and the output of some other xapps. On the other hand, the SDL cluster 3410 may access larger E2 node input data and xApp output data sets. The post-processor 3418 uses algorithms (e.g., constrained optimization solvers) to post-process the data stored in the SDL data store 3420, rather than using a machine-trained network to perform such post-processing. In other words, in some embodiments, the post-processor 3418 does not use a machine-trained network, but still performs its operations using the hardware accelerator(s) of its host computer (e.g., through a bypass path).

Some embodiments also use post processor 3418 to provide the current state of the E2 node when the first xApp begins subscribing to the state of the E2 node. After the second xApp has subscribed earlier to receive the state of the E2 node, the near RT RIC stores a plurality of data tuples related to the state of this node for a period of time. When the first xApp subsequently subscribes to the state of the E2 node, and this xApp or data path cluster attempts to access this state of the first xApp, the post processor 3418 retrieves all of the data tuples previously stored in the SDL store for this E2 node and uses these to calculate the current state of the E2 node, which is then provided to the first xApp either directly or through the data path cluster.

SDL data store 3420 is an in-memory database. In some embodiments, the in-memory database is a database that is loaded into and runs from computer system memory (e.g., host computer volatile memory, including its RAM). One example of such a database is Redis. The size of the data store is selected in some embodiments to minimize search and storage latency. Once the existing data store reaches its maximum desired size, some embodiments create additional instances of this data store in the same or different instances of the SDL.

Also, as shown in fig. 34, in some embodiments, the SDL 3410 HAs an active data store 3420 and a standby data store 3421 for HA reasons. Furthermore, some embodiments allow the data store 3420 of different SDL instances of different RIC instances to synchronize some or all of their data in the background for HA and/or data availability reasons. In some embodiments, the xApp and RIC components read and write data to active SDL storage that is synchronized with standby SDL storage in the background. The RIC in these embodiments may seamlessly switch to a standby SDL storage when an active SDL storage fails. Also, the RIC may switch to a standby SDL storage when an active SDL storage is upgraded. This allows the SDL storage to be uninterrupted.

As mentioned above with reference to fig. 13, the RIC SDK includes an SDL cache that provides a local SDL storage for the xApp, and this cache synchronizes its data with the data store of the SDL. In some embodiments, this RIC SDK cache pulls data from the master SDL data store in bulk and periodically. It helps reduce the number of requests issued to the primary SDL cluster. The RIC SDK cache also reduces latency in reading SDL data by providing a portion of this data locally. RIC SDK caching also speeds up the time to write data to the SDL by allowing data to be written locally on the xApp cluster and synchronized in the background. In some embodiments, the size of the SDK cache is smaller than the data store 3420 of the SDL. Moreover, in some embodiments, this size is based on the requirements of a set of one or more xapps executing on the cluster with the RIC SDK.

In some embodiments, the data sources of SDL 3410 include (1) control threads and data path threads of data path cluster 3450, (2) xApp through RIC SDKs, (3) servers and ML models and policy stores of non-RT RIC accessed through the A1 interface, (4) xApp cluster configuration store, and (5) configuration servers of SMO. In some embodiments, examples of system data in the SDL (e.g., written by the control thread) include (1) E2 node information, (2) cell information, (3) UE information, (4) E2 node reports, (5) KPM (key performance metric) reports, and (6) topology information (from the EMS adapter).

Examples of SDL transactions in some embodiments include (1) a data IO cluster (control or data IO thread) writing data to the SDL that is then read by an xApp, (2) an xApp reading the written data from another xApp, or an xApp writing data to the SDL for another xApp, (3) an xApp writing the SDL such that a service container (e.g., post processor) operating in the SDL cluster performs an operation on the written data (e.g., by using a GPU service or using only a general purpose CPU), after which the result of this operation is retrieved from the SDL at the same xApp or another xApp, (4) as part of an A1 subscription, a non-RT RIC reading data from or writing data to the SDL, (5) an SMO storing O1 configuration data in the SDL, and (6) a non-RT RIC storing ML data in the SDL.

Fig. 35 conceptually illustrates an electronic system 3500 with which some embodiments of the invention are implemented. Electronic system 3500 may be a computer (e.g., desktop computer, personal computer, tablet computer, server computer, mainframe, blade computer, etc.) or any other type of electronic device. Such electronic system 3500 includes various types of computer-readable media and interfaces for a variety of other types of computer-readable media. Electronic system 3500 includes bus 3505, processing unit(s) 3510, system memory 3525, read-only memory 3530, persistent storage devices 3535, input devices 3540, and output devices 3545.

Bus 3505 generally represents all of the system, peripheral devices, and chipset buses that communicatively connect the numerous internal devices of electronic system 3500. For example, a bus 3505 communicatively connects the processing unit(s) 3510 with the read-only memory 3530, the system memory 3525, and the permanent storage device 3535.

From these various memory units, the processing unit(s) 3510 retrieve instructions to execute and data to process to perform the processes of the present invention. In different embodiments, the processing unit(s) 3510 may be a single processor or a multi-core processor.

Read Only Memory (ROM) 3530 stores static data and instructions required by the processing unit(s) 3510 and other modules of the electronic system 3500. On the other hand, persistent storage 3535 is a read-write memory device. This device 3535 is a non-volatile memory unit that can store instructions and data even when the electronic system 3500 is turned off. Some embodiments of the invention use a mass storage device (such as a magnetic or optical disk and its corresponding disk drive) as the permanent storage device 3535.

Other embodiments use removable storage devices (such as floppy disks, flash drives, etc.) as permanent storage 3535. Like persistent storage 3535, system memory 3525 is a read-write memory device. However, unlike storage device 3535, system memory 3525 is a volatile read-write memory, such as random access memory. The system memory 3525 stores some of the instructions and data that the processor needs at runtime. In some embodiments, the processes of the present invention are stored in system memory 3525, persistent storage 3535, and/or read-only memory 3530. From among these various memory units, the processing unit(s) 3510 retrieve instructions to execute and data to process to perform processes of some embodiments.

Bus 3505 is also connected to input and output devices 3540 and 3545. Input device 3540 enables a user to communicate information and selection commands to electronic system 3500. Input devices 3540 include an alphanumeric keyboard and pointing device (also referred to as a "cursor control device"). The output device 3545 displays images generated by the electronic system 3500. The output device 3545 includes a printer and a display device, such as a Cathode Ray Tube (CRT) or a Liquid Crystal Display (LCD). Some embodiments include devices that function as both input and output devices, such as touch screens.

Finally, as shown in fig. 35, bus 3505 also couples electronic system 3500 to network 3565 via a network adapter (not shown). In this manner, the computer may be part of a computer network (such as a local area network ("LAN"), a wide area network ("WAN") or an intranet or a network of networks (such as the Internet).

Some embodiments include electronic components, such as microprocessors, storage devices, and memory, that store computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as a computer-readable storage medium, a machine-readable medium, or a machine-readable storage medium). Some examples of such computer-readable media include RAM, ROM, compact disk read-only (CD-ROM), compact disk recordable (CD-R), compact disk rewriteable (CD-RW), digital versatile disk read-only (e.g., DVD-ROM, dual layer DVD-ROM), various recordable/rewriteable DVDs (e.g., DVD-RAM, DVD-RW, dvd+rw, etc.), flash memory (e.g., SD card), mini-SD card, micro-SD card, etc.), magnetic and/or solid state disk drives, read-only and recordable Discs, super-density optical discs, any other optical or magnetic medium, and floppy disks. The computer readable medium may store a computer program executable by at least one processing unit and including a set of instructions for performing various operations. Calculation ofExamples of a computer program or computer code include machine code, such as produced by a compiler, and files containing higher level code that are executed by a computer, electronic component, or microprocessor using an interpreter.

While the above discussion primarily refers to a microprocessor or multi-core processor executing software, some embodiments are performed by one or more integrated circuits, such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA). In some embodiments, such integrated circuits execute instructions stored on the circuit itself.

As used in this specification, the terms "computer," "server," "processor," and "memory" all refer to electronic or other technical devices. These terms do not include a person or group of people. For the purposes of this specification, the term display or display refers to displaying on an electronic device. The terms "computer-readable medium," "computer-readable medium," and "machine-readable medium," as used in this specification, are entirely limited to tangible physical objects that store information in a computer-readable form. These terms do not include any wireless signals, wired download signals, and any other transitory signals.

Although the invention has been described with reference to numerous specific details, one of ordinary skill in the art will recognize that the invention can be embodied in other specific forms without departing from the spirit of the invention. For example, the multiple figures conceptually illustrate the process. The specific operations of these processes may not be performed in the exact order shown and described. The specific operations may not be performed in one continuous sequence of operations and different specific operations may be performed in different embodiments. Furthermore, the process may be implemented using several sub-processes, or as part of a larger macro process.

Moreover, the several embodiments described above only show one hardware accelerator per host computer. However, one of ordinary skill will recognize that the methods and architectures of some embodiments may be used to provide direct, pass-through access to multiple hardware accelerators on one host computer. Furthermore, several of the embodiments described above relate to xApp operation and near RT RIC communication with xApp. One of ordinary skill will recognize that these embodiments are equally applicable to edge applications in telecommunications networks and near RT RIC communications with edge applications. It is therefore to be understood that the invention is not to be limited by the foregoing illustrative details, but is to be defined by the appended claims.

Claims (35)

1. A method of performing control plane operations in a Radio Access Network (RAN), the method comprising:

deploying a plurality of machines on a host computer;

on each machine:

deploying a control plane application to perform control plane operations;

a RAN Intelligent Controller (RIC) SDK is configured to serve as an interface between a control plane application on the same machine and a set of one or more elements of the RAN.

2. The method of claim 1, wherein on each machine, the RIC SDK includes a set of network connectivity procedures that establish network connections to the set of RAN elements for control plane applications, and allows control plane applications on the machine to relinquish having the set of network connectivity procedures.

3. The method of claim 2, wherein the set of network connectivity procedures of each RIC SDK of each machine establishes and maintains network connections between the machine and the set of RAN elements used by the control plane application of the machine, and handles data packet transmissions to and from the set of RAN elements by the control plane application.

4. The method of claim 1, wherein the control plane application on each machine communicates with the set of RAN elements through high level API (application program interface) calls that the RAN SDK converts to low level API calls.

5. The method of claim 4, wherein at least a subset of the low-level API calls are specified by a standard specification body.

6. The method of claim 4, wherein the high level API calls are made in a high level programming language and the low level API calls include low level calls that establish and maintain network connections and pass data packets through the connections.

7. The method of claim 4, wherein the set of RAN elements includes CUs (central unit) and DUs (distributed units) of the RAN.

8. The method of claim 7, wherein the RAN SDK on each machine communicates with the CUs and DUs over a low-level, standard-specified E2 interface, and the control plane application on the machine communicates with the CUs and DUs over the RAN SDK using high-level API calls that specify E2 interface operations at a high-level application layer that do not include low-level transport or network operations.

9. The method of claim 4, wherein the set of RAN elements comprises network elements of RIC.

10. The method of claim 9, wherein the RIC element comprises at least one Shared Data Layer (SDL) element.

11. The method of claim 9, wherein the RIC element comprises at least one data path input/output (I/O) element.

12. The method of claim 9, wherein the RIC element comprises at least one service management element.

13. A method of control plane application communication in a Radio Access Network (RAN), the method comprising:

deploying a plurality of control plane applications for execution on a plurality of host computers;

a plurality of RAN Intelligent Controllers (RIC) are deployed to execute on the plurality of host computers to implement a distributed RIC that serves as a communication interface between control plane applications.

14. The method of claim 13, further comprising configuring the first RIC to receive Application Programming Interface (API) calls from at least a first control plane application and forward the API calls to at least a second control plane application.

15. The method of claim 14, wherein the first control plane application and the second control plane application execute on the same host computer.

16. The method of claim 14, wherein the first RIC and the first control plane application execute on a first host computer and the second control plane application executes on a second host computer, and configuring the first RIC comprises configuring the first RIC to forward API calls from the first control plane application to the second RIC executing on the second computer for forwarding by the second RIC to the second control plane application.

17. The method of claim 14, wherein the first control plane application and the second control plane application are developed by two different application developers that communicate with each other through a distributed RIC using a common set of RIC APIs.

18. The method of claim 14, wherein configuring the first RIC comprises configuring the first RIC to add one or more parameters to the API call when the first RIC forwards the API call from the first control application to the second control application.

19. The method of claim 14, wherein the first control plane application and the second control plane application execute on first and second machines, the execution of the first and second machines executing on one host computer, the method further comprising configuring the RIC SDKs on each machine to receive and forward API calls between the distributed RIC and the first and second control plane applications.

20. The method of claim 19, wherein on each machine, the RIC SDK includes a set of network connectivity procedures that establish network connections to the set of RAN elements for the control plane application, and allows the control plane application on that machine to relinquish having the set of network connectivity procedures.

21. The method of claim 20, wherein the set of network connectivity procedures of each RIC SDK of each machine establishes and maintains network connections between the machine and the set of RAN elements used by the control plane application of the machine, and handles data packet transmissions to and from the set of RAN elements by the control plane application.

22. The method of claim 19, wherein the control plane application on each machine communicates with the set of RAN elements through high level API (application program interface) calls, the RAN SDK converting the high level API calls to low level API calls, wherein at least a subset of the low level API calls are specified by a standard specification body.

23. The method of claim 19, wherein the control plane application on each machine communicates with the set of RAN elements through high level API (application program interface) calls, the RAN SDK converting the high level API calls into low level API calls, wherein the high level API calls are made in a high level programming language, and the low level API calls include low level calls that establish and maintain network connections and pass data packets through the connections.

24. A method for a control plane application to use a Shared Data Layer (SDL) storage of a Radio Access Network (RAN), the method comprising:

Deploying a control plane application for execution on a host computer;

deploying an SDL cache on a host computer; and

at least a subset of the access requests are stored using an SDL of the SDL cache processing control plane application.

25. The method of claim 24, wherein the control plane application is executing on a machine executing on a host computer and the SDL cache is executing on the machine.

26. The method of claim 24, wherein the control plane application is executing on a machine executing on a host computer and the SDL is cached on the host computer.

27. The method of claim 24, further comprising deploying a RAN Intelligent Controller (RIC) to execute on the host computer and synchronize data stored in the SDL cache with data stored in the SDL storage.

28. The method of claim 27, wherein deploying the RIC comprises deploying the RIC to implement a distributed RIC with other RIC executing on other host computers.

29. The method of claim 28, wherein the SDL storage device operates on a host computer that is different from the host computer on which the control plane application is executing.

30. The method of claim 28, wherein at least a portion of the SDL storage operates on a host computer on which the control plane application executes.

31. The method of claim 24, wherein the control plane application is executing on a machine executing on a host computer, the method further comprising configuring the RIC SDK on the machine to process storage access requests from the control plane application.

32. The method of claim 31, wherein the SDL cache is part of a RIC SDK.

33. The method of claim 32, wherein

Deploying the RIC comprises deploying the RIC to implement a distributed RIC with other RICs executing on other host computers, an

The SDL storage is part of a distributed RIC.

34. The method of claim 33, wherein when the RIC SDK is unable to process the SDL access request from the control plane application via the SDL cache, the RIC SDK forwards the SDL access request to the SDL storage.

35. The method of claim 34, wherein the RIC SDK is unable to process the SDL access request via the SDL cache when the SDL cache does not store data requested by the control plane application.