WO2024131395A1 - Service performance measurement method and apparatus, and device, storage medium and program product - Google Patents

Abstract

The present disclosure relates to the technical field of mobile communications. Provided are a service performance measurement method and apparatus, and an electronic device, a computer-readable storage medium and a computer program product. The method comprises: acquiring a first time at which a functional network element receives a request of a target model providing service and a second time at which the target model providing service outputs a result, calculating the difference between the first time and the second time to obtain time length information; and generating the sum of the total number of times of model training and a target training duration according to the time length information and a training threshold, and obtaining an average model training duration according to the total number of times of model training and the total model training duration. The present disclosure can evaluate the service performance of the functional network element on the basis of quantified service performance measurement results including the total number of times of model training, the total model training duration or the average model training duration.

Description

服务性能测量方法、装置、设备、存储介质及程序产品Service performance measurement method, device, equipment, storage medium and program product

本公开基于申请号为202211644343.2、申请日为2022年12月20日、发明名称为《服务性能测量方法、装置、电子设备及存储介质》的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此引入本公开作为参考。This disclosure is based on the Chinese patent application with application number 202211644343.2, application date December 20, 2022, and invention name “Service Performance Measurement Method, Device, Electronic Device and Storage Medium”, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into this disclosure as a reference.

技术领域Technical Field

本公开涉及移动通信技术领域，尤其涉及一种服务性能测量方法、装置、电子设备、计算机可读存储介质及计算机程序产品。The present disclosure relates to the field of mobile communication technology, and in particular to a service performance measurement method, device, electronic device, computer-readable storage medium, and computer program product.

背景技术Background technique

在整个基于ML(Machine Learning，机器学习)的网络智能化实现框架下，模型训练时的开销是最为显著的，模型训练过程对功能网元性能的影响，要显著地高于功能网元提供的其他服务对功能网元性能的影响，获取功能网元是否触发了模型训练这一信息，是评估网络智能化框架中网元性能的重要依据，相关技术无法区别功能网元是否触发了模型训练过程，无法对功能网元触发的模型训练过程进行观测。In the entire network intelligence implementation framework based on ML (Machine Learning), the overhead of model training is the most significant. The impact of the model training process on the performance of functional network elements is significantly higher than the impact of other services provided by the functional network elements on the performance of functional network elements. Obtaining the information on whether the functional network elements have triggered model training is an important basis for evaluating the performance of network elements in the network intelligence framework. Related technologies cannot distinguish whether the functional network elements have triggered the model training process, and cannot observe the model training process triggered by the functional network elements.

发明内容Summary of the invention

本公开提供一种服务性能测量方法、装置、电子设备及计算机可读存储介质，至少在一定程度上克服相关技术中无法观测功能网元触发的模型训练信息的问题。The present disclosure provides a service performance measurement method, device, electronic device and computer-readable storage medium, which at least to a certain extent overcome the problem in related technologies that model training information triggered by functional network elements cannot be observed.

根据本公开的一个方面，提供一种服务性能测量方法，包括：According to one aspect of the present disclosure, a service performance measurement method is provided, comprising:

获取功能网元提供目标模型提供服务的时间长度信息，其中，所述目标模型提供服务是用于提供模型的服务；Acquire time length information of providing a target model service by a functional network element, wherein the target model service is a service for providing a model;

根据所述时间长度信息及训练门限，生成服务性能测量结果。A service performance measurement result is generated according to the time length information and the training threshold.

在本公开的一个实施例中，所述模型为机器学习ML模型。In one embodiment of the present disclosure, the model is a machine learning ML model.

在本公开的一个实施例中，所述服务性能测量结果包括以下至少之一：模型训练总次数、模型训练总时长、模型训练平均时长。In one embodiment of the present disclosure, the service performance measurement result includes at least one of the following: the total number of model training times, the total model training duration, and the average model training duration.

在本公开的一个实施例中，所述根据所述时间长度信息及训练门限，生成服务性能测量结果包括：获取一定时间内所述时间长度信息大于所述训练门限的次数，生成所述模型训练总次数。In one embodiment of the present disclosure, generating the service performance measurement result according to the time length information and the training threshold includes: obtaining the number of times the time length information is greater than the training threshold within a certain period of time, and generating the total number of model training times.

在本公开的一个实施例中，所述获取一定时间内所述时间长度信息大于所述训 练门限的次数，生成所述模型训练总次数包括：In one embodiment of the present disclosure, the time length information obtained within a certain period of time is greater than the training The number of training thresholds to generate the total number of model training times includes:

将计时器的时间长度调整为与所述训练门限一致；Adjusting the time length of the timer to be consistent with the training threshold;

当所述功能网元接收到所述目标模型提供服务的请求时，启动所述计时器进行计时；When the functional network element receives a request for service provided by the target model, the timer is started to count;

根据所述功能网元是否在所述计时器超时前提供结果的信息，生成所述模型训练总次数。The total number of model training times is generated according to whether the functional network element provides result information before the timer times out.

在本公开的一个实施例中，所述根据所述功能网元是否在所述计时器超时前提供结果的信息，生成所述模型训练总次数包括：In one embodiment of the present disclosure, generating the total number of model training times according to whether the functional network element provides the result information before the timer times out includes:

判断所述功能网元是否在所述计时器超时前提供结果；Determining whether the functional network element provides a result before the timer times out;

若是，则所述时间长度信息小于或等于所述训练门限，不计数；If yes, the time length information is less than or equal to the training threshold and is not counted;

否则，所述时间长度信息大于所述训练门限，计数；Otherwise, the time length information is greater than the training threshold, and counting is performed;

根据一定时间内所述时间长度信息大于所述训练门限的次数，生成所述模型训练总次数。The total number of model training times is generated according to the number of times the time length information is greater than the training threshold within a certain period of time.

在本公开的一个实施例中，所述根据所述功能网元是否在所述计时器超时前提供结果的信息，生成所述模型训练总次数包括：In one embodiment of the present disclosure, generating the total number of model training times according to whether the functional network element provides the result information before the timer times out includes:

判断所述功能网元是否在所述计时器超时前提供结果；Determining whether the functional network element provides a result before the timer times out;

若是，则所述时间长度信息小于所述训练门限，不计数；If yes, the time length information is less than the training threshold and is not counted;

否则，所述时间长度信息大于或等于所述训练门限，计数；Otherwise, the time length information is greater than or equal to the training threshold, and counts;

根据一定时间内所述时间长度信息大于或等于所述训练门限的次数，生成所述模型训练总次数。The total number of model training times is generated according to the number of times the time length information is greater than or equal to the training threshold within a certain period of time.

在本公开的一个实施例中，所述根据所述时间长度信息及训练门限，生成服务性能测量结果包括：In one embodiment of the present disclosure, generating a service performance measurement result according to the time length information and the training threshold includes:

获取一定时间内目标训练时长，其中，所述目标训练时长为大于或等于所述训练门限的时间长度信息；Obtaining a target training duration within a certain period of time, wherein the target training duration is time length information that is greater than or equal to the training threshold;

计算所述目标训练时长的总和，生成所述模型训练总时长。The sum of the target training durations is calculated to generate the total model training duration.

在本公开的一个实施例中，所述根据所述时间长度信息及训练门限，生成服务性能测量结果包括：In one embodiment of the present disclosure, generating a service performance measurement result according to the time length information and the training threshold includes:

根据所述模型训练总次数及所述模型训练总时长得到所述模型训练平均时长。The average duration of the model training is obtained according to the total number of model training times and the total duration of the model training.

在本公开的一个实施例中，所述获取功能网元提供目标模型提供服务的时间长度信息包括： In one embodiment of the present disclosure, obtaining information about the length of time that the functional network element provides services provided by the target model includes:

获取所述功能网元接收所述目标模型提供服务的请求的第一时间；Obtaining the first time when the functional network element receives a request for service provided by the target model;

获取所述请求对应的所述目标模型提供服务输出结果的第二时间；Obtaining a second time at which the target model corresponding to the request provides a service output result;

计算所述第一时间与所述第二时间的差值，得到所述时间长度信息。The difference between the first time and the second time is calculated to obtain the time length information.

在本公开的一个实施例中，所述目标模型提供服务包括用于指示如下内容的参数：提供模型、更新模型或重新训练模型。In one embodiment of the present disclosure, the target model providing service includes parameters for indicating the following contents: providing a model, updating a model, or retraining a model.

在本公开的一个实施例中，还包括：In one embodiment of the present disclosure, it further includes:

获取所述目标模型的类型标识；根据所述类型标识确定对应的分析类型过滤器；Obtaining a type identifier of the target model; determining a corresponding analysis type filter according to the type identifier;

通过所述分析类型过滤器得到所述目标模型提供服务的时间长度信息。The time length information of the service provided by the target model is obtained through the analysis type filter.

在本公开的一个实施例中，所述功能网元为网络数据分析功能网元。In one embodiment of the present disclosure, the functional network element is a network data analysis functional network element.

根据本公开的另一个方面，还提供了一种服务性能测量装置，包括：时间长度获取模块、测量结果生成模块；According to another aspect of the present disclosure, there is also provided a service performance measurement device, including: a time length acquisition module, a measurement result generation module;

时间长度获取模块，获取功能网元提供目标机器学习模型提供服务的时间长度信息，其中，所述目标模型提供服务是用于提供模型的服务；A time length acquisition module is used to acquire time length information of a target machine learning model service provided by a functional network element, wherein the target model service is a service for providing a model;

测量结果生成模块，根据所述时间长度信息及训练门限，生成服务性能测量结果。The measurement result generating module generates a service performance measurement result according to the time length information and the training threshold.

根据本公开的另一个方面，还提供了一种电子设备，包括：处理器；以及存储器，用于存储所述处理器的可执行指令；其中，所述处理器配置为经由执行所述可执行指令来执行上述任意一项所述服务性能测量方法。According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned service performance measurement methods by executing the executable instructions.

根据本公开的另一个方面，还提供了一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现上述任意一项所述的服务性能测量方法。According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the service performance measurement method described in any one of the above is implemented.

根据本公开的另一个方面，还提供了一种计算机程序产品，包括计算机程序，所述计算机程序被处理器执行时实现上述的服务性能测量方法。According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the above-mentioned service performance measurement method is implemented.

应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，并不能限制本公开。It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

附图说明BRIEF DESCRIPTION OF THE DRAWINGS

此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本公开的实施例，并与说明书一起用于解释本公开的原理。显而易见地，下面描述中的附图仅仅是本公开的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。 The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

图1示出本公开实施例中一种服务性能测量方法流程图；FIG1 shows a flow chart of a service performance measurement method in an embodiment of the present disclosure;

图2示出本公开实施例中一种时间长度信息获取方法流程图；FIG2 shows a flow chart of a method for obtaining time length information in an embodiment of the present disclosure;

图3示出本公开实施例中一种模型训练总次数生成方法流程图；FIG3 shows a flow chart of a method for generating the total number of model training times in an embodiment of the present disclosure;

图4示出本公开实施例中一种NWDAF分析模型提供服务的性能测量方法流程图；FIG4 shows a flow chart of a method for measuring performance provided by a NWDAF analysis model in an embodiment of the present disclosure;

图5示出本公开实施例中一种NWDAF的服务性能测量结果生成方法流程图；FIG5 shows a flow chart of a method for generating a service performance measurement result of NWDAF in an embodiment of the present disclosure;

图6示出本公开实施例中一种服务性能测量装置示意图；FIG6 shows a schematic diagram of a service performance measurement device according to an embodiment of the present disclosure;

图7示出本公开实施例中一种电子设备的结构框图。FIG. 7 shows a structural block diagram of an electronic device in an embodiment of the present disclosure.

具体实施方式Detailed ways

现在将参考附图更全面地描述示例实施方式。然而，示例实施方式能够以多种形式实施，且不应被理解为限于在此阐述的范例；相反，提供这些实施方式使得本公开将更加全面和完整，并将示例实施方式的构思全面地传达给本领域的技术人员。所描述的特征、结构或特性可以以任何合适的方式结合在一个或更多实施方式中。Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be more comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

此外，附图仅为本公开的示意性图解，并非一定是按比例绘制。图中相同的附图标记表示相同或类似的部分，因而将省略对它们的重复描述。附图中所示的一些方框图是功能实体，不一定必须与物理或逻辑上独立的实体相对应。可以采用软件形式来实现这些功能实体，或在一个或多个硬件模块或集成电路中实现这些功能实体，或在不同网络和/或处理器装置和/或微控制器装置中实现这些功能实体。In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and/or processor devices and/or microcontroller devices.

为了便于理解，下面首先对本公开涉及到的几个名词进行解释如下：For ease of understanding, several terms involved in the present disclosure are explained below:

NWDAF(Network Data Analytics Function，网络数据分析功能)是5GC中的网元，支持从NFs、AFs以及OAM收集数据，并向NFs、AFs以及OAM提供分析信息，NWDAF可以提供分析服务、数据管理服务以及模型相关的服务等。NWDAF (Network Data Analytics Function) is a network element in 5GC, which supports collecting data from NFs, AFs and OAM, and providing analysis information to NFs, AFs and OAM. NWDAF can provide analysis services, data management services and model-related services.

MDAS(Management Data Analytics Service，管理数据分析服务)提供不同网络相关参数的数据分析等。MDAS (Management Data Analytics Service) provides data analysis of different network-related parameters.

VNFC(Virtualized Network Function Component，虚拟化网络功能组件)是VNF(Virtualized Network Function，虚拟化网络功能层)的组件，用于完成VNF的某项特定功能。VNFC (Virtualized Network Function Component) is a component of VNF (Virtualized Network Function) and is used to complete a specific function of VNF.

ML(Machine Leaming，机器学习)通过研究算法和统计模型，让计算机***使用这些算法和统计模型，依靠模式和推理来执行特定的任务，能基于样本数据建立数学模型，以便在没有被明确编程的情况下作出预测或决策。 ML (Machine Leaming) studies algorithms and statistical models, allowing computer systems to use these algorithms and statistical models to perform specific tasks based on patterns and reasoning. It can build mathematical models based on sample data to make predictions or decisions without being explicitly programmed.

下面结合附图及实施例对本示例实施方式进行详细说明。The present exemplary implementation is described in detail below with reference to the accompanying drawings and embodiments.

首先，本公开实施例中提供了一种服务性能测量方法，该方法可以由任意具备计算处理能力的电子设备执行。First, a service performance measurement method is provided in an embodiment of the present disclosure. The method can be executed by any electronic device with computing and processing capabilities.

图1示出本公开实施例中一种服务性能测量方法流程图，如图1所示，本公开实施例中提供的服务性能测量方法包括如下步骤：FIG. 1 shows a flow chart of a service performance measurement method in an embodiment of the present disclosure. As shown in FIG. 1 , the service performance measurement method provided in an embodiment of the present disclosure includes the following steps:

S102，获取功能网元提供目标模型提供服务的时间长度信息；S102, obtaining information on the length of time the functional network element provides services for the target model;

在一个实施例中，功能网元是可以提供模型提供服务的网元，功能网元包括但不限于网络数据分析功能网元、MDAS。In one embodiment, the functional network element is a network element that can provide model services, and the functional network element includes but is not limited to a network data analysis functional network element and an MDAS.

网络数据分析功能网元作为5GC(5G核心网)中大数据采集和智能分析的独立网元，具备能力标准化、汇聚网络数据、实时性更高、支持闭环可控等特点。As an independent network element for big data collection and intelligent analysis in 5GC (5G core network), the network data analysis functional network element has the characteristics of standardized capabilities, aggregation of network data, higher real-time performance, and support for closed-loop control.

MDAS(Management Data Analytics Service，管理数据分析服务)提供不同网络相关参数的数据分析等。MDAS (Management Data Analytics Service) provides data analysis of different network-related parameters.

在一个实施例中，模型可以是机器学习ML模型，目标ML模型提供服务是用于提供ML模型的服务ML Model Provisioning，用于提供ML模型的服务；目标ML模型提供服务包括但不限于用于指示如下内容的参数：提供模型、更新模型或重新训练模型。In one embodiment, the model may be a machine learning (ML) model, and the target ML model provisioning service is a service for providing ML models, ML Model Provisioning, which is a service for providing ML models; the target ML model provisioning service includes but is not limited to parameters for indicating: providing a model, updating a model, or retraining a model.

时间长度信息为待测的功能实体提供每个目标ML模型提供服务输出结果的时间消耗，可通过获取功能网元接收ML模型提供服务的服务请求或订阅请求的时间、与该服务请求对应的响应时间、与该订阅请求对应的订阅通知时间来计算时间长度信息。The time length information provides the time consumption of each target ML model to provide the service output results for the functional entity to be tested. The time length information can be calculated by obtaining the time when the functional network element receives the service request or subscription request provided by the ML model, the response time corresponding to the service request, and the subscription notification time corresponding to the subscription request.

在一个实施例中，目标ML模型服务不包含提供服务结果的时间或者时延要求。In one embodiment, the target ML model service does not include time or latency requirements for providing service results.

S104，根据时间长度信息及训练门限，生成服务性能测量结果。S104: Generate a service performance measurement result according to the time length information and the training threshold.

ML模型训练过程对功能网元性能的影响，要显著地高于功能网元提供的其他服务对NWDAF性能的影响；训练门限为判断ML模型的训练是否发生的阈值，在一个实施例中，训练门限可根据用户需要进行设置，也可以根据ML模型历史数据进行设置，对比不作限制。The impact of the ML model training process on the performance of the functional network element is significantly higher than the impact of other services provided by the functional network element on the NWDAF performance; the training threshold is a threshold for determining whether the training of the ML model occurs. In one embodiment, the training threshold can be set according to user needs or according to the historical data of the ML model, and there is no restriction on comparison.

服务性能测量结果是根据ML模型训练信息生成的用于评估功能网元性能的数据，服务性能测量结果能识别功能网元是否触发模型训练过程或事件，并进一步能针对功能网元触发的模型训练过程的性能指标测量。The service performance measurement results are data generated based on the ML model training information for evaluating the performance of functional network elements. The service performance measurement results can identify whether the functional network element triggers the model training process or event, and can further measure the performance indicators of the model training process triggered by the functional network element.

在一个实施例中，服务性能测量结果包括以下至少之一：模型训练总次数、模 型训练总时长、模型训练平均时长等。In one embodiment, the service performance measurement result includes at least one of the following: total number of model training times, model The total training time and average training time of the model are as follows.

在一个实施例中，可根据需要用户设置等于训练门限是否计数，模型训练总次数生成方法包括：获取一定时间内时间长度信息大于训练门限，生成模型训练总次数；模型训练总次数生成方法也可包括：获取一定时间内时间长度信息大于或等于训练门限，生成模型训练总次数。In one embodiment, the user can set whether to count when the training threshold is equal to the required value. The method for generating the total number of model training times includes: obtaining time length information within a certain period of time greater than the training threshold, and generating the total number of model training times; the method for generating the total number of model training times may also include: obtaining time length information within a certain period of time greater than or equal to the training threshold, and generating the total number of model training times.

在一个实施例中，模型训练总时长生成方法包括：获取一定时间内目标训练时长，其中，目标训练时长为大于或等于训练门限的时间长度信息。In one embodiment, a method for generating a total model training duration includes: obtaining a target training duration within a certain period of time, wherein the target training duration is time length information that is greater than or equal to a training threshold.

在一个实施例中，获取一定时间内时间长度信息大于训练门限的次数，生成模型训练总次数中，对应的目标训练时长为大于训练门限的时间长度信息，并计算目标训练时长的总和，生成对应的模型训练总时长，并根据该模型训练总时长与模型训练总次数生成对应的模型训练平均时长。In one embodiment, the number of times that the time length information is greater than the training threshold within a certain period of time is obtained, and the total number of model training times is generated. The corresponding target training time is the time length information greater than the training threshold, and the sum of the target training times is calculated to generate the corresponding total model training time, and the corresponding average model training time is generated based on the total model training time and the total number of model training times.

在一个实施例中，获取一定时间内时间长度信息大于或等于训练门限的次数，生成模型训练总次数中，对应的目标训练时长为大于或等于训练门限的时间长度信息，并计算目标训练时长的总和，生成对应的模型训练总时长，并根据该模型训练总时长与模型训练总次数生成对应的模型训练平均时长。In one embodiment, the number of times that the time length information is greater than or equal to the training threshold within a certain period of time is obtained, and the total number of model training times is generated, the corresponding target training time is the time length information greater than or equal to the training threshold, and the sum of the target training times is calculated to generate the corresponding total model training time, and the corresponding average model training time is generated based on the total model training time and the total number of model training times.

在一个实施例中，模型训练总次数、模型训练总时长或模型训练平均时长可以根据权重计算获得。In one embodiment, the total number of model training times, the total model training duration, or the average model training duration can be obtained based on weight calculation.

在一个实施例中，本公开实施例以时间长度信息大于训练门限，生成模型训练总次数为例进行介绍；获取一定时间内大于训练门限的时间长度信息，设置多个阈值生成多个阈值区间，多个阈值区间与多个权重系数一一对应，获取在每个阈值区间的时间长度信息次数，计算该次数与对应的权重系数的乘积生成每个阈值区间的次数，每个阈值区间的次数的和为根据权重计算的模型训练总次数。In one embodiment, the disclosed embodiment takes the time length information greater than the training threshold as an example to generate the total number of model training times; the time length information greater than the training threshold within a certain period of time is obtained, multiple thresholds are set to generate multiple threshold intervals, the multiple threshold intervals correspond one-to-one to multiple weight coefficients, the number of time length information in each threshold interval is obtained, the product of the number and the corresponding weight coefficient is calculated to generate the number of each threshold interval, and the sum of the number of each threshold interval is the total number of model training calculated according to the weight.

在一个实施例中，以设置三个阈值为例，即第一阈值、第二阈值及第三阈值，第三阈值大于第二阈值，第二阈值大于第一阈值；并设置三个阈值生成三个阈值区间，对应的三个权重系数，即第一权重系数、第二权重系数及第三权重系数，三个权重系数的总和为1，获取一定时间内时间长度信息在第一阈值区间的次数，即时间长度信息大于第一阈值，且时间长度信息小于或等于第二阈值的次数得到第一次数，获取一定时间内时间长度信息在第二阈值区间的次数，即时间长度信息大于第二阈值，且时间长度信息小于或等于第三阈值的次数得到第二次数，获取一定时间内时间长度信息在第三阈值区间的次数，即时间长度信息大于第三阈值的次数得到第三 次数，根据权重计算得到的模型训练总次数为第一次数与第一权重系数乘积、第二次数与第二权重系数乘积、第三次数与第三权重系数乘积的和。In one embodiment, taking setting three thresholds as an example, namely, a first threshold, a second threshold and a third threshold, the third threshold is greater than the second threshold, and the second threshold is greater than the first threshold; and setting the three thresholds to generate three threshold intervals, corresponding to three weight coefficients, namely, a first weight coefficient, a second weight coefficient and a third weight coefficient, the sum of the three weight coefficients is 1, obtaining the number of times the time length information is in the first threshold interval within a certain period of time, namely, the number of times the time length information is greater than the first threshold and the number of times the time length information is less than or equal to the second threshold, obtaining the first number, obtaining the number of times the time length information is in the second threshold interval within a certain period of time, namely, the number of times the time length information is greater than the second threshold and the number of times the time length information is less than or equal to the third threshold, obtaining the second number, obtaining the number of times the time length information is in the third threshold interval within a certain period of time, namely, the number of times the time length information is greater than the third threshold, obtaining the third number. The total number of model training times calculated according to the weights is the sum of the product of the first number and the first weight coefficient, the product of the second number and the second weight coefficient, and the product of the third number and the third weight coefficient.

在一个实施例中，获取一定时间内大于训练门限的时间长度信息，设置多个阈值生成多个阈值区间，多个阈值区间与多个权重系数一一对应，获取在每个阈值区间的目标训练时长，计算该目标训练时长与对应的权重系数的乘积生成每个阈值区间的目标训练时长，每个阈值区间的目标训练时长的和为根据权重计算的模型训练总时长。In one embodiment, information on the length of time greater than a training threshold within a certain period of time is obtained, multiple thresholds are set to generate multiple threshold intervals, the multiple threshold intervals correspond one-to-one to multiple weight coefficients, the target training duration in each threshold interval is obtained, the product of the target training duration and the corresponding weight coefficient is calculated to generate the target training duration for each threshold interval, and the sum of the target training duration for each threshold interval is the total model training duration calculated according to the weight.

在一个实施例中，以设置三个阈值为例，即第一阈值、第二阈值及第三阈值，第三阈值大于第二阈值，第二阈值大于第一阈值；并设置三个阈值生成三个阈值区间，对应的三个权重系数，即第一权重系数、第二权重系数及第三权重系数，三个权重系数的总和为1，获取一定时间内在第一阈值区间的第一目标训练时长，即第一目标训练时长大于第一阈值，且第一目标训练时长小于或等于第二阈值；获取一定时间内在第二阈值区间的第二目标训练时长，即第二目标训练时长大于第二阈值，且第二目标训练时长小于或等于第三阈值；获取一定时间内在第三阈值区间的第三目标训练时长，即第三目标训练时长大于第三阈值，根据权重计算得到的模型训练总时长为第一目标训练时长与第一权重系数乘积、第一目标训练时长与第二权重系数乘积、第一目标训练时长与第三权重系数乘积的和。In one embodiment, taking setting three thresholds as an example, namely, a first threshold, a second threshold and a third threshold, the third threshold is greater than the second threshold, and the second threshold is greater than the first threshold; and setting the three thresholds to generate three threshold intervals, corresponding to three weight coefficients, namely, a first weight coefficient, a second weight coefficient and a third weight coefficient, the sum of the three weight coefficients is 1, and obtaining a first target training duration in the first threshold interval within a certain period of time, namely, the first target training duration is greater than the first threshold, and the first target training duration is less than or equal to the second threshold; obtaining a second target training duration in the second threshold interval within a certain period of time, namely, the second target training duration is greater than the second threshold, and the second target training duration is less than or equal to the third threshold; obtaining a third target training duration in the third threshold interval within a certain period of time, namely, the third target training duration is greater than the third threshold, and the total model training duration calculated according to the weight is the sum of the product of the first target training duration and the first weight coefficient, the product of the first target training duration and the second weight coefficient, and the product of the first target training duration and the third weight coefficient.

在一个实施例中，根据权重计算得到的模型训练总次数及模型训练总时长，得到根据权重计算得到的模型训练平均时长。In one embodiment, the total number of model training times and the total model training duration calculated according to the weights are used to obtain the average model training duration calculated according to the weights.

在一个实施例中，通过服务性能测量结果可以反映实现ML模型训练功能的VNFC(Virtualized Network Function Component，虚拟化网络功能组件)实例的性能，也可以反映功能网元的性能，评估该VNFC的能量效率。In one embodiment, the service performance measurement results can reflect the performance of the VNFC (Virtualized Network Function Component) instance that implements the ML model training function, and can also reflect the performance of the functional network element and evaluate the energy efficiency of the VNFC.

相关技术中，对于功能网元内部的事件的观测，通常的做法是设计特定的服务接口，通过内部记录事件以及服务化接口，对外部进行事件记录的开放，对于服务时间的观测，普遍用于测量服务提供者的时延性能。In the related technology, for the observation of events inside functional network elements, the usual practice is to design a specific service interface, open the event record to the outside through internal event recording and service-oriented interface. The observation of service time is generally used to measure the latency performance of the service provider.

上述实施例中，获取功能网元提供ML模型相关的服务的时间长度信息，根据上述时间长度信息是否达到设定的训练门限，判断模型训练的发生，量化模型训练对功能网元性能的影响，而不需要对外发出事件提醒，通过模型训练总次数、模型训练总时长、模型训练平均时长任一个或多个的组合，作为待测功能网元的服务性能测量结果评估功能网元等的性能，不需要功能网元主动上报就能观测功能网元内 部处理事件。In the above embodiment, the time length information of the functional network element providing the service related to the ML model is obtained, and the occurrence of model training is judged according to whether the above time length information reaches the set training threshold, and the impact of model training on the performance of the functional network element is quantified without issuing event reminders to the outside. The total number of model training times, the total model training time, and the average model training time are used as any one or more combinations of the service performance measurement results of the functional network element to be tested to evaluate the performance of the functional network element, etc., without the functional network element actively reporting, and the internal performance of the functional network element can be observed. Department handles events.

图2示出本公开实施例中一种时间长度信息获取方法流程图，如图2所示，本公开实施例中提供的时间长度信息获取方法包括如下步骤：FIG2 shows a flow chart of a method for obtaining time length information in an embodiment of the present disclosure. As shown in FIG2 , the method for obtaining time length information provided in an embodiment of the present disclosure includes the following steps:

S202，获取功能网元接收目标模型提供服务的请求的第一时间；S202, obtaining the first time when the functional network element receives a request for a service provided by a target model;

目标模型提供服务为功能网元消费者发送的模型提供服务；在一个实施例中，请求包括但不限于订阅请求或服务请求，请求包括但不限于：功能网元消费者标识、功能网元标识、目标模型类型标识等。The target model provides services for the model sent by the functional network element consumer; in one embodiment, the request includes but is not limited to a subscription request or a service request, and the request includes but is not limited to: functional network element consumer identifier, functional network element identifier, target model type identifier, etc.

在一个实施例中，功能网元接收功能网元消费者发送的订阅请求或服务请求，获取功能网元接收目标模型提供服务的订阅请求或服务请求的第一时间。In one embodiment, the functional network element receives a subscription request or a service request sent by a functional network element consumer, and obtains the first time when the functional network element receives the subscription request or the service request for the service provided by the target model.

S204，获取请求对应的目标模型提供服务输出结果的第二时间；S204, obtaining a second time at which the target model corresponding to the request provides a service output result;

在一个实施例中，获取功能网元接收功能网元消费者发送的订阅请求后发送订阅通知的第二时间，或获取功能网元接收功能网元消费者发送的服务请求后发送响应的第二时间。In one embodiment, the second time at which the functional network element sends a subscription notification after receiving a subscription request sent by the functional network element consumer is obtained, or the second time at which the functional network element sends a response after receiving a service request sent by the functional network element consumer is obtained.

S206，计算第一时间与第二时间的差值，得到时间长度信息。S206, calculating the difference between the first time and the second time to obtain time length information.

在一个实施例中，获取目标模型的类型标识；根据类型标识确定对应的分析类型过滤器；通过分析类型过滤器得到目标模型提供服务的时间长度信息。In one embodiment, a type identifier of a target model is obtained; a corresponding analysis type filter is determined according to the type identifier; and time length information of services provided by the target model is obtained through the analysis type filter.

在一个实施例中，获取与待测功能网元相关的时间长度信息，根据待测功能网元提供模型服务的类型标识分别进行聚类分析，生成与类型标识对应的分析类型过滤器。In one embodiment, the time length information related to the functional network element to be tested is obtained, cluster analysis is performed according to the type identification of the model service provided by the functional network element to be tested, and an analysis type filter corresponding to the type identification is generated.

在一个实施例中，待测的待测功能网元接收的针对特定类型的模型提供服务的服务请求或订阅请求，其中，服务请求或订阅请求包括模型的类型标识，测量待测的待测功能网元提供每个当前模型提供服务的时间长度信息，将多个时间长度信息根据类型标识进行聚类分析，生成与类型标识对应的分析类型过滤器。In one embodiment, the functional network element to be tested receives a service request or a subscription request for providing services for a specific type of model, wherein the service request or the subscription request includes a type identifier of the model, measures the time length information for providing services for each current model provided by the functional network element to be tested, performs cluster analysis on multiple time length information according to the type identifier, and generates an analysis type filter corresponding to the type identifier.

在一个实施例中，分析类型过滤器可以周期进行更新。In one embodiment, the analysis type filter may be updated periodically.

上述实施例中，网络功能实体或实例的管理，依赖于网络功能实体或实例的性能观测和评价，可通过功能网元与功能网元消费者的交互信息，得到时间长度信息，NWDAF在向服务消费者提供分析模型的时候，根据时间长度信息可以自行判断是否需要对模型进行更新，也即是否需要进行模型训练，而不需要对外发出事件提醒。In the above embodiments, the management of network function entities or instances depends on the performance observation and evaluation of the network function entities or instances. The time length information can be obtained through the interaction information between the functional network element and the functional network element consumer. When NWDAF provides the analysis model to the service consumer, it can judge whether the model needs to be updated, that is, whether model training is needed, based on the time length information, without issuing event reminders to the outside.

图3示出本公开实施例中一种模型训练总次数生成方法流程图，如图3所示，本公开实施例中提供的模型训练总次数生成方法包括如下步骤： FIG3 shows a flow chart of a method for generating the total number of model training times in an embodiment of the present disclosure. As shown in FIG3 , the method for generating the total number of model training times provided in an embodiment of the present disclosure includes the following steps:

S302，将计时器的时间长度调整为与训练门限一致。S302, adjusting the time length of the timer to be consistent with the training threshold.

例如，训练门限为1分钟，则计时器的时间长度调整为1分钟。For example, if the training threshold is 1 minute, the time length of the timer is adjusted to 1 minute.

S304，当功能网元接收到目标模型提供服务的请求时，启动计时器进行计时；S304, when the functional network element receives a request for service provided by the target model, it starts a timer for timing;

S306，判断功能网元是否在计时器超时前提供结果；S306, determining whether the functional network element provides a result before the timer times out;

S308，若是，则时间长度信息小于或等于训练门限，不计数；S308, if yes, the time length information is less than or equal to the training threshold and is not counted;

S310，否则，时间长度信息大于训练门限，计数；S310, otherwise, the time length information is greater than the training threshold, and count;

S312，根据一定时间内时间长度信息大于训练门限的次数，生成模型训练总次数。S312, generating the total number of model training times according to the number of times the time length information is greater than the training threshold within a certain period of time.

在一个实施例中，可根据用户需要设置时间长度信息等于训练门限时是否计数，本公开实施例以时间长度信息大于训练门限计数为例进行介绍。In one embodiment, whether to count when the time length information is equal to the training threshold can be set according to user needs. The embodiment of the present disclosure takes the example of counting when the time length information is greater than the training threshold as an example.

在一个实施例中，以计时器的时间长度为1分钟为例，在一定时间内，当功能网元接收到第一个目标模型提供服务的服务请求或订阅请求时，启动计时器进行计时，当功能网元在1分钟内提供请求结果或订阅结果，则不计数；当功能网元接收到第二个目标模型提供服务的服务请求或订阅请求时，启动计时器进行计时，当功能网元在1分钟内未提供请求结果或订阅结果，则通过计数器计数为1；功能网元接收到第三个目标模型提供服务的服务请求或订阅请求时，启动计时器进行计时，当功能网元在1分钟内未提供请求结果或订阅结果，则通过计数器计数为2；因而，在该时间范围内时间长度信息大于训练门限的次数为2，模型训练总次数为2。In one embodiment, taking the time length of a timer of 1 minute as an example, within a certain period of time, when the functional network element receives a service request or subscription request for service provided by the first target model, the timer is started for timing, and when the functional network element provides the request result or subscription result within 1 minute, the count is not made; when the functional network element receives a service request or subscription request for service provided by the second target model, the timer is started for timing, and when the functional network element does not provide the request result or subscription result within 1 minute, the count through the counter is 1; when the functional network element receives a service request or subscription request for service provided by the third target model, the timer is started for timing, and when the functional network element does not provide the request result or subscription result within 1 minute, the count through the counter is 2; therefore, the number of times the time length information is greater than the training threshold within this time range is 2, and the total number of model training times is 2.

上述实施例中，根据上述时间长度信息是否达到设定的训练门限，判断模型训练的发生，量化模型训练对功能网元性能的影响，而不需要对外发出事件提醒，通过模型训练总次数作为待测功能网元的服务性能测量结果评估功能网元等的性能，不需要功能网元主动上报就能观测功能网元内部处理事件。In the above embodiment, the occurrence of model training is judged based on whether the above time length information reaches the set training threshold, and the impact of model training on the performance of functional network elements is quantified without issuing event reminders to the outside. The total number of model training times is used as the service performance measurement result of the functional network element to be tested to evaluate the performance of the functional network element, and the internal processing events of the functional network element can be observed without the functional network element actively reporting.

图4示出本公开实施例中一种NWDAF分析模型提供服务的性能测量方法流程图，如图4所示，本公开实施例中提供的NWDAF分析模型提供服务的性能测量方法包括如下步骤：FIG4 shows a flow chart of a performance measurement method for services provided by a NWDAF analysis model in an embodiment of the present disclosure. As shown in FIG4 , the performance measurement method for services provided by a NWDAF analysis model in an embodiment of the present disclosure includes the following steps:

Rel-15开始，3GPP(3rd Generation Partnership Project，第三代合作伙伴计划)在核心网中引入了NWDAF(Network Data Analytics Function，网络数据分析功能)进行网络数据分析；3GPP Rel-17对NWDAF的架构进行了增强，其中之一是通过Logical Decomposition(逻辑分解)，允许NWDAF在提供分析服务的同时，也可以通过服务化接口，提供ML模型。 Starting from Rel-15, 3GPP (3rd Generation Partnership Project) introduced NWDAF (Network Data Analytics Function) in the core network for network data analysis; 3GPP Rel-17 enhanced the architecture of NWDAF, one of which was through Logical Decomposition, allowing NWDAF to provide ML models through service-oriented interfaces while providing analysis services.

S402，测量待测的NWDAF实例接收的针对特定类型的ML模型提供服务的服务请求或订阅请求的时间t₁；S402, measuring the time t ₁ of a service request or a subscription request for providing a service for a specific type of ML model received by the NWDAF instance to be tested;

S404，测量与S102中每个服务请求或订阅请求对应的NWDAF提供ML模型输出结果的时间t₂；S404, measuring the time t ₂ for the NWDAF corresponding to each service request or subscription request in S102 to provide the ML model output result;

在一个实施例中，测量NWDAF实例接收ML模型提供服务的服务请求t₁，测量与该服务请求对应的NWDAF提供ML模型响应的时间t₂。In one embodiment, the time t ₁ when the NWDAF instance receives a service request provided by the ML model is measured, and the time t ₂ when the NWDAF provides a ML model response corresponding to the service request is measured.

在一个实施例中，测量NWDAF实例接收ML模型提供服务的订阅请求t₁，测量与该订阅请求对应的NWDAF提供ML模型订阅通知的时间t₂。In one embodiment, the time t ₁ when the NWDAF instance receives a subscription request for a service provided by the ML model is measured, and the time t ₂ when the NWDAF instance provides a subscription notification for the ML model corresponding to the subscription request is measured.

S406，计算t_opt＝t₂-t₁，得到待测的NWDAF实例提供每个当前ML模型提供服务的时间长度信息，即ML模型输出结果的时间消耗；S406, calculate _topt = _t2 - _t1 , and obtain the time length information of the service provided by the NWDAF instance to be tested for each current ML model, that is, the time consumption of the ML model output result;

S408，针对特定NWDAF实例或特定的ML模型类型，生成分析类型过滤器。S408 , generating an analysis type filter for a specific NWDAF instance or a specific ML model type.

不同的NWDAF实例的资源配置不同，同时，不同NWDAF实例能够提供的分析模型种类，以及模型的更新状态也不相同。因此，所生成的分析类型过滤器，需要同时考虑上述因素，才有可能得到准确的结果。Different NWDAF instances have different resource configurations. At the same time, different NWDAF instances can provide different types of analysis models and model update status. Therefore, the generated analysis type filter needs to consider the above factors at the same time to obtain accurate results.

在一个实施例中，首先基于在特定NWDAF实例记录t_opt；其次，按照该NWDAF实例提供的分析服务的类型标识的数量进行聚类分析；最后，生成与类型标识对应的分析类型过滤器。In one embodiment, first, based on the _topt recorded in a specific NWDAF instance; second, cluster analysis is performed according to the number of type identifiers of analysis services provided by the NWDAF instance; and finally, an analysis type filter corresponding to the type identifier is generated.

在一个实施例中，待测的NWDAF实例接收的针对特定类型的ML模型提供服务的服务请求或订阅请求，其中，服务请求或订阅请求包括ML模型的类型标识，测量待测的NWDAF实例提供每个当前ML模型提供服务的时间长度信息，将多个时间长度信息根据类型标识进行聚类分析，生成与类型标识对应的分析类型过滤器。In one embodiment, the NWDAF instance to be tested receives a service request or a subscription request for providing services for a specific type of ML model, wherein the service request or the subscription request includes a type identifier of the ML model, measures the time length information of the service provided by each current ML model by the NWDAF instance to be tested, performs cluster analysis on multiple time length information according to the type identifier, and generates an analysis type filter corresponding to the type identifier.

在一个实施例中，分析类型过滤器可以周期进行更新，不需要在每次判断的时候重新生成；分析类型过滤器的更新周期可以根据用户需要进行设置，例如，用户可设置1小时更新分析类型过滤器；分析类型过滤器的更新周期也可以根据历史数据进行设置，例如，获取关于分析类型过滤器的历史数据，该分析类型过滤器已输入500个时间长度信息则更新分析类型过滤器。In one embodiment, the analysis type filter can be updated periodically and does not need to be regenerated every time a judgment is made; the update period of the analysis type filter can be set according to user needs, for example, the user can set the update of the analysis type filter to 1 hour; the update period of the analysis type filter can also be set according to historical data, for example, historical data about the analysis type filter is obtained, and if the analysis type filter has entered 500 time length information, the analysis type filter is updated.

S410，根据当前NWDAF实例收到的服务请求或订阅请求中的类型标识，将t_opt送入特定的分析类型过滤器，判断当前NWDAF是否触发了ML模型的训练过程。S410: According to the type identifier in the service request or subscription request received by the current NWDAF instance, the _topt is sent to a specific analysis type filter to determine whether the current NWDAF triggers the training process of the ML model.

在一个实施例中，在针对每个类型标识生成的分析类型过滤器以t_opt作为输入， 将输出分为两类，分别代表是否发生ML模型训练过程；例如，在一定时间内，输入的所有t_opt均小于训练门限，则没有发生ML模型训练过程，在一定时间内，输入的一个或多个t_opt均小于训练门限，则发生ML模型训练过程。In one embodiment, the analysis type filter generated for each type identifier takes _topt as input. The output is divided into two categories, representing whether the ML model training process occurs. For example, if all _topts input within a certain period of time are less than the training threshold, the ML model training process does not occur. If one or more _topts input within a certain period of time are less than the training threshold, the ML model training process occurs.

S412，基于这一方法得到的服务性能测量结果，可以进一步应用于一系列的有关NWDAF性能，NWDAF提供的ML模型性能的评估中，结合在NWDAF中配置的重训练触发条件，可以提供对NWDAF所提供分析服务的泛化性能进行观测和量化，用于进一步调整和优化NWDAF提供分析服务的效果，如准确性，一致性，连续性等。S412, the service performance measurement results obtained based on this method can be further applied to a series of evaluations related to NWDAF performance and the performance of the ML model provided by NWDAF. Combined with the retraining trigger conditions configured in NWDAF, it can provide observation and quantification of the generalization performance of the analysis service provided by NWDAF, which can be used to further adjust and optimize the effect of the analysis service provided by NWDAF, such as accuracy, consistency, continuity, etc.

虚拟化网络功能组件(Virtualized Network Function Component，VNFC)是VNF(Virtualized Network Function，虚拟化网络功能层)的组件，一个或多个VNFC组成一个VNF，用于完成VNF的某项特定功能；功能网元是指基于VNF实现的网元，一个或多个VNF构成一个功能网元；功能网元包括但不限于NWDAF等网元。Virtualized Network Function Component (VNFC) is a component of VNF (Virtualized Network Function). One or more VNFCs constitute a VNF, which is used to complete a specific function of the VNF. Functional network element refers to a network element implemented based on VNF. One or more VNFs constitute a functional network element. Functional network elements include but are not limited to network elements such as NWDAF.

VNF实例化的过程都需要获取每个VNFC的规格，然后以VNFC为单位，依次按序向虚拟化云平台发起实例化请求，虚拟化云平台根据每个VNFC启动资源的需求及本地资源视图情况，选择合适的计算节点进行单个VNFC的实例化。The VNF instantiation process requires obtaining the specifications of each VNFC, and then initiating instantiation requests to the virtualization cloud platform in sequence, taking VNFC as the unit. The virtualization cloud platform selects the appropriate computing node to instantiate a single VNFC based on the requirements of each VNFC startup resource and the local resource view.

在一个实施例中，通过服务性能测量结果可以反映NWDAF的性能，也可以反映实现ML模型训练功能的VNFC实例的性能，评估该VNFC的能量效率。In one embodiment, the service performance measurement result can reflect the performance of the NWDAF, and can also reflect the performance of the VNFC instance that implements the ML model training function, and evaluate the energy efficiency of the VNFC.

上述实施例中，模型训练过程对NWDAF性能的影响，要显著地高于NWDAF提供的其他服务对NWDAF性能的影响，在不改动NWDAF服务框架的情况下，通过观测NWDAF的服务类型，服务参数和服务相应触发时间的数据，实现对NWDAF内部处理过程或模型训练事件的测量，扩展了对NWDAF行为的观测能力及测量能力，除了具备通常意义上的观测NWDAF时延性能的能力外，不需要NWDAF主动上报，从而为准确估计NWDAF性能提供了NWDAF触发模型训练的信息。In the above embodiment, the impact of the model training process on the performance of NWDAF is significantly higher than the impact of other services provided by NWDAF on the performance of NWDAF. Without changing the NWDAF service framework, the measurement of the internal processing process of NWDAF or the model training event is achieved by observing the service type, service parameters and service corresponding trigger time data of NWDAF, thereby expanding the observation and measurement capabilities of NWDAF behavior. In addition to the ability to observe the NWDAF delay performance in the usual sense, NWDAF does not need to actively report, thereby providing information on NWDAF triggering model training for accurate estimation of NWDAF performance.

图5示出本公开实施例中一种NWDAF的服务性能测量结果生成方法流程图，如图5所示，本公开实施例中提供的NWDAF的服务性能测量结果生成方法包括如下步骤：FIG5 shows a flow chart of a method for generating a service performance measurement result of NWDAF in an embodiment of the present disclosure. As shown in FIG5 , the method for generating a service performance measurement result of NWDAF provided in an embodiment of the present disclosure includes the following steps:

在某一个时间点t₀，及一个时间窗长度L，通过上述参数和所提的方法，可以得到NWDAF的如下性能指标：At a certain time point t ₀ and a time window length L, the following performance indicators of NWDAF can be obtained through the above parameters and the proposed method:

S502，通过一个计数器c_MT统计在时间长度L内，通过分析类型过滤器，得到 针对特定ML模型的模型训练总次数的统计结果。S502, using a counter c _MT to count the time length L, and analyzing the type filter to obtain Statistics of the total number of model training times for a specific ML model.

S504，通过一个计数器c_MTTime，在其清零后，对在时间长度L内，判定为发生了模型训练的t_opt进行累加，得到针对特定ML模型的模型训练总时长的测量结果。S504, using a counter c _MTTime , after it is cleared, the _topts determined to have undergone model training within the time length L are accumulated to obtain a measurement result of the total model training duration for the specific ML model.

S506，根据模型训练总次数及模型训练总时长，得到NWDAF提供特定ML模型的模型训练平均时长的测量结果，其公式如下：
S506, according to the total number of model training times and the total model training time, obtain the measurement result of the average model training time of the specific ML model provided by NWDAF, and the formula is as follows:

其中，c_MT为模型训练总次数；Among them, c _MT is the total number of model training;

c_MTTime为模型训练总时长；c _MTTime is the total training time of the model;

c_MTAvgTime为模型训练平均时长。c _MTAvgTime is the average training time of the model.

S508，针对具体的应用场合，{c_MT,c_MTTine,c_MTAvgTime}三者中的任一个或多个的组合，均为待测NWDAF的服务性能测量结果。S508: For specific application scenarios, any one or more combinations of {c _MT , c _MTTine , c _MTAvgTime } are all service performance measurement results of the NWDAF to be tested.

上述实施例中，通过模型训练总次数、模型训练总时长、模型训练平均时长任一个或多个的组合，作为待测功能网元的服务性能测量结果评估功能网元等的性能，具备检测可能造成NWDAF资源大量占用的模型训练事件的能力，不需要功能网元主动上报就能观测功能网元内部处理事件。In the above embodiment, the performance of the functional network element, etc. is evaluated by using any one or more combinations of the total number of model training times, the total model training duration, and the average model training duration as the service performance measurement results of the functional network element to be tested. The system has the ability to detect model training events that may cause a large amount of NWDAF resources to be occupied, and can observe internal processing events of the functional network element without the functional network element actively reporting.

基于同一发明构思，本公开实施例中还提供了一种服务性能测量装置，如下面的实施例。由于该装置实施例解决问题的原理与上述方法实施例相似，因此该装置实施例的实施可以参见上述方法实施例的实施，重复之处不再赘述。Based on the same inventive concept, the present disclosure also provides a service performance measurement device, such as the following embodiment. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

图6示出本公开实施例中一种服务性能测量装置示意图，如图6所示，该服务性能测量装置6包括：时间长度获取模块601、测量结果生成模块602；FIG6 shows a schematic diagram of a service performance measurement device in an embodiment of the present disclosure. As shown in FIG6 , the service performance measurement device 6 includes: a time length acquisition module 601 and a measurement result generation module 602;

时间长度获取模块601，获取功能网元提供目标模型提供服务的时间长度信息，其中，目标模型提供服务是用于提供ML模型的服务；The time length acquisition module 601 acquires the time length information of the target model service provided by the functional network element, wherein the target model service is used to provide the ML model service;

测量结果生成模块602，根据时间长度信息及训练门限，生成服务性能测量结果。The measurement result generating module 602 generates a service performance measurement result according to the time length information and the training threshold.

上述实施例中，获取功能网元提供模型相关的服务的时间长度信息，根据上述时间长度信息是否达到设定的训练门限，判断模型训练的发生，量化模型训练对功能网元性能的影响，而不需要对外发出事件提醒，通过模型训练总次数、模型训练总时长、模型训练平均时长任一个或多个的组合，作为待测功能网元的服务性能测量结果评估功能网元等的性能，不需要功能网元主动上报就能观测功能网元内部处理事件。 In the above embodiment, the time length information of the functional network element providing model-related services is obtained, and the occurrence of model training is judged based on whether the above time length information reaches the set training threshold, and the impact of model training on the performance of the functional network element is quantified without issuing event reminders to the outside. The performance of the functional network element is evaluated by using any one or more combinations of the total number of model training times, the total duration of model training, and the average duration of model training as the service performance measurement results of the functional network element to be tested, and the internal processing events of the functional network element can be observed without the functional network element actively reporting.

所属技术领域的技术人员能够理解，本公开的各个方面可以实现为***、方法或程序产品。因此，本公开的各个方面可以具体实现为以下形式，即：完全的硬件实施方式、完全的软件实施方式(包括固件、微代码等)，或硬件和软件方面结合的实施方式，这里可以统称为“电路”、“模块”或“***”。Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "systems".

下面参照图7来描述根据本公开的这种实施方式的电子设备700。图7显示的电子设备700仅仅是一个示例，不应对本公开实施例的功能和使用范围带来任何限制。The electronic device 700 according to this embodiment of the present disclosure is described below with reference to Fig. 7. The electronic device 700 shown in Fig. 7 is only an example and should not bring any limitation to the functions and scope of use of the embodiment of the present disclosure.

如图7所示，电子设备700以通用计算设备的形式表现。电子设备700的组件可以包括但不限于：上述至少一个处理单元710、上述至少一个存储单元720、连接不同***组件(包括存储单元720和处理单元710)的总线730。As shown in Fig. 7, the electronic device 700 is in the form of a general computing device. The components of the electronic device 700 may include but are not limited to: at least one processing unit 710, at least one storage unit 720, and a bus 730 connecting different system components (including the storage unit 720 and the processing unit 710).

其中，所述存储单元存储有程序代码，所述程序代码可以被所述处理单元710执行，使得所述处理单元710执行本说明书上述“示例性方法”部分中描述的根据本公开各种示例性实施方式的步骤。The storage unit stores program codes, which can be executed by the processing unit 710, so that the processing unit 710 executes the steps described in the above “Exemplary Method” section of this specification according to various exemplary embodiments of the present disclosure.

例如，所述处理单元710可以执行上述方法实施例的如下步骤：获取功能网元提供目标机器学习ML模型提供服务的时间长度信息；根据时间长度信息及训练门限，生成服务性能测量结果。For example, the processing unit 710 can execute the following steps of the above method embodiment: obtain the time length information of the functional network element providing the service of the target machine learning ML model; generate a service performance measurement result based on the time length information and the training threshold.

例如，所述处理单元710可以执行上述方法实施例的如下步骤：测量待测的NWDAF实例接收的针对特定类型的ML模型提供服务的服务请求或订阅请求的时间t₁；测量与S102中每个服务请求或订阅请求对应的NWDAF提供ML模型输出结果的时间t₂；计算t_opt＝t₂-t₁，得到待测的NWDAF实例提供每个当前ML模型提供服务的时间长度信息；针对特定NWDAF实例或特定的ML模型类型，生成分析类型过滤器；根据当前NWDAF实例收到的服务请求或订阅请求中的类型标识，将t_opt送入特定的分析类型过滤器，判断当前NWDAF是否触发了ML模型的训练过程；基于服务性能测量结果，可以进一步应用于有关NWDAF性能、NWDAF提供的ML模型性能的评估中。For example, the processing unit 710 may perform the following steps of the above method embodiment: measuring the time t ₁ when the NWDAF instance to be tested receives a service request or a subscription request for providing a service for a specific type of ML model; measuring the time t ₂ when the NWDAF corresponding to each service request or subscription request in S102 provides an ML model output result; calculating _topt = t ₂ - t ₁ to obtain time length information of the service provided by the NWDAF instance to be tested for each current ML model; generating an analysis type filter for a specific NWDAF instance or a specific ML model type; sending _topt to a specific analysis type filter according to a type identifier in a service request or subscription request received by the current NWDAF instance to determine whether the current NWDAF triggers the training process of the ML model; based on the service performance measurement result, it can be further applied to the evaluation of the NWDAF performance and the ML model performance provided by the NWDAF.

例如，所述处理单元710可以执行上述方法实施例的如下步骤：获取功能网元提供目标机器学习ML模型提供服务的时间长度信息；根据时间长度信息及训练门限，生成服务性能测量结果。For example, the processing unit 710 can execute the following steps of the above method embodiment: obtain the time length information of the functional network element providing the service of the target machine learning ML model; generate a service performance measurement result based on the time length information and the training threshold.

存储单元720可以包括易失性存储单元形式的可读介质，例如随机存取存储单元(RAM)7201和/或高速缓存存储单元7202，还可以进一步包括只读存储单元(ROM)7203。 The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 7201 and/or a cache memory unit 7202 , and may further include a read-only memory unit (ROM) 7203 .

存储单元720还可以包括具有一组(至少一个)程序模块7205的程序/实用工具7204，这样的程序模块7205包括但不限于：操作***、一个或者多个应用程序、其它程序模块以及程序数据，这些示例中的每一个或某种组合中可能包括网络环境的实现。The storage unit 720 may also include a program/utility 7204 having a set (at least one) of program modules 7205, such program modules 7205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

总线730可以为表示几类总线结构中的一种或多种，包括存储单元总线或者存储单元控制器、***总线、图形加速端口、处理单元或者使用多种总线结构中的任意总线结构的局域总线。Bus 730 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

电子设备700也可以与一个或多个外部设备740(例如键盘、指向设备、蓝牙设备等)通信，还可与一个或者多个使得用户能与该电子设备700交互的设备通信，和/或与使得该电子设备700能与一个或多个其它计算设备进行通信的任何设备(例如路由器、调制解调器等等)通信。这种通信可以通过输入/输出(I/O)接口750进行。The electronic device 700 may also communicate with one or more external devices 740 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device 700, and/or any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input/output (I/O) interface 750.

并且，电子设备700还可以通过网络适配器760与一个或者多个网络(例如局域网(LAN)，广域网(WAN)和/或公共网络，例如因特网)通信。如图所示，网络适配器760通过总线730与电子设备700的其它模块通信。应当明白，尽管图中未示出，可以结合电子设备700使用其它硬件和/或软件模块，包括但不限于：微代码、设备驱动器、冗余处理单元、外部磁盘驱动阵列、RAID***、磁带驱动器以及数据备份存储***等。Furthermore, the electronic device 700 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and/or public network, such as the Internet) through a network adapter 760. As shown, the network adapter 760 communicates with other modules of the electronic device 700 through a bus 730. It should be understood that, although not shown in the figure, other hardware and/or software modules can be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

通过以上的实施方式的描述，本领域的技术人员易于理解，这里描述的示例实施方式可以通过软件实现，也可以通过软件结合必要的硬件的方式来实现。因此，根据本公开实施方式的技术方案可以以软件产品的形式体现出来，该软件产品可以存储在一个非易失性存储介质(可以是CD-ROM，U盘，移动硬盘等)中或网络上，包括若干指令以使得一台计算设备(可以是个人计算机、服务器、终端装置、或者网络设备等)执行根据本公开实施方式的方法。Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

在本公开的示例性实施例中，还提供了一种计算机可读存储介质，该计算机可读存储介质可以是可读信号介质或者可读存储介质。图7示出本公开实施例中一种计算机可读存储介质的示意图，如图7所示，该计算机可读存储介质700上存储有能够实现本公开上述方法的程序产品。在一些可能的实施方式中，本公开的各个方面还可以实现为一种程序产品的形式，其包括程序代码，当所述程序产品在终端设备上运行时，所述程序代码用于使所述终端设备执行本说明书上述“示例性方法”部分中描述的根据本公开各种示例性实施方式的步骤。 In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. FIG. 7 shows a schematic diagram of a computer-readable storage medium in an embodiment of the present disclosure. As shown in FIG. 7 , a program product capable of implementing the above-mentioned method of the present disclosure is stored on the computer-readable storage medium 700. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps described in the above “Exemplary Method” section of this specification according to various exemplary implementations of the present disclosure.

例如，本公开实施例中的程序产品被处理器执行时实现如下步骤的方法：获取功能网元提供目标机器学习ML模型提供服务的时间长度信息；根据时间长度信息及训练门限，生成服务性能测量结果。For example, when the program product in the embodiment of the present disclosure is executed by a processor, the method implements the following steps: obtaining the time length information of the target machine learning ML model providing services by the functional network element; generating a service performance measurement result based on the time length information and the training threshold.

例如，本公开实施例中的程序产品被处理器执行时实现如下步骤的方法：测量待测的NWDAF实例接收的针对特定类型的ML模型提供服务的服务请求或订阅请求的时间t₁；测量与S102中每个服务请求或订阅请求对应的NWDAF提供ML模型输出结果的时间t₂；计算t_opt＝t₂-t₁，得到待测的NWDAF实例提供每个当前ML模型提供服务的时间长度信息；针对特定NWDAF实例或特定的ML模型类型，生成分析类型过滤器；根据当前NWDAF实例收到的服务请求或订阅请求中的类型标识，将t_opt送入特定的分析类型过滤器，判断当前NWDAF是否触发了ML模型的训练过程；基于服务性能测量结果，可以进一步应用于有关NWDAF性能、NWDAF提供的ML模型性能的评估中。For example, when the program product in the embodiment of the present disclosure is executed by a processor, the following steps are implemented: measuring the time t ₁ of the service request or subscription request for providing services for a specific type of ML model received by the NWDAF instance to be tested; measuring the time t ₂ of the NWDAF providing the ML model output result corresponding to each service request or subscription request in S102; calculating _topt = t ₂ - t ₁ to obtain the time length information of the service provided by each current ML model by the NWDAF instance to be tested; generating an analysis type filter for a specific NWDAF instance or a specific ML model type; sending _topt to a specific analysis type filter according to the type identifier in the service request or subscription request received by the current NWDAF instance to determine whether the current NWDAF has triggered the training process of the ML model; based on the service performance measurement result, it can be further applied to the evaluation of the NWDAF performance and the ML model performance provided by the NWDAF.

例如，本公开实施例中的程序产品被处理器执行时实现如下步骤的方法：通过一个计数器c_MT统计在时间长度L内，通过分析类型过滤器，得到针对特定ML模型的模型训练总次数的统计结果；通过一个计数器c_MTTime，在其清零后，对在时间长度L内，判定为发生了模型训练的t_opt进行累加，得到针对特定ML模型的模型训练总时长的测量结果；根据模型训练总次数及模型训练总时长，得到NWDAF提供特定ML模型的模型训练平均时长的测量结果。For example, when the program product in the embodiment of the present disclosure is executed by a processor, the following steps are implemented: a counter c _MT is used to count the total number of model training times for a specific ML model within a time length L, and a type filter is analyzed to obtain a statistical result; a counter c _MTTime is used, and after it is cleared, _topts that are determined to have undergone model training within the time length L are accumulated to obtain a measurement result of the total model training duration for the specific ML model; based on the total number of model training times and the total model training duration, a measurement result of the average model training duration of the specific ML model provided by NWDAF is obtained.

本公开中的计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

在本公开中，计算机可读存储介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了可读程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。可读信号介质还可以是可读存储介质以外的任何可读介质，该可读介质可以发送、传播或者传输用于由指令执行***、装置或者器件使用或者与其结合使用的程序。In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

可选地，计算机可读存储介质上包含的程序代码可以用任何适当的介质传输，包括但不限于无线、有线、光缆、RF等等，或者上述的任意合适的组合。Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

在具体实施时，可以以一种或多种程序设计语言的任意组合来编写用于执行本 公开操作的程序代码，所述程序设计语言包括面向对象的程序设计语言—诸如Java、C++等，还包括常规的过程式程序设计语言—诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算设备上执行、部分地在用户设备上执行、作为一个独立的软件包执行、部分在用户计算设备上部分在远程计算设备上执行、或者完全在远程计算设备或服务器上执行。In specific implementation, the program for executing the present invention can be written in any combination of one or more programming languages. Program code for public operations, the programming language includes object-oriented programming languages such as Java, C++, etc., and also includes conventional procedural programming languages such as "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

在涉及远程计算设备的情形中，远程计算设备可以通过任意种类的网络，包括局域网(LAN)或广域网(WAN)，连接到用户计算设备，或者，可以连接到外部计算设备(例如利用因特网服务提供商来通过因特网连接)。Where a remote computing device is involved, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

应当注意，尽管在上文详细描述中提及了用于动作执行的设备的若干模块或者单元，但是这种划分并非强制性的。实际上，根据本公开的实施方式，上文描述的两个或更多模块或者单元的特征和功能可以在一个模块或者单元中具体化。反之，上文描述的一个模块或者单元的特征和功能可以进一步划分为由多个模块或者单元来具体化。It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

此外，尽管在附图中以特定顺序描述了本公开中方法的各个步骤，但是，这并非要求或者暗示必须按照该特定顺序来执行这些步骤，或是必须执行全部所示的步骤才能实现期望的结果。附加的或备选的，可以省略某些步骤，将多个步骤合并为一个步骤执行，以及/或者将一个步骤分解为多个步骤执行等。In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and/or one step may be decomposed into multiple steps, etc.

通过以上实施方式的描述，本领域的技术人员易于理解，这里描述的示例实施方式可以通过软件实现，也可以通过软件结合必要的硬件的方式来实现。因此，根据本公开实施方式的技术方案可以以软件产品的形式体现出来，该软件产品可以存储在一个非易失性存储介质(可以是CD-ROM，U盘，移动硬盘等)中或网络上，包括若干指令以使得一台计算设备(可以是个人计算机、服务器、移动终端、或者网络设备等)执行根据本公开实施方式的方法。Through the description of the above implementation modes, it is easy for those skilled in the art to understand that the example implementation modes described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation mode of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation mode of the present disclosure.

本领域技术人员在考虑说明书及实践这里公开的发明后，将容易想到本公开的其它实施方案。本公开旨在涵盖本公开的任何变型、用途或者适应性变化，这些变型、用途或者适应性变化遵循本公开的一般性原理并包括本公开未公开的本技术领域中的公知常识或惯用技术手段。说明书和实施例仅被视为示例性的，本公开的真正范围和精神由所附的权利要求指出。 Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims (17)

1. 一种服务性能测量方法，其中，包括：A service performance measurement method, comprising:

   获取功能网元提供目标模型提供服务的时间长度信息，其中，所述目标模型提供服务是用于提供模型的服务；Acquire time length information of providing a target model service by a functional network element, wherein the target model service is a service for providing a model;

   根据所述时间长度信息及训练门限，生成服务性能测量结果。A service performance measurement result is generated according to the time length information and the training threshold.
2. 根据权利要求1所述的服务性能测量方法，其中，所述模型为机器学习ML模型。The service performance measurement method according to claim 1, wherein the model is a machine learning (ML) model.
3. 根据权利要求1所述的服务性能测量方法，其中，所述服务性能测量结果包括以下至少之一：模型训练总次数、模型训练总时长、模型训练平均时长。The service performance measurement method according to claim 1, wherein the service performance measurement result includes at least one of the following: the total number of model training times, the total model training duration, and the average model training duration.
4. 根据权利要求3所述的服务性能测量方法，其中，所述根据所述时间长度信息及训练门限，生成服务性能测量结果包括：The service performance measurement method according to claim 3, wherein generating the service performance measurement result according to the time length information and the training threshold comprises:

   获取一定时间内所述时间长度信息大于所述训练门限的次数，生成所述模型训练总次数。Obtain the number of times the time length information is greater than the training threshold within a certain period of time, and generate the total number of model training times.
5. 根据权利要求4所述的服务性能测量方法，其中，所述获取一定时间内所述时间长度信息大于所述训练门限的次数，生成所述模型训练总次数包括：According to the service performance measurement method of claim 4, wherein the obtaining of the number of times the time length information is greater than the training threshold within a certain period of time and generating the total number of model training times comprises:

   将计时器的时间长度调整为与所述训练门限一致；Adjusting the time length of the timer to be consistent with the training threshold;

   当所述功能网元接收到所述目标模型提供服务的请求时，启动所述计时器进行计时；When the functional network element receives a request for service provided by the target model, the timer is started to count;

   根据所述功能网元是否在所述计时器超时前提供结果的信息，生成所述模型训练总次数。The total number of model training times is generated according to whether the functional network element provides result information before the timer times out.
6. 根据权利要求5所述的服务性能测量方法，其中，所述根据所述功能网元是否在所述计时器超时前提供结果的信息，生成所述模型训练总次数包括：The service performance measurement method according to claim 5, wherein the generating the total number of model training times according to whether the functional network element provides the result information before the timer times out comprises:

   判断所述功能网元是否在所述计时器超时前提供结果；Determining whether the functional network element provides a result before the timer times out;

   若是，则所述时间长度信息小于或等于所述训练门限，不计数；If yes, the time length information is less than or equal to the training threshold and is not counted;

   否则，所述时间长度信息大于所述训练门限，计数；Otherwise, the time length information is greater than the training threshold, and counting is performed;

   根据一定时间内所述时间长度信息大于所述训练门限的次数，生成所述模型训练总次数。The total number of model training times is generated according to the number of times the time length information is greater than the training threshold within a certain period of time.
7. 根据权利要求5所述的服务性能测量方法，其中，所述根据所述功能网元是否在所述计时器超时前提供结果的信息，生成所述模型训练总次数包括：The service performance measurement method according to claim 5, wherein the generating the total number of model training times according to whether the functional network element provides the result information before the timer times out comprises:

   判断所述功能网元是否在所述计时器超时前提供结果； Determining whether the functional network element provides a result before the timer times out;

   若是，则所述时间长度信息小于所述训练门限，不计数；If yes, the time length information is less than the training threshold and is not counted;

   否则，所述时间长度信息大于或等于所述训练门限，计数；Otherwise, the time length information is greater than or equal to the training threshold, and counts;

   根据一定时间内所述时间长度信息大于或等于所述训练门限的次数，生成所述模型训练总次数。The total number of model training times is generated according to the number of times the time length information is greater than or equal to the training threshold within a certain period of time.
8. 根据权利要求3所述的服务性能测量方法，其中，所述根据所述时间长度信息及训练门限，生成服务性能测量结果包括：The service performance measurement method according to claim 3, wherein generating the service performance measurement result according to the time length information and the training threshold comprises:

   获取一定时间内目标训练时长，其中，所述目标训练时长为大于或等于所述训练门限的时间长度信息；Obtaining a target training duration within a certain period of time, wherein the target training duration is time length information that is greater than or equal to the training threshold;

   计算所述目标训练时长的总和，生成所述模型训练总时长。The sum of the target training durations is calculated to generate the total model training duration.
9. 根据权利要求3所述的服务性能测量方法，其中，所述根据所述时间长度信息及训练门限，生成服务性能测量结果包括：The service performance measurement method according to claim 3, wherein generating the service performance measurement result according to the time length information and the training threshold comprises:

   根据所述模型训练总次数及所述模型训练总时长得到所述模型训练平均时长。The average duration of the model training is obtained according to the total number of model training times and the total duration of the model training.
10. 根据权利要求1所述的服务性能测量方法，其中，所述获取功能网元提供目标模型提供服务的时间长度信息包括：The service performance measurement method according to claim 1, wherein the obtaining of the time length information of the functional network element providing the service of the target model comprises:

    获取所述功能网元接收所述目标模型提供服务的请求的第一时间；Obtaining the first time when the functional network element receives a request for service provided by the target model;

    获取所述请求对应的所述目标模型提供服务输出结果的第二时间；Obtaining a second time at which the target model corresponding to the request provides a service output result;

    计算所述第一时间与所述第二时间的差值，得到所述时间长度信息。The difference between the first time and the second time is calculated to obtain the time length information.
11. 根据权利要求1所述的服务性能测量方法，其中，所述目标模型提供服务包括用于指示如下内容的参数：提供模型、更新模型或重新训练模型。The service performance measurement method according to claim 1, wherein the target model provides a service including parameters for indicating the following contents: providing a model, updating a model or retraining a model.
12. 根据权利要求1所述的服务性能测量方法，其中，还包括：The service performance measurement method according to claim 1, further comprising:

    获取所述目标模型的类型标识；Obtaining a type identifier of the target model;

    根据所述类型标识确定对应的分析类型过滤器；Determine a corresponding analysis type filter according to the type identifier;

    通过所述分析类型过滤器得到所述目标模型提供服务的时间长度信息。The time length information of the service provided by the target model is obtained through the analysis type filter.
13. 根据权利要求1所述的服务性能测量方法，其中，所述功能网元为网络数据分析功能网元。The service performance measurement method according to claim 1, wherein the functional network element is a network data analysis functional network element.
14. 一种服务性能测量装置，其中，包括：A service performance measurement device, comprising:

    时间长度获取模块，获取功能网元提供模型提供服务的时间长度信息，其中，所述目标模型提供服务是用于提供模型的服务；A time length acquisition module, which acquires time length information of a service provided by a model by a functional network element, wherein the service provided by the target model is a service used to provide a model;

    测量结果生成模块，根据所述时间长度信息及训练门限，生成服务性能测量结果。 The measurement result generating module generates a service performance measurement result according to the time length information and the training threshold.
15. 一种电子设备，其中，包括：An electronic device, comprising:

    处理器；以及Processor; and

    存储器，用于存储所述处理器的可执行指令；A memory, configured to store executable instructions of the processor;

    其中，所述处理器配置为经由执行所述可执行指令来执行权利要求1～13中任意一项所述服务性能测量方法。The processor is configured to execute the service performance measurement method according to any one of claims 1 to 13 by executing the executable instructions.
16. 一种计算机可读存储介质，其上存储有计算机程序，其中，所述计算机程序被处理器执行时实现权利要求1～13中任意一项所述的服务性能测量方法。A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the service performance measurement method according to any one of claims 1 to 13.
17. 一种计算机程序产品，包括计算机程序，其中，所述计算机程序被处理器执行时实现权利要求1～13中任意一项所述的服务性能测量方法。 A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the service performance measurement method according to any one of claims 1 to 13 is implemented.