CN114116520A - Algorithm evaluation method, device, gateway and storage medium - Google Patents
Algorithm evaluation method, device, gateway and storage medium Download PDFInfo
- Publication number
- CN114116520A CN114116520A CN202111490278.8A CN202111490278A CN114116520A CN 114116520 A CN114116520 A CN 114116520A CN 202111490278 A CN202111490278 A CN 202111490278A CN 114116520 A CN114116520 A CN 114116520A
- Authority
- CN
- China
- Prior art keywords
- evaluation
- service
- algorithm
- evaluation service
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3688—Test management for test execution, e.g. scheduling of test suites
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3664—Environments for testing or debugging software
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3696—Methods or tools to render software testable
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The embodiment of the disclosure relates to an algorithm evaluation method, an algorithm evaluation device, a gateway and a storage medium. In at least one embodiment of the disclosure, an algorithm evaluation script is not used for directly docking a platform, but an evaluation service is created, a gateway encapsulates an evaluation service client and an evaluation service request based on an evaluation service type, and then the gateway can transparently transmit the algorithm evaluation request sent by the platform to the evaluation service to complete algorithm evaluation, so that the effect of decoupling the evaluation service from the platform is realized, and the platform is convenient to be compatible with different types of evaluation services.
Description
Technical Field
The embodiment of the disclosure relates to the technical field of computers, in particular to an algorithm evaluation method, an algorithm evaluation device, a gateway and a storage medium.
Background
The measurement of the effect of the machine learning algorithm is based on the algorithm evaluation, and in order to increase the automation and visualization degree of the algorithm evaluation, the process of the algorithm evaluation needs to be standardized, and the integration is generated from uploading data to an evaluation report.
However, the algorithm evaluation scripts of different algorithms are different, the algorithm evaluation script is higher in cost and maintenance cost when being accessed to the machine learning platform, and in addition, the algorithm evaluation script is higher in coupling property and higher in customization degree when being directly connected to the machine learning platform. Therefore, it is desirable to provide an algorithm evaluation scheme, which reduces the cost of accessing the machine learning platform and the maintenance cost, and reduces the coupling degree with the machine learning platform.
Disclosure of Invention
In order to solve at least one problem in the prior art, at least one embodiment of the present disclosure provides an algorithm evaluating method, apparatus, gateway, and storage medium.
In a first aspect, an embodiment of the present disclosure provides an algorithm evaluating method, including:
receiving an algorithm evaluation request, wherein information carried by the algorithm evaluation request comprises an evaluation service name and an algorithm service name;
acquiring an algorithm evaluation case set corresponding to the algorithm evaluation request;
acquiring corresponding evaluation service configuration information based on the evaluation service name, wherein the evaluation service configuration information comprises an evaluation service type which is a protocol type of the evaluation service;
based on the evaluation service name and the evaluation service type, an evaluation service client and an evaluation service request are obtained by packaging, wherein the evaluation service request comprises an algorithm service name;
and distributing the evaluation service request and each case in the algorithm evaluation case set to an evaluation service corresponding to the evaluation service request through an evaluation service client, and controlling the evaluation service to perform algorithm evaluation based on the algorithm service name and each case.
In some embodiments, the obtaining of the evaluation service client and the evaluation service request by packaging based on the evaluation service name and the evaluation service type includes:
based on the fact that the evaluation service type is a hypertext transfer protocol, obtaining an evaluation service client and an evaluation service request corresponding to the hypertext transfer protocol according to client class encapsulation of the hypertext transfer protocol; the client class of the hypertext transfer protocol is derived based on a client base class;
or, based on the remote procedure call protocol as the evaluation service type, obtaining an evaluation service client and an evaluation service request corresponding to the remote procedure call protocol according to the client class encapsulation of the remote procedure call protocol; the client class of the remote procedure call protocol is derived based on a client base class.
In some embodiments, evaluating the service configuration information further comprises a current limit parameter value;
before distributing each case in the evaluation service request and the algorithm evaluation case set to the evaluation service corresponding to the evaluation service request through the evaluation service client, the algorithm evaluation method further comprises the following steps:
determining whether a current limit condition is satisfied based on the current limit parameter value; if the evaluation service requirement is met, after the time interval between the current time and the last distribution is larger than or equal to a preset time interval threshold, the step of distributing each case in the evaluation service request and the algorithm evaluation case set to the evaluation service corresponding to the evaluation service request through the evaluation service client is executed.
In some embodiments, the evaluating service performing the algorithm evaluation based on the algorithm service name and the use cases comprises:
the evaluation service receives evaluation service requests and all cases based on processing layer codes in the evaluation service;
the evaluation service calls the algorithm service corresponding to the algorithm service name in the evaluation service request based on each use case and the service layer code in the evaluation service, and receives the result returned by the algorithm service for each use case;
the service layer code and the processing layer code are generated based on the algorithm service name and the interface definition language file, and the algorithm service name and the interface definition language file correspond to the same algorithm service.
In some embodiments, the service layer code further comprises a custom evaluation code;
after obtaining the result returned by the algorithm service, the algorithm evaluating method further comprises the following steps:
and the evaluation service evaluates the result based on the user-defined evaluation code.
In some embodiments, the information carried by each use case in the algorithm evaluation use case set comprises a log identifier;
after the evaluation service carries out algorithm evaluation based on the algorithm service name and each use case, the algorithm evaluation method further comprises the following steps:
and the evaluation service generates an algorithm evaluation log corresponding to each use case, and the algorithm evaluation log comprises log marks of the corresponding use cases.
In a second aspect, an embodiment of the present disclosure further provides an algorithm evaluating device, including:
the receiving unit is used for receiving an algorithm evaluation request, wherein information carried by the algorithm evaluation request comprises an evaluation service name and an algorithm service name;
the first obtaining unit is used for obtaining an algorithm evaluation case set corresponding to the algorithm evaluation request;
the second obtaining unit is used for obtaining corresponding evaluation service configuration information based on the evaluation service name, wherein the evaluation service configuration information comprises an evaluation service type, and the evaluation service type is a protocol type of the evaluation service;
the evaluation service client is used for evaluating the service type of the evaluation service client;
and the distribution unit is used for distributing the evaluation service request and each case in the algorithm evaluation case set to an evaluation service corresponding to the evaluation service request through the evaluation service client, and controlling the evaluation service to perform algorithm evaluation based on the algorithm service name and each case.
In some embodiments, the encapsulation unit is configured to obtain an evaluation service client and an evaluation service request corresponding to the hypertext transfer protocol according to encapsulation of a client class of the hypertext transfer protocol based on that the evaluation service type is the hypertext transfer protocol; the client class of the hypertext transfer protocol is derived based on a client base class; or, based on the remote procedure call protocol as the evaluation service type, obtaining an evaluation service client and an evaluation service request corresponding to the remote procedure call protocol according to the client class encapsulation of the remote procedure call protocol; the client class of the remote procedure call protocol is derived based on a client base class.
In a third aspect, an embodiment of the present disclosure further provides a gateway, including: a processor and a memory; the processor is adapted to perform the steps of the algorithm evaluation method according to any of the embodiments of the first aspect by calling a program or instructions stored in the memory.
In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, where the computer-readable storage medium stores a program or instructions, and the program or instructions cause a computer to execute the steps of the algorithm evaluating method according to any embodiment of the first aspect.
It can be seen that in at least one embodiment of the present disclosure, an algorithm evaluation script is not used to directly interface a platform, but an evaluation service is created, a gateway encapsulates an evaluation service client and an evaluation service request based on an evaluation service type, and then the gateway can transparently transmit the algorithm evaluation request sent by the platform to the evaluation service to complete algorithm evaluation, so that an effect of decoupling the evaluation service from the platform is achieved, and the platform is compatible with different types of evaluation services.
Drawings
To more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art according to the drawings.
FIG. 1 is an exemplary flow chart of an algorithm evaluation method provided by an embodiment of the present disclosure;
FIG. 2 is a diagram of an exemplary application scenario provided by an embodiment of the present disclosure;
FIG. 3 is an exemplary flow chart of a flow control provided by an embodiment of the present disclosure;
FIG. 4 is an exemplary flowchart illustrating an algorithm evaluation performed by an evaluation service based on an algorithm service name and each case in an algorithm evaluation case set according to an embodiment of the present disclosure;
FIG. 5 is an exemplary block diagram of an algorithm evaluation device provided by the embodiments of the present disclosure;
fig. 6 is an exemplary block diagram of an electronic device provided by an embodiment of the present disclosure.
Detailed Description
In order that the above objects, features and advantages of the present disclosure can be more clearly understood, the present disclosure will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the embodiments described are only a few embodiments of the present disclosure, and not all embodiments. The specific embodiments described herein are merely illustrative of the disclosure and are not intended to be limiting. All other embodiments derived by one of ordinary skill in the art from the described embodiments of the disclosure are intended to be within the scope of the disclosure.
It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions.
Fig. 1 is an exemplary flowchart of an algorithm evaluation method provided in an embodiment of the present disclosure, where an execution subject of the algorithm evaluation method is a gateway, and the algorithm evaluation method may include, but is not limited to, the following steps 101 to 104:
101. and receiving an algorithm evaluation request, wherein the information carried by the algorithm evaluation request comprises an evaluation service name and an algorithm service name.
The gateway may receive an algorithm evaluation request sent by a platform, where the platform is, for example, a machine learning algorithm platform, or a platform for testing a machine learning algorithm.
The evaluation service name is the name of the evaluation service, and the evaluation service is a service created in advance. In some embodiments, the profiling service is created by the platform.
The algorithmic service name is the name of the algorithmic service, and the algorithmic service is another service created in advance. In some embodiments, the algorithmic service is created by the platform.
102. And obtaining an algorithm evaluation case set corresponding to the algorithm evaluation request.
After receiving the algorithm evaluation request sent by the platform, the gateway can also obtain an algorithm evaluation Case set corresponding to the algorithm evaluation request sent by the platform, wherein the algorithm evaluation Case set comprises a plurality of cases (Case).
103. And acquiring corresponding evaluation service configuration information based on the evaluation service name, wherein the evaluation service configuration information comprises an evaluation service type which is a protocol type of the evaluation service.
The gateway can obtain the evaluation service configuration information corresponding to the evaluation service name from the dynamic configuration center based on the evaluation service name. The dynamic configuration center can be understood as a configured service for configuring different evaluation services, and therefore, the dynamic configuration center stores configuration information of different evaluation services.
The evaluation service configuration information includes, but is not limited to, an evaluation service name and an evaluation service type. Profiling service types include, but are not limited to, hypertext Transfer Protocol (HTTP) and Remote Procedure Call Protocol (RPC).
104. And based on the evaluation service name and the evaluation service type, packaging to obtain an evaluation service client and an evaluation service request, wherein the evaluation service request comprises an algorithm service name.
And the gateway packages and obtains an evaluation service Client (HTTP Client) and an evaluation service Request (Post Request) corresponding to the HTTP based on the fact that the evaluation service type is the HTTP. For example, if the evaluation service name is a and the evaluation service type is HTTP, the gateway packages the evaluation service client side with the corresponding domain name based on the evaluation service name a, and records the result as a (HTTP).
And the gateway is an RPC based on the evaluation service type, and an evaluation service Client (RPC Client) and an evaluation service Request (RPC Request) corresponding to the RPC are obtained by packaging. For example, if the evaluation service name is b and the evaluation service type is RPC, the gateway packages the evaluation service client side of the corresponding unique identification service based on the evaluation service name b, and records the result as b (RPC).
It should be noted that the flow of the HTTP Client and the Post Request corresponding to the HTTP encapsulation is as follows: deriving a Client class (Http Client class) of the hypertext transfer protocol based on a Client base class (Client base class) in advance, and then obtaining an evaluation service Client (Http Client) and an evaluation service Request (Post Request) corresponding to the Http according to the Http Client class when the evaluation service type is the Http.
The RPC Client and RPC Request flow corresponding to RPC encapsulation is as follows: and deriving a Client class (RPC Client class) of the remote procedure call protocol based on a Client base class (Client base class) in advance, and when the evaluation service type is RPC, packaging according to the RPC Client class to obtain an evaluation service Client (RPC Client) and an evaluation service Request (RPC Request) corresponding to the RPC.
105. And distributing the evaluation service request and each case in the algorithm evaluation case set to an evaluation service corresponding to the evaluation service request through an evaluation service client, and controlling the evaluation service to perform algorithm evaluation based on the algorithm service name and each case.
For example, the gateway transmits each case in the evaluation service Request (Post Request) and the algorithm evaluation case set to the evaluation service a corresponding to the evaluation service Request (Post Request) through the evaluation service client a (http). And the evaluation service a performs algorithm evaluation based on the algorithm service name and each use case carried in the evaluation service request.
For another example, the gateway transmits an evaluation service Request (RPC Request) and each case in the algorithm evaluation case set to an evaluation service b corresponding to the evaluation service Request (RPC Request) through an evaluation service client b (RPC). And the evaluation service b performs algorithm evaluation based on the algorithm service name and each use case carried in the evaluation service request.
Therefore, in at least one embodiment of the disclosure, an algorithm evaluation script is not used for directly docking a platform, but an evaluation service is created, a gateway encapsulates an evaluation service client and an evaluation service request based on an evaluation service type, and then the gateway can transparently transmit the algorithm evaluation request sent by the platform to the evaluation service to complete algorithm evaluation, so that the effect of decoupling the evaluation service and the platform is realized, the platform is compatible with evaluation services of different types, and the problem that the platform is difficult to be compatible with the algorithm evaluation scripts of different types due to the fact that the algorithm evaluation script is directly coupled to the platform and the customization degree is high in the prior art is solved.
In addition, at least one embodiment of the disclosure realizes the effect of decoupling the evaluation service from the platform, so that the platform is compatible with evaluation services of different types, the evaluation service does not need to be customized for the platform, and the cost and maintenance cost of the evaluation service access platform are reduced.
In addition, at least one embodiment of the disclosure realizes the effect of decoupling the evaluation service and the platform, so that the maintenance and upgrading cost of coping with algorithm iteration can be reduced, and the method is safe and controllable.
In addition, aiming at the problem that in the existing algorithm evaluation process, multiple offline and online environments need to be covered and network physical isolation exists between the environments, at least one embodiment of the disclosure solves the requirement of test coverage on multiple research and development environments after environment physical isolation through a gateway on the premise of ensuring data safety.
Fig. 2 is an exemplary application scenario diagram provided by the embodiment of the present disclosure, in fig. 2, a platform may send an algorithm evaluation request to a gateway, where information carried by the algorithm evaluation request includes an evaluation service name and an algorithm service name.
In fig. 2, the dynamic configuration center configures different evaluation services and stores configuration information of the different evaluation services. The evaluation service configuration information includes, but is not limited to, an evaluation service name and an evaluation service type.
In fig. 2, after receiving an algorithm evaluation request sent by a platform, a gateway may extract an evaluation service name and an algorithm service name carried in the algorithm evaluation request, and then actively obtain, based on the evaluation service name, evaluation service configuration information corresponding to the evaluation service name from a dynamic configuration center.
In fig. 2, the gateway packages the evaluation service client and the evaluation service request based on the evaluation service name carried in the algorithm evaluation request and the evaluation service type in the evaluation service configuration information, where the information carried in the evaluation service request includes, but is not limited to, the algorithm service name.
In fig. 2, the gateway distributes the evaluation service request obtained by encapsulation to the corresponding evaluation service through the evaluation service client obtained by encapsulation.
In fig. 2, after receiving the evaluation service request, the evaluation service may extract an algorithm service name carried in the evaluation service request, further invoke a corresponding algorithm service based on the algorithm service name, and receive a result returned by the algorithm service.
Fig. 3 is an exemplary flow chart of flow control provided by the embodiments of the present disclosure. The flow control is executed before the gateway distributes the evaluation service request to the corresponding evaluation service through the evaluation service client. In this embodiment, the evaluation service configuration information further includes a current limiting parameter value, where the current limiting parameter is, for example, the number of requests processed Per Second (Query Per Second, QPS).
In fig. 3, the flow control includes the following steps 301 to 305:
301. and the gateway acquires evaluation service configuration information corresponding to the evaluation service name from the dynamic configuration center.
302. And the gateway extracts the current limiting parameter value from the evaluation service configuration information.
303. The gateway judges whether a current limiting condition is met or not based on the current limiting parameter value; if the current limit condition is satisfied, go to step 304; if the current limiting condition is not satisfied, step 305 is executed. In some embodiments, the gateway may employ a current limit (ratelimit) algorithm to determine whether a current limit condition is satisfied. The current limit (ratelimit) algorithm belongs to the mature technology in the field and is not described in detail.
304. And after the time interval between the current time and the last distribution is greater than or equal to a preset time interval threshold, the gateway distributes the evaluation service request and each case in the algorithm evaluation case set to the evaluation service corresponding to the evaluation service request through the evaluation service client.
305. And the gateway directly distributes the evaluation service request and each case in the algorithm evaluation case set to the evaluation service corresponding to the evaluation service request through the evaluation service client.
Therefore, through flow control, the gateway can be prevented from frequently sending requests to the evaluation service in a short time, the impact on the evaluation service is reduced, and the evaluation efficiency is improved.
In some embodiments, the gateway may also perform flow control for sending each use case in the algorithm evaluation use case set, where the flow control is similar to the flow control shown in fig. 3 and is not described again.
Therefore, the gateway can send the cases in the algorithm evaluation case set to the evaluation service one by one, so as to realize real-time evaluation of the algorithm and support real-time display of evaluation indexes and evaluation progress.
In addition, compared with the prior art in which an asynchronous mode is adopted for evaluation, the evaluation process cannot be displayed in real time, and further, the case with failed evaluation cannot be checked in real time, and if the case with failed evaluation needs to be retried, complex troubleshooting and recovery operations are required. In the embodiment of the disclosure, the real-time evaluation of the algorithm can be realized, and for the case with failed evaluation, the case with failed evaluation can be immediately retried, so that the retry efficiency is improved.
In some embodiments, the information carried by each use case in the set of use cases evaluated by the algorithm includes a log identifier (LogID). The LogID may be generated by the platform using a SnowFlake algorithm (SnowFlake). After the evaluation service carries out algorithm evaluation on the basis of the algorithm service name and each use case, the evaluation service generates an algorithm evaluation log corresponding to each use case, and the algorithm evaluation log comprises log marks of the corresponding use cases.
Therefore, the LogID ensures the uniqueness of transmission of each use case in a distributed system under mass data, and is convenient for inquiring the algorithm evaluation log of each use case.
In some embodiments, the gateway may add an algorithm-test-located (algorithmically-test-located) field to the HTTP-Header for the type of profiling service being HTTP, and write the LogID to the field. And the evaluation service can further extract the value of the algorithm-test-located field from the HTTP-Header, namely, the LogID.
In some embodiments, for the case that the evaluation service type is RPC, the gateway may add an algorithm-test-located (algorithmic-test-located) field in a general Base structure of the evaluation service Request (RPC Request), and then write the LogID into the field. And then the evaluation service can extract the value of the algorithm-test-registered field from the general Base structure body, namely extracting the LogID.
It can be seen that by defining a unified LogID extraction object: and if a new protocol type is added, the algorithm-test-location field can be set at a specific part of the new protocol, and the specific part of the protocol is extracted in a template mode to further obtain the LogID.
In addition, compared with the prior art in which an asynchronous mode is adopted for evaluation, the evaluation process cannot be displayed in real time, and further, the case with failed evaluation cannot be checked in real time, and if the case with failed evaluation needs to be retried, complex troubleshooting and recovery operations are required. In the embodiment of the disclosure, the real-time evaluation of the algorithm can be realized, the failed use cases can be immediately retried by positioning the evaluation failed use cases through the LogID, and the retry efficiency is improved.
In addition, because the real-time evaluation of the algorithm can be realized in the embodiment of the disclosure, and the evaluation service can generate the algorithm evaluation logs corresponding to the use cases, the algorithm evaluation logs substantially belong to the streaming logs, and the algorithm evaluation logs comprise the logIDs of the corresponding use cases, so that the use cases in the algorithm evaluation case set can be conveniently analyzed one by one in a buried manner, and the log analysis cost is reduced.
Fig. 4 is an exemplary flowchart of performing algorithm evaluation by an evaluation service based on an algorithm service name and each use case in an algorithm evaluation use case set according to an embodiment of the present disclosure, as shown in fig. 4, including the following steps 401 and 402:
401. the evaluation service receives an evaluation service request and each case in the algorithm evaluation case set based on a processing layer code in the evaluation service, and information carried in the evaluation service request includes but is not limited to an algorithm service name.
A handling layer (handler) in the evaluation service is used to interact with the gateway. The handling layer (handler) provides a generic interface for interaction with the gateway, which can be understood as an access interface for the profiling service.
The evaluation service receives an evaluation service request sent by the gateway through a processing layer (handler), and further obtains an algorithm service name carried in the evaluation service request. In addition, the evaluation service receives each case in the case set through an algorithm sent by the gateway through a processing layer (handler).
402. The evaluation service calls the algorithm service corresponding to the algorithm service name in the evaluation service request based on each case in the algorithm evaluation case set and the service layer code in the evaluation service, and receives the result returned by the algorithm service for each case.
A service layer (service) in the evaluation service is used for interacting with the algorithmic service. The evaluation service acquires the algorithm service name and each case in the algorithm evaluation case set from the processing layer (handler) through the service layer (service), and further interacts with the algorithm service corresponding to the algorithm service name to obtain the result returned by the algorithm service for each case. The result returned by the algorithm service can be provided to the user as an algorithm evaluation result.
The service layer code and the processing layer code are generated based on an algorithm service name and an Interface Definition Language (IDL) file, and the algorithm service name and the IDL file correspond to the same algorithm service.
The IDL file includes, but is not limited to: calling interfaces of algorithm services, calling parameters, returning parameters and the like. In some embodiments, one profiling service may call multiple algorithm services, for example, one profiling service may call an algorithm service of an HTTP protocol, and may also call an algorithm service of an RPC protocol, that is, an IDL file includes call interfaces of multiple algorithm services, call parameters, and return parameters.
Besides service layer codes and processing layer codes, the evaluation service also comprises IDL files, and a thrift file is added under the directory of the IDL files, wherein the thrift is an interface description language and a binary communication protocol and is used for defining and creating cross-language services. The thread file includes, but is not limited to, a calling interface of an algorithm service, a structure for calling a parameter, a structure for returning a parameter, and the like.
In some embodiments, the platform generates service layer code and process layer code in the profiling service through thrift2pyi (a thrift tool) based on the algorithmic service name and the IDL file.
In some embodiments, a service layer (service) code may also include, but is not limited to, a custom profile code. After the evaluation service obtains the result returned by the algorithm service through a service layer (service), the result can be evaluated based on the self-defined evaluation code.
For example, the evaluation service tags the result based on the custom evaluation code, and the tag value is used to mark whether the result is a better result or a worse result, thereby determining statistical data of the algorithm effect based on the tag values corresponding to different cases.
Therefore, in at least one embodiment of the disclosure, the service layer code and the processing layer code in the evaluation service can be automatically generated based on the algorithm service name and the IDL file, the service layer code and the processing layer code do not need to be manually edited, and the cost of the evaluation service access platform is reduced.
It is noted that, for simplicity of description, the foregoing method embodiments are described as a series of acts or combination of acts, but those skilled in the art will appreciate that the disclosed embodiments are not limited by the order of acts described, as some steps may occur in other orders or concurrently with other steps in accordance with the disclosed embodiments. In addition, those skilled in the art can appreciate that the embodiments described in the specification all belong to alternative embodiments.
Fig. 5 is an exemplary block diagram of an algorithm evaluation device according to an embodiment of the present disclosure, where the algorithm evaluation device may be implemented as a gateway or a part of the gateway in fig. 2. As shown in fig. 5, the algorithm evaluation device may be divided into a plurality of units, which may include, but are not limited to: a receiving unit 51, a first acquiring unit 52, a second acquiring unit 53, a packaging unit 54 and a distributing unit 55.
The receiving unit 51 is configured to receive an algorithm evaluation request, where information carried by the algorithm evaluation request includes an evaluation service name and an algorithm service name.
The first obtaining unit 52 obtains an algorithm evaluation case set corresponding to the algorithm evaluation request;
and the second obtaining unit 53 is configured to obtain corresponding evaluation service configuration information based on the evaluation service name, where the evaluation service configuration information includes an evaluation service type, and the evaluation service type is a protocol type of the evaluation service.
And the encapsulating unit 54 is configured to encapsulate to obtain an evaluation service client and an evaluation service request based on the evaluation service name and the evaluation service type, where the evaluation service request includes an algorithm service name.
And the distribution unit 55 is used for distributing the evaluation service request and each case in the algorithm evaluation case set to the evaluation service corresponding to the evaluation service request through the evaluation service client, and controlling the evaluation service to perform algorithm evaluation based on the algorithm service name and each case.
In some embodiments, the encapsulating unit 53 is configured to, based on that the evaluation service type is a hypertext transfer protocol, encapsulate, according to a client class of the hypertext transfer protocol, to obtain an evaluation service client and an evaluation service request corresponding to the hypertext transfer protocol, where the client class of the hypertext transfer protocol is a class derived based on a client base class; or, based on the fact that the evaluation service type is a remote procedure call protocol, the evaluation service client and the evaluation service request corresponding to the remote procedure call protocol are obtained by packaging according to the client class of the remote procedure call protocol, wherein the client class of the remote procedure call protocol is a class derived based on the client base class.
In some embodiments, evaluating the service configuration information further includes a current limit parameter value. The distribution unit 54 is configured to determine whether a current limiting condition is satisfied based on the current limiting parameter value; if the current limiting condition is met, after the time interval between the current time and the last distribution is greater than or equal to a preset time interval threshold value, distributing each case in the evaluation service request and the algorithm evaluation case set to an evaluation service corresponding to the evaluation service request through an evaluation service client; and if the current limiting condition is not met, directly distributing each case in the evaluation service request and the algorithm evaluation case set to an evaluation service corresponding to the evaluation service request through the evaluation service client.
In some embodiments, the evaluation service receives an evaluation service request and each case in an algorithm evaluation case set based on a processing layer code in the evaluation service, further calls an algorithm service corresponding to an algorithm service name in the evaluation service request based on each case and a service layer code in the evaluation service, and receives a result returned by the algorithm service for each case. The service layer code and the processing layer code are generated based on the algorithm service name and the interface definition language file, and the algorithm service name and the interface definition language file correspond to the same algorithm service.
In some embodiments, the service layer code further includes a custom profile code. And after the evaluation service acquires the result returned by the algorithm service, evaluating the result based on the user-defined evaluation code.
In some embodiments, the information carried by each use case in the algorithm evaluation use case set comprises log identification. And after the evaluation service performs algorithm evaluation on the basis of the algorithm service name and each use case, generating an algorithm evaluation log corresponding to each use case, wherein the algorithm evaluation log comprises a log identifier of the corresponding use case.
It should be noted that, for the specific details of each unit of the algorithm evaluating device, reference may be made to the algorithm evaluating method shown in fig. 1 and its related embodiments, and details are not repeated for avoiding repetition.
In some embodiments, the division of each unit in the algorithm evaluating device is only one logic function division, and there may be another division manner in actual implementation, for example, at least two units in each unit of the algorithm evaluating device may be implemented as one unit; each unit in the algorithm evaluating device can also be divided into a plurality of subunits. It will be understood that the various units or sub-units may be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application.
Fig. 6 is a schematic structural diagram of a gateway according to an embodiment of the present disclosure. As shown in fig. 6, the gateway includes: at least one processor 61, at least one memory 62 and at least one communication interface 63. The various components in the electronic device are coupled together by a bus system 64. And a communication interface 63 for information transmission with an external device. Understandably, the bus system 64 is used to enable connective communication between these components. The bus system 64 includes a power bus, a control bus, and a status signal bus in addition to the data bus. For clarity of illustration, however, the various buses are labeled as bus system 64 in fig. 6.
It will be appreciated that the memory 62 in the present embodiment can be either volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory.
In some embodiments, memory 62 stores elements, executable units or data structures, or a subset thereof, or an expanded set thereof: an operating system and an application program.
The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, and the like, and is used for implementing various basic tasks and processing hardware-based tasks. The application programs, including various application programs such as a Media Player (Media Player), a Browser (Browser), etc., are used to implement various application tasks. The program for implementing the algorithm evaluation method provided by the embodiment of the present disclosure may be included in an application program.
In the embodiment of the present disclosure, the processor 61 calls a program or an instruction stored in the memory 62, specifically, may be a program or an instruction stored in an application program, and the processor 61 is configured to execute the steps of the algorithm evaluation method provided in the embodiment of the present disclosure.
The algorithm evaluation method provided by the embodiment of the disclosure can be applied to the processor 61, or implemented by the processor 61. The processor 61 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 61. The Processor 61 may be a general-purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, or discrete hardware components. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The steps of the algorithm evaluation method provided by the embodiment of the disclosure can be directly embodied as the execution of a hardware decoding processor, or the combination of hardware and software units in the decoding processor. The software elements may be located in ram, flash, rom, prom, or eprom, registers, among other storage media that are well known in the art. The storage medium is located in a memory 62, and the processor 61 reads the information in the memory 62 and performs the steps of the method in combination with its hardware.
The embodiments of the present disclosure further provide a computer-readable storage medium, where a program or an instruction is stored in the computer-readable storage medium, and the program or the instruction enables a computer to execute the steps of the embodiments of the algorithm evaluation method, which are not described herein again to avoid repeated descriptions. In some embodiments, the computer-readable storage medium is a non-transitory computer-readable storage medium.
The embodiments of the present disclosure further provide a computer program product, where the computer program product includes a computer program, the computer program is stored in a computer-readable storage medium, and at least one processor of the computer reads and executes the computer program from the storage medium, so that the computer executes the steps of the embodiments of the algorithm evaluation method, which is not described herein again to avoid repeated descriptions.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
Those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than others, combinations of features of different embodiments are meant to be within the scope of the disclosure and form different embodiments.
Those skilled in the art will appreciate that the description of each embodiment has a respective emphasis, and reference may be made to the related description of other embodiments for those parts of an embodiment that are not described in detail.
Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations fall within the scope defined by the appended claims.
Claims (10)
1. An algorithm evaluation method, characterized in that the method comprises:
receiving an algorithm evaluation request, wherein information carried by the algorithm evaluation request comprises an evaluation service name and an algorithm service name;
acquiring an algorithm evaluation case set corresponding to the algorithm evaluation request;
acquiring corresponding evaluation service configuration information based on the evaluation service name, wherein the evaluation service configuration information comprises an evaluation service type which is a protocol type of the evaluation service;
based on the evaluation service name and the evaluation service type, an evaluation service client and an evaluation service request are obtained by packaging, wherein the evaluation service request comprises the algorithm service name;
and distributing the evaluation service request and each case in the algorithm evaluation case set to an evaluation service corresponding to the evaluation service request through the evaluation service client, and controlling the evaluation service to perform algorithm evaluation based on the algorithm service name and each case.
2. The method according to claim 1, wherein the encapsulating the evaluation service client and the evaluation service request based on the evaluation service name and the evaluation service type comprises:
based on the fact that the evaluation service type is a hypertext transfer protocol, obtaining an evaluation service client and an evaluation service request corresponding to the hypertext transfer protocol according to client class encapsulation of the hypertext transfer protocol; the client class of the hypertext transfer protocol is derived based on a client base class;
or, based on the fact that the evaluation service type is a remote procedure call protocol, obtaining an evaluation service client and an evaluation service request corresponding to the remote procedure call protocol according to client class encapsulation of the remote procedure call protocol; the client class of the remote procedure call protocol is derived based on a client base class.
3. The method according to claim 2, wherein the evaluation service configuration information further includes a current limiting parameter value;
before the evaluating service client distributes each case in the evaluating service request and the algorithm evaluating case set to the evaluating service corresponding to the evaluating service request, the method further comprises:
determining whether a current limit condition is satisfied based on the current limit parameter value; if the evaluation service request is met, the step of distributing the evaluation service request and each case in the algorithm evaluation case set to the evaluation service corresponding to the evaluation service request through the evaluation service client is executed after the time interval between the current time and the last distribution is larger than or equal to a preset time interval threshold.
4. The method according to claim 1, wherein the evaluating service performing algorithm evaluation based on the algorithm service name and the use cases comprises:
the evaluation service receives the evaluation service request and each use case based on a processing layer code in the evaluation service;
the evaluation service calls the algorithm service corresponding to the algorithm service name in the evaluation service request based on each use case and the service layer code in the evaluation service, and receives the result returned by the algorithm service for each use case;
the service layer code and the processing layer code are generated based on the algorithm service name and the interface definition language file, and the algorithm service name and the interface definition language file correspond to the same algorithm service.
5. The method according to claim 4, wherein the service layer code further comprises a custom evaluation code;
after the result returned by the algorithm service is obtained, the method further comprises the following steps:
and the evaluating service evaluates the result based on the user-defined evaluating code.
6. The method according to claim 1, wherein the information carried by each use case in the algorithm evaluation use case set comprises a log identifier;
after the evaluation service carries out algorithm evaluation based on the algorithm service name and each use case, the method further comprises the following steps:
and the evaluation service generates an algorithm evaluation log corresponding to each use case, and the algorithm evaluation log comprises log marks of the corresponding use cases.
7. An algorithm evaluation device, characterized in that the device comprises:
the system comprises a receiving unit, a judging unit and a judging unit, wherein the receiving unit is used for receiving an algorithm evaluation request, and information carried by the algorithm evaluation request comprises an evaluation service name and an algorithm service name;
the first obtaining unit is used for obtaining an algorithm evaluation case set corresponding to the algorithm evaluation request;
the second obtaining unit is used for obtaining corresponding evaluation service configuration information based on the evaluation service name, wherein the evaluation service configuration information comprises an evaluation service type, and the evaluation service type is a protocol type of the evaluation service;
the evaluation service client is used for evaluating the service name of the evaluation service client and evaluating the service type of the evaluation service client;
and the distribution unit is used for distributing the evaluation service request and all cases in the algorithm evaluation case set to the evaluation service corresponding to the evaluation service request through the evaluation service client, and controlling the evaluation service to perform algorithm evaluation based on the algorithm service name and all cases.
8. The apparatus of claim 7, wherein the encapsulation unit is configured to:
based on the fact that the evaluation service type is a hypertext transfer protocol, obtaining an evaluation service client and an evaluation service request corresponding to the hypertext transfer protocol according to client class encapsulation of the hypertext transfer protocol; the client class of the hypertext transfer protocol is derived based on a client base class;
or, based on the fact that the evaluation service type is a remote procedure call protocol, obtaining an evaluation service client and an evaluation service request corresponding to the remote procedure call protocol according to client class encapsulation of the remote procedure call protocol; the client class of the remote procedure call protocol is derived based on a client base class.
9. A gateway, comprising: a processor and a memory;
the processor is adapted to perform the steps of the method of any one of claims 1 to 6 by calling a program or instructions stored in the memory.
10. A computer-readable storage medium, characterized in that it stores a program or instructions for causing a computer to carry out the steps of the method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111490278.8A CN114116520B (en) | 2021-12-08 | 2021-12-08 | Algorithm evaluation method, device, gateway and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111490278.8A CN114116520B (en) | 2021-12-08 | 2021-12-08 | Algorithm evaluation method, device, gateway and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114116520A true CN114116520A (en) | 2022-03-01 |
| CN114116520B CN114116520B (en) | 2023-05-26 |
Family
ID=80367443
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111490278.8A Active CN114116520B (en) | 2021-12-08 | 2021-12-08 | Algorithm evaluation method, device, gateway and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114116520B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115145813A (en) * | 2022-06-30 | 2022-10-04 | 北京达佳互联信息技术有限公司 | Model evaluation method and device |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140270494A1 (en) * | 2013-03-15 | 2014-09-18 | Sri International | Computer vision as a service |
| US20150379429A1 (en) * | 2014-06-30 | 2015-12-31 | Amazon Technologies, Inc. | Interactive interfaces for machine learning model evaluations |
| CN110362492A (en) * | 2019-07-18 | 2019-10-22 | 腾讯科技（深圳）有限公司 | Intelligent algorithm test method, device, server, terminal and storage medium |
| CN111176961A (en) * | 2019-12-05 | 2020-05-19 | 腾讯科技（深圳）有限公司 | Application program testing method and device and storage medium |
| CN112463301A (en) * | 2020-11-30 | 2021-03-09 | 常州微亿智造科技有限公司 | Container-based model training test tuning and deployment method and device |
| US20210089433A1 (en) * | 2019-09-20 | 2021-03-25 | Unity Technologies Sf | Method and system for artificial intelligence based video game testing |
| CN112631577A (en) * | 2021-03-10 | 2021-04-09 | 北京瑞莱智慧科技有限公司 | Model scheduling method, model scheduler and model safety test platform |
| CN112905459A (en) * | 2021-02-08 | 2021-06-04 | 中国工商银行股份有限公司 | Service interface testing method and device, electronic equipment and storage medium |
| CN113377664A (en) * | 2021-06-25 | 2021-09-10 | 上海商汤科技开发有限公司 | Model testing method and device, electronic device and storage medium |
| CN113380041A (en) * | 2021-08-12 | 2021-09-10 | 深圳市城市交通规划设计研究中心股份有限公司 | Application scene testing system, method, equipment and storage medium based on vehicle-road collaborative simulation |
-
2021
- 2021-12-08 CN CN202111490278.8A patent/CN114116520B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140270494A1 (en) * | 2013-03-15 | 2014-09-18 | Sri International | Computer vision as a service |
| US20150379429A1 (en) * | 2014-06-30 | 2015-12-31 | Amazon Technologies, Inc. | Interactive interfaces for machine learning model evaluations |
| CN110362492A (en) * | 2019-07-18 | 2019-10-22 | 腾讯科技（深圳）有限公司 | Intelligent algorithm test method, device, server, terminal and storage medium |
| US20210089433A1 (en) * | 2019-09-20 | 2021-03-25 | Unity Technologies Sf | Method and system for artificial intelligence based video game testing |
| CN111176961A (en) * | 2019-12-05 | 2020-05-19 | 腾讯科技（深圳）有限公司 | Application program testing method and device and storage medium |
| CN112463301A (en) * | 2020-11-30 | 2021-03-09 | 常州微亿智造科技有限公司 | Container-based model training test tuning and deployment method and device |
| CN112905459A (en) * | 2021-02-08 | 2021-06-04 | 中国工商银行股份有限公司 | Service interface testing method and device, electronic equipment and storage medium |
| CN112631577A (en) * | 2021-03-10 | 2021-04-09 | 北京瑞莱智慧科技有限公司 | Model scheduling method, model scheduler and model safety test platform |
| CN113377664A (en) * | 2021-06-25 | 2021-09-10 | 上海商汤科技开发有限公司 | Model testing method and device, electronic device and storage medium |
| CN113380041A (en) * | 2021-08-12 | 2021-09-10 | 深圳市城市交通规划设计研究中心股份有限公司 | Application scene testing system, method, equipment and storage medium based on vehicle-road collaborative simulation |
Non-Patent Citations (2)
| Title |
|---|
| E.J WEYUKER: "Experience with performance testing of software systems:issues,an approach,and case study", 《IEEE TRANSACTIONS ON SOFTWARE ENGINEERING》 * |
| 吕晨: "基于机器学***台的研究与实现", 《中国优秀硕士学位论文全文数据库 (信息科技辑)》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115145813A (en) * | 2022-06-30 | 2022-10-04 | 北京达佳互联信息技术有限公司 | Model evaluation method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114116520B (en) | 2023-05-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111083225B (en) | Data processing method and device in Internet of things platform and Internet of things platform | |
| CN109684847B (en) | Automatic repairing method, device, equipment and storage medium for script loopholes | |
| CN108628748B (en) | Automatic test management method and automatic test management system | |
| CN111683066B (en) | Heterogeneous system integration method, heterogeneous system integration device, computer equipment and storage medium | |
| CN107957940B (en) | Test log processing method, system and terminal | |
| CN110647471B (en) | Interface test case generation method, electronic device and storage medium | |
| CN108363588B (en) | Method for realizing interaction between web and native function, electronic device and readable storage medium | |
| CN111694749A (en) | Automatic interface testing method and device, computer equipment and readable storage medium | |
| CN104268241A (en) | Attribute display method in configuration database | |
| CN111813701A (en) | HTTP-based interface testing method and device, computer equipment and storage medium | |
| CN116755844B (en) | Data processing method, device and equipment of simulation engine and storage medium | |
| CN110647415A (en) | Database detection method, device and equipment and computer readable storage medium | |
| CN103324567B (en) | A kind of adjustment method of application engine and debug system | |
| CN114116520B (en) | Algorithm evaluation method, device, gateway and storage medium | |
| CN111752820B (en) | gRPC interface pressure test method, computer equipment and storage medium | |
| CN112256557B (en) | Program regression testing method, device, system, computer equipment and storage medium | |
| CN112035163A (en) | Software development kit configuration method, device and storage medium | |
| CN112235124A (en) | Method and device for configuring pico-cell, storage medium and electronic device | |
| CN110881030A (en) | Logstack-based method and device for recording operation logs of web service administrator | |
| CN116719735A (en) | Test case generation method and device | |
| CN111400027A (en) | Distributed task processing method, device and system | |
| CN112699000A (en) | Data processing method and device, readable storage medium and electronic equipment | |
| US9479396B2 (en) | XML based generic UNIX discovery framework | |
| CN114003484A (en) | Log output method, device and equipment for interface test and storage medium | |
| CN111506846A (en) | Web page generation method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant after: Douyin Vision Co.,Ltd. Address before: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant before: Tiktok vision (Beijing) Co.,Ltd. Address after: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant after: Tiktok vision (Beijing) Co.,Ltd. Address before: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant before: BEIJING BYTEDANCE NETWORK TECHNOLOGY Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |