CN113051162A - Experiment testing method, device, computer system and readable storage medium - Google Patents
Experiment testing method, device, computer system and readable storage medium Download PDFInfo
- Publication number
- CN113051162A CN113051162A CN202110304767.3A CN202110304767A CN113051162A CN 113051162 A CN113051162 A CN 113051162A CN 202110304767 A CN202110304767 A CN 202110304767A CN 113051162 A CN113051162 A CN 113051162A
- Authority
- CN
- China
- Prior art keywords
- experiment
- tested
- target
- information
- identification node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000002474 experimental method Methods 0.000 title claims abstract description 316
- 238000012360 testing method Methods 0.000 title claims abstract description 78
- 238000004590 computer program Methods 0.000 claims abstract description 25
- 238000000034 method Methods 0.000 claims description 41
- 230000015654 memory Effects 0.000 claims description 11
- 238000012545 processing Methods 0.000 claims description 11
- 238000010276 construction Methods 0.000 claims description 10
- 238000004422 calculation algorithm Methods 0.000 claims description 8
- 238000004806 packaging method and process Methods 0.000 claims description 4
- 230000004044 response Effects 0.000 description 17
- 230000006870 function Effects 0.000 description 16
- 238000010586 diagram Methods 0.000 description 12
- 238000004891 communication Methods 0.000 description 8
- 238000013461 design Methods 0.000 description 8
- 238000011156 evaluation Methods 0.000 description 5
- 238000002955 isolation Methods 0.000 description 5
- 238000010998 test method Methods 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 4
- 238000011161 development Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000007726 management method Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000002829 reductive effect Effects 0.000 description 3
- 108010001267 Protein Subunits Proteins 0.000 description 2
- 238000009933 burial Methods 0.000 description 2
- 238000004140 cleaning Methods 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 239000000758 substrate Substances 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000000670 limiting effect Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3684—Test management for test design, e.g. generating new test cases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3688—Test management for test execution, e.g. scheduling of test suites
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3696—Methods or tools to render software testable
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The present disclosure provides an experimental testing method, comprising: constructing a hash ring, wherein the hash ring comprises at least one experiment identification node, and each experiment identification node corresponds to an experiment to be tested; determining a target experiment identification node on the hash ring, wherein the target experiment identification node is an experiment identification node corresponding to attribute identification information, and the attribute identification information is information carried in user request information; and dynamically loading and executing the running file of the target experiment to be tested, which is matched with the target experiment identification node. The disclosure also provides an experimental testing device, a computer system, a readable storage medium and a computer program product.
Description
Technical Field
The present disclosure relates to the field of computer technologies and internet technologies, and in particular, to an experimental testing method, an experimental testing device, a computer system, a readable storage medium, and a computer program product.
Background
During the design, development and operation of internet products, a plurality of product design and operation schemes are often selected, for example, in the design of product interface functions, a selection question of whether a certain button is red or blue, or left or right is given. In the face of the selection problem, the solution may be to introduce an experimental testing tool, and perform online operation on the schemes simultaneously to perform testing, so as to collect and verify the experience feedback of the user on different designs, thereby assisting the developer in making an appropriate decision.
In implementing the disclosed concept, the inventors found that there are at least the following problems in the related art: under the condition that a plurality of factors influence each other, the traditional experiment test method cannot support independence between experiments, so that the traditional experiment test result is inaccurate.
Disclosure of Invention
In view of the above, the present disclosure provides an experimental testing method, an experimental testing device, a computer system, a readable storage medium, and a computer program product.
One aspect of the present disclosure provides an experimental testing method, comprising:
constructing a hash ring, wherein the hash ring comprises at least one experiment identification node, and each experiment identification node corresponds to an experiment to be tested;
determining a target experiment identification node on the hash ring, wherein the target experiment identification node is an experiment identification node corresponding to attribute identification information, and the attribute identification information is information carried in user request information; and
and dynamically loading and executing the running file of the target experiment to be tested, which is matched with the target experiment identification node.
According to an embodiment of the present disclosure, determining a target experiment identification node on a hash ring includes:
responding to the user request information, and acquiring a hash ring;
processing the attribute identification information by using a consistent hash algorithm to obtain a hash result;
determining a target location on the hash ring corresponding to the hash result based on the hash result; and
and determining target experiment identification nodes on the hash ring according to a preset search rule based on the target position.
According to the embodiment of the disclosure, the target experiment to be tested matched with the target experiment identification node comprises a plurality of experiment modules to be tested, wherein each experiment module to be tested in the plurality of experiment modules to be tested comprises a first experiment version to be tested and a second experiment version to be tested;
wherein, determining the target experiment identification node on the hash ring further comprises:
and determining a target experiment version to be tested in each of a plurality of experiment modules to be tested in the target experiment to be tested based on the attribute identification information.
According to an embodiment of the present disclosure, wherein constructing the hash ring includes:
acquiring experiment configuration information of an experiment to be tested according to a preset time interval; and constructing a Hash ring based on the experiment configuration information of the experiment to be tested.
According to an embodiment of the present disclosure, the method further includes:
determining target experiment information to be tested matched with the target experiment identification node; and
storing target experimental information to be tested into a cache;
the dynamic loading and execution of the running file of the target experiment to be tested matched with the target experiment identification node comprises the following steps:
responding to the user request information, and acquiring target experiment information to be tested corresponding to the attribute identification information in the cache; and
and dynamically loading and executing the running files of the target experiment to be tested by utilizing the remote procedure call request based on the information of the target experiment to be tested, wherein the running files of the target experiment to be tested comprise at least one packaged running file.
According to an embodiment of the present disclosure, the method further includes:
under the condition that the target experiment information to be tested in the cache is not obtained, determining the target experiment information to be tested according to a preset test rule; and the remote procedure call request is utilized to dynamically load and execute the operation file of the target experiment to be tested based on the target experiment information to be tested.
According to an embodiment of the present disclosure, the method further includes:
and carrying out data point burying on the data associated with the experiment to be tested on the hash ring.
Another aspect of the present disclosure provides an experimental testing device, comprising:
the system comprises a construction module, a test module and a test module, wherein the construction module is used for constructing a hash ring, the hash ring comprises at least one experiment identification node, and each experiment identification node corresponds to an experiment to be tested;
the determining module is used for determining a target experiment identification node on the hash ring, wherein the target experiment identification node is an experiment identification node corresponding to the attribute identification information, and the attribute identification information is information carried in the user request information; and
and the execution module is used for dynamically loading and executing the running file of the target experiment to be tested, which is matched with the target experiment identification node.
Yet another aspect of the present disclosure provides a computer system comprising:
one or more processors;
a memory for storing one or more programs,
wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method described above.
Yet another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions for implementing the above-described method when executed.
Yet another aspect of the disclosure provides a computer program product comprising computer executable instructions that when executed perform the method described above.
According to the embodiment of the present disclosure, since an experimental test method is adopted, it includes: constructing a hash ring, wherein the hash ring comprises at least one experiment identification node, and each experiment identification node corresponds to an experiment to be tested; determining a target experiment identification node on the hash ring, wherein the target experiment identification node is an experiment identification node corresponding to attribute identification information, and the attribute identification information is information carried in user request information; and the technical means of dynamically loading and executing the running file of the target experiment to be tested matched with the target experiment identification node are adopted, the user flow is shunted by utilizing the Hash ring, the flow isolation among experiments is supported, and the complementary influence among the experiments to be tested is ensured. Therefore, the technical problem that the prior art cannot support independence among experiments and cause the traditional experiment test result to be inaccurate is at least partially solved, and the technical effects that the experiments to be tested are supported to be independent and not influenced mutually and the test result is accurate and effective are achieved.
Drawings
The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure with reference to the accompanying drawings, in which:
FIG. 1 schematically illustrates an exemplary system architecture to which the experimental testing methods and apparatus of the present disclosure may be applied;
FIG. 2 schematically illustrates an application scenario of an experimental testing method according to an embodiment of the present disclosure;
FIG. 3 schematically illustrates a flow chart of an experimental testing method according to an embodiment of the present disclosure;
fig. 4 schematically illustrates a schematic diagram of a hash ring structure according to another embodiment of the present disclosure;
FIG. 5 schematically illustrates a schematic diagram of determining a target experimental identification node on a hash ring according to another embodiment of the present disclosure;
FIG. 6 schematically illustrates a flow chart of an experimental testing method according to another embodiment of the present disclosure;
FIG. 7 schematically illustrates a block diagram of an experimental testing device according to an embodiment of the present disclosure; and
FIG. 8 schematically illustrates a block diagram of a computer system suitable for implementing an experimental testing method according to an embodiment of the present disclosure.
Detailed Description
Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.
All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.
Where a convention analogous to "at least one of A, B and C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B and C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). Where a convention analogous to "A, B or at least one of C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B or C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).
The embodiment of the disclosure provides an experimental testing method. The method comprises the steps of constructing a hash ring, wherein the hash ring comprises at least one experiment identification node, and each experiment identification node corresponds to an experiment to be tested; determining a target experiment identification node on the hash ring, wherein the target experiment identification node is an experiment identification node corresponding to attribute identification information, and the attribute identification information is information carried in user request information; and dynamically loading and executing the running file of the target experiment to be tested, which is matched with the target experiment identification node.
According to the embodiment of the disclosure, the experimental testing method can support flow isolation among the experiments to be tested, and ensure mutual independence and mutual noninfluency among the experiments to be tested.
Fig. 1 schematically illustrates an exemplary system architecture 100 to which experimental testing methods and apparatus may be applied, according to an embodiment of the disclosure. It should be noted that fig. 1 is only an example of a system architecture to which the embodiments of the present disclosure may be applied to help those skilled in the art understand the technical content of the present disclosure, and does not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments or scenarios.
As shown in fig. 1, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 serves as a medium for providing communication links between the terminal devices 101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired and/or wireless communication links, and so forth.
The user may use the terminal devices 101, 102, 103 to interact with the server 105 via the network 104 to receive or send messages or the like. The terminal devices 101, 102, 103 may have installed thereon various communication client applications, such as a shopping-like application, a web browser application, a search-like application, an instant messaging tool, a mailbox client, and/or social platform software, etc. (by way of example only).
The terminal devices 101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 105 may be a server providing various services, such as a background management server (for example only) providing support for websites browsed by users using the terminal devices 101, 102, 103. The background management server may analyze and perform other processing on the received data such as the user request, and feed back a processing result (e.g., a webpage, information, or data obtained or generated according to the user request) to the terminal device.
It should be noted that the experimental testing method provided by the embodiment of the present disclosure may be generally executed by the server 105. Accordingly, the experimental testing device provided by the embodiments of the present disclosure may be generally disposed in the server 105. The experimental testing method provided by the embodiment of the present disclosure may also be performed by a server or a server cluster that is different from the server 105 and is capable of communicating with the terminal devices 101, 102, 103 and/or the server 105. Accordingly, the experimental testing device provided by the embodiment of the present disclosure may also be disposed in a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, and 103 and/or the server 105.
It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Fig. 2 schematically illustrates an application scenario of an experimental testing method according to an embodiment of the present disclosure.
As shown in fig. 2, with the development of computer technology and network transmission technology, intelligent response becomes an important means for serving users. The system not only can answer the questions of the user according to the knowledge base, but also can provide the function of business information inquiry, for example, an intelligent response system applied to invoicing, hastening delivery, checking logistics, changing addresses and the like provided in the E-commerce field, an intelligent response system applied to consultation of matters such as medical insurance, social insurance, public deposit, admission and the like provided in the government affairs field, or an intelligent response system applied to telephone charge checking, flow checking, package buying, failure reporting, password changing and the like in the telecommunication field. In addition, intelligent response media such as intelligent robot voice response, telephone intelligent response or information sending intelligent response can be adopted.
By using the experimental testing method of the embodiment of the disclosure, the cost is saved, and meanwhile, the work efficiency and the customer satisfaction are greatly improved due to accurate and quick automatic reply.
In the product interface function design of the intelligent response system, the problem of selecting among a plurality of excellent designs is often faced. The test of the AB experiment can be carried out by introducing an AB test tool, so that the experience feedback of the user to different designs can be collected and verified, and the correct selection of a product design or an operator can be assisted. For example, the new model and the old model are analyzed in the AB experiment in table 1. For each newly developed function or model optimization, the split codes of the AB experiment, such as the split codes of the old model a and the new model B, may be added; and the ratio of tests of the configured a experiment and B experiment, for example, may be designed as shown in table 1, and the split ratio 1 of the a experiment and B experiment is performed: 1; after the function is on line, based on data buried points of the experiment A and the experiment B and corresponding user evaluation, experience feedback of the function or the model is obtained, and based on user feedback data (user requirements are changed into manual quantity, and user evaluation data of response praise or step on by the user), whether the user experience is positively influenced by the new scheme is analyzed, so that whether the new model or the new function is on line in full quantity is decided.
TABLE 1
| Model identification AB | AB ratio | User rating |
| Old model A | 50％ | Praise: 20%, stepping: 80 percent of |
| New model B | 50％ | Praise: 60%, stepping: 40 percent of |
However, in the course of implementing the present disclosure, it is found that, in the intelligent response system, the main modules may include intention identification (rule identification, model identification), dialogue management (slot filling decision, task response decision, policy question and answer matching), answer management (answer acquisition, answer rendering, answer pushing), and the like. The existing AB experiment test scheme needs AB experiments and adds shunting codes of the AB experiments, so that the existing AB experiment scheme cannot uniformly control all AB experiments. That is, each module configures the AB experiment by itself, configures the shunt ratio of the respective AB experiment by itself, all the AB experiments take effect simultaneously and affect each other, and other factors cannot be eliminated, a single-factor AB experiment cannot be performed, and a designated multi-factor AB experiment cannot be supported.
According to the embodiment of the disclosure, a test experiment method is provided, and by using the experiment test method of the embodiment of the disclosure, flow isolation among experiments to be tested can be supported, and mutual independence and mutual noninfluency among the experiments to be tested are ensured.
Fig. 3 schematically illustrates a flow chart of an experimental testing method according to an embodiment of the present disclosure.
As shown in fig. 3, the method includes operations S310 to S330.
In operation S310, a hash ring is constructed, where the hash ring includes at least one experiment identification node thereon, and each experiment identification node corresponds to an experiment to be tested.
According to the embodiment of the disclosure, an annular linked list, such as a hash ring, can be constructed based on an experiment to be tested, wherein the hash value range of the hash ring is 0-232I.e. can be mapped onto the hash ring by means of the hash value.
According to the embodiment of the disclosure, a plurality of virtual nodes are hashed on a hash ring, and N virtual nodes can be identified as at least one experiment identification node, where N is an integer greater than or equal to 1. That is, a plurality of virtual nodes may be identified as one experiment identification node, and each experiment identification node corresponds to an experiment to be tested.
In operation S320, a target experiment identification node on the hash ring is determined, where the target experiment identification node is an experiment identification node corresponding to the attribute identification information, and the attribute identification information is information carried in the user request information.
According to the embodiment of the disclosure, after the user request information is received, the target experiment identification node on the hash ring is determined based on the attribute identification information carried in the user request information, so that the user request is shunted to perform different experiences of experiments to be tested.
In operation S330, a run file of a target experiment to be tested, which matches the target experiment identification node, is dynamically loaded and executed.
According to the embodiment of the disclosure, each experiment identification node on the hash ring corresponds to one experiment to be tested, after the target experiment identification node is determined, the running file of the target experiment to be tested, which is matched with the target experiment identification node, can be determined, and the running of the experiment to be tested can be carried out by loading and executing the running file.
According to the embodiment of the disclosure, compared with a static loading mode, the embodiment of the disclosure adopts a dynamic loading mode, and effectively realizes flexible loading and execution of various experiments to be tested.
According to other embodiments of the present disclosure, the user request splitting may also be implemented using splitting codes, for example, splitting by an IF-ELSE syntax decision branch. After the AB experiment test is completed, the AB experiment shunting codes need to be cleaned regularly, useless branches are removed, and on-line branches are reserved. Under the condition of not cleaning, abandoned codes after the AB experiment test is finished are accumulated, and the code readability is poor; under the condition of cleaning, related service codes need to be modified, so that a large amount of development and test work is introduced, the working efficiency is reduced, and the labor cost is increased.
According to the embodiment of the disclosure, by using the experimental test method disclosed by the disclosure, the user request is shunted by using the hash ring, so that a shunting mode of shunting by using shunting codes is abandoned, redundant codes are not generated, the working efficiency is improved, and the labor cost is reduced; and mutual independence between experiments to be tested is realized, the mutual exclusivity of shunting is ensured, the problem of mutual influence is avoided, and a single-factor or multi-factor combined test mode is supported.
The method shown in fig. 3 is further described with reference to fig. 4-6 in conjunction with specific embodiments.
Fig. 4 schematically illustrates a schematic diagram of a hash ring structure according to another embodiment of the present disclosure.
As shown in fig. 4, the hash ring includes a plurality of experiment identification nodes, each experiment identification node may be allocated with a plurality of virtual nodes, and after calculating the attribute identification information requested by the user by using the hash function, the obtained hash value is mapped onto the corresponding virtual node, that is, the corresponding experiment identification node, such as experiment 1, experiment 2, and the like to be tested.
According to the embodiment of the disclosure, the experiment to be tested can be configured at any time according to the actual operation condition, for example, the off-line or on-line of a certain model in the experiment to be tested, the update of a certain version, and the like.
According to the embodiment of the disclosure, experimental configuration information of an experiment to be tested can be acquired according to a preset time interval; and constructing a Hash ring based on the experiment configuration information of the experiment to be tested.
The Hash ring is constructed by the embodiment of the disclosure, so that the Hash ring can be updated in time, dynamic modification and increase and decrease of the experiment to be tested are supported by dynamic loading and updating, the Hash ring is adaptive to an operation test scheme updated at any time, the adjustment is flexible, and the operability is strong.
Fig. 5 schematically illustrates a schematic diagram of determining a target experiment identification node on a hash ring according to another embodiment of the present disclosure.
In order to accurately adapt to the hash ring updated at any time, the hash ring can be acquired in response to the user request information, so that the experimental identification node on the determined hash ring is the latest operation test scheme. After the hash ring is obtained, processing the attribute identification information by using a consistent hash algorithm to obtain a hash result; based on the hash result, a target location on the hash ring corresponding to the hash result is determined. Such as sessionID-userA and sessionID-userB in fig. 5, that identifies the target location on the hash ring determined based on the user's attribute identification information. It should be noted that the target location may be a location of the experimental identification node on the hash ring, but may not be the location of the experimental identification node on the hash ring.
According to the embodiment of the disclosure, the preset search rule may be that, when the target position is the position of the experiment identification node on the hash ring, the experiment identification node corresponding to the target position is determined as the target experiment identification node. And under the condition that the target position is not the position of the experiment identification node on the hash ring, clockwise searching a nearest experiment identification node based on the target position, and determining the nearest experiment identification node as the target experiment identification node. Such as experiment 2 and experiment 4 in fig. 5.
According to the embodiment of the disclosure, the attribute identification information may be session ID (identity document) information of the user, and based on the session ID information, it can be ensured that each sentence of session can execute distributed consistent hash calculation.
According to the embodiment of the disclosure, the session ID information of the user can be processed by using a consistent hash algorithm, and a hash function is called to perform calculation to obtain a hash result.
According to other embodiments of the present disclosure, direct hash modulo calculation experiment splitting may also be employed. However, by adopting the hash modulo calculation, all ongoing session test experiments are affected under the condition that the to-be-tested experiments are dynamically increased or decreased, so that a large number of same sessions (i.e., user requests) enter different to-be-tested experiments. The problem that when user evaluation data statistics is carried out, the same session occurs in different to-be-tested experiments, and further the session evaluation data occurring in a plurality of to-be-tested experiments cannot be judged to be the evaluation of which to-be-tested experiment; in addition, it may also result in failure to evaluate the impact of the experimental handover to be tested on the user experience.
The embodiment of the disclosure calculates the user session ID by using a consistent Hash algorithm, thereby not only meeting the relative uniformity, but also supporting the requirement of minimum influence of experimental dynamic loading.
According to an optional embodiment of the present disclosure, the target experiment to be tested matched with the target experiment identification node may be a single-factor experiment, but is not limited thereto, and may also be a multi-factor combination experiment. The multi-factor combined target experiment to be tested may be, for example, a target experiment to be tested including a plurality of experiment modules to be tested, wherein each of the plurality of experiment modules to be tested includes a first version of the experiment to be tested and a second version of the experiment to be tested (i.e., an AB experiment).
According to the embodiment of the disclosure, after the user request is shunted by using the hash ring, only the user session ID is shunted to a specific target experiment to be tested, and the version of the experiment to be tested in the single experiment module to be tested in the specific target experiment to be tested is not determined.
According to an embodiment of the present disclosure, a final determination result of the target experiment to be tested may be as shown in table 2, and a target experiment version to be tested in each of the plurality of experiment modules to be tested in the target experiment to be tested may be determined based on the attribute identification information. For example, the target experimental information to be tested of sessionId-useRA is determined as 2020-05-20-1 model version of the model identification module, and the target experimental information to be tested of sessionId-useRB is determined as 2020-10-08-1 model version of the model identification module and 2020-06-02-1 model version of the slot filling decision module which are executed in sequence.
TABLE 2
According to the embodiment of the disclosure, after the target experiment information to be tested matched with the target experiment identification node is determined, the target experiment information to be tested can be stored in the cache. As in table 3, the target experimental information to be tested may be stored in the distributed cache Redis as follows.
TABLE 3
| Key | Value |
| sessionId-userA model identification module | 2020-05-20-1 model |
| sessionId-userB model identification module | 2020-10-08-1 model |
| sessionId-userB _ Slot filling decision Module | 2020-06-02-1 groove filling |
According to an embodiment of the present disclosure, dynamically loading and executing a run file of a target experiment to be tested that matches a target experiment identification node may include the following operations.
Responding to the user request information, and acquiring target experiment information to be tested corresponding to the attribute identification information in the cache; and dynamically loading and executing the running file of the target experiment to be tested by utilizing the remote procedure call request based on the information of the target experiment to be tested, wherein the running file of the target experiment to be tested comprises at least one packaged running file.
According to the embodiment of the disclosure, the operation of dynamically loading and executing the operation file of the target experiment to be tested, which is matched with the target experiment identification node, may support the principle of dynamically loading a Class file (Java Class file, a binary file) from a Jar package (compressed into a file package, or packed into a Jar package) by using a Class loader based on Java (a high level programming language), and load and execute the operation file of the target experiment to be tested, which is matched with the target experiment identification node, according to the target experiment identification node.
According to the embodiment of the disclosure, the Java running source code of the target experiment to be tested, for example, the running Class, in the embodiment of the disclosure may be compiled and generated into a Class file, then the Class file is packaged into a Jar package (i.e., a package running file), and during execution, a Class loader is used for dynamic loading.
According to the embodiment of the disclosure, different experimental versions to be tested can be loaded into different Jar packages respectively, and the name of each Jar package can be named by the name of the experimental version to be tested; deployed into a specified directory. It should be noted that, in the dynamic loading and executing mode of the embodiment of the present disclosure, a core logic code is not used, and only a remote procedure call request, that is, an RPC service request, is used to call the Jar package file.
For example, the NLU module framework application server calls 2020-05-20-1 model jar, 2020-10-08-1 model jar and the like by using RPC service requests.
According to the embodiment of the disclosure, the running of the experiment to be tested is supported by using the running file at the Jar package level, dynamic loading is carried out, the coupling of the experiment code and the service code is avoided, the service code is not required to be modified and cleaned after the subsequent application module is on line, the workload of development and testing is reduced, and the working efficiency is improved.
According to an optional embodiment of the present disclosure, the designated directory may be periodically scanned at preset time intervals, for example, and the newly deployed Jar package is used as an experiment to be tested to perform experiment configuration, so as to obtain experiment configuration information, and the experiment configuration information is constructed in the hash ring.
By using the optional embodiment of the disclosure, a new operation file is obtained based on a preset time interval, and the operation file is constructed on the hash ring, so that a test experiment can be performed in time, and online application can be performed.
According to the embodiment of the disclosure, the target experiment information to be tested can be stored in the cache, under the condition of responding to the user request information, the target experiment information to be tested corresponding to the user session ID in the cache is obtained, and the running file of the target experiment to be tested is dynamically loaded and executed from the specified path by utilizing the remote call request.
According to other embodiments of the disclosure, under the condition that the target experiment information to be tested in the cache is not obtained, for example, the cache is empty, the target experiment information to be tested is determined according to a preset test rule; and the remote procedure call request is utilized to dynamically load and execute the operation file of the target experiment to be tested based on the target experiment information to be tested.
According to an embodiment of the present disclosure, the preset test rule may be a Jar package file that loads and executes a latest version of an experiment to be tested. By utilizing the embodiment of the disclosure, flexible calling can be realized.
According to the embodiment of the disclosure, when the experiment testing method of the embodiment of the disclosure is performed, data related to the experiment to be tested on the hash ring is subjected to data point burying.
According to an embodiment of the present disclosure, the data associated with the experiment to be tested may be session context information, a session ID, a user IP, and the like.
According to the embodiment of the present disclosure, whether a certain module or version performs data point burying on relevant data in the operation process in the embodiment of the present disclosure may be determined by determining whether the operation version is an implementation experiment version to be tested on a hash ring, that is, by determining whether the operation version is a test stage.
According to the embodiment of the disclosure, data embedding is controllable by utilizing the to-be-tested experiment on the hash ring, so that the associated data of the to-be-tested experiment on the hash ring in the version or the module can be realized by judging the to-be-tested experiment on the hash ring, namely, the data embedding is produced when the AB experiment is started, and the data embedding is stopped after the AB experiment is closed. Therefore, the problems that data are still buried and a large amount of data storage space is occupied in the online stage of the test experiment instead of the test stage are solved.
Fig. 6 schematically illustrates a flow chart of an experimental testing method according to another embodiment of the present disclosure.
As shown in fig. 6, the AB experiment configuration component, the AB experiment shunting component, and the module dynamic loading component may be utilized to perform the experimental testing method of the embodiments of the present disclosure.
According to an embodiment of the present disclosure, the experimental testing method may include the following operations.
In operation S611, the AB experiment configuration component is utilized to perform an experiment configuration on the module to be tested and the version to be tested.
In operation S612, the configured experimental configuration information is stored in a configuration database.
In operation S621, the AB experiment offloading module is used to start a timing task, read experiment configuration information from the configuration database, and determine whether the experiment configuration information is modified, such as updated, deleted, or added.
In operation S622, in the case that the experiment configuration information is changed, the AB experiment offloading component is used to construct a Hash (Hash) ring based on the new experiment configuration information, and the Hash ring is cached in the cache Redis.
In operation S623, user request information is received.
In operation S624, attribute identification information, such as session ID information, in the user request information is processed using a consistent hashing algorithm.
In operation S625, the hash ring is read from the cache, and split, and target experimental information to be tested (i.e., AB experimental result) corresponding to the attribute identification information in the user request is determined.
In operation S626, the target experimental information to be tested (i.e., the AB experimental result) is cached in the cache Redis.
In operation S627, data burial is performed on data associated with the target experimental information to be tested.
In operation S631, the module dynamic loading component introduces the frame application RPC entry, and reads the target experiment information to be tested (i.e., the AB experiment result) from the cache based on the user request information.
In operation S632, the core logic Jar package file is loaded with an RPC request.
In operation S633, the core logic Jar package file is executed and a test result is returned.
By using the experimental test method of the embodiment of the disclosure, based on the consistent Hash algorithm, the mutual isolation of AB experimental shunts is realized; the method can support single-factor or multi-factor experiments and ensure that experimental data do not influence each other.
In addition, the embodiment of the disclosure also supports AB experiments at the Jar package level, avoids the coupling of the experiment codes and the service codes, and does not need to clean the AB experiment codes when the service codes are on line.
In addition, the data embedding operation of the embodiment of the disclosure is controllable, the data embedding is performed when the AB experiment is started, and the data embedding is stopped after the AB experiment is closed.
In conclusion, the experimental testing method disclosed by the embodiment of the disclosure has the advantages of improving the testing efficiency, reducing the labor cost, along with high flexibility and strong real-time performance.
Fig. 7 schematically illustrates a block diagram of an experimental testing device according to an embodiment of the present disclosure.
As shown in fig. 7, the experimental testing device 700 includes a construction module 710, a determination module 720, and an execution module 730.
A constructing module 710, configured to construct a hash ring, where the hash ring includes at least one experiment identification node, and each experiment identification node corresponds to an experiment to be tested;
a determining module 720, configured to determine a target experiment identification node on the hash ring, where the target experiment identification node is an experiment identification node corresponding to the attribute identification information, and the attribute identification information is information carried in the user request information; and
and the executing module 730 is configured to dynamically load and execute the running file of the target experiment to be tested, which is matched with the target experiment identification node.
According to the embodiment of the disclosure, flow isolation among experiments to be tested can be supported, and mutual independence and mutual noninfluency among the experiments to be tested are ensured.
According to an embodiment of the present disclosure, a determination module includes a first response unit, a processing unit, a first determination unit, and a second determination unit.
The first response unit is used for responding to the user request information and acquiring the hash ring;
the processing unit is used for processing the attribute identification information by utilizing a consistent hash algorithm to obtain a hash result;
a first determination unit configured to determine, based on the hash result, a target position on the hash ring corresponding to the hash result; and
and the second determining unit is used for determining the target experiment identification node on the hash ring according to a preset searching rule based on the target position.
According to the embodiment of the disclosure, the target experiment to be tested matched with the target experiment identification node comprises a plurality of experiment modules to be tested, wherein each experiment module to be tested in the plurality of experiment modules to be tested comprises a first experiment version to be tested and a second experiment version to be tested.
According to an embodiment of the present disclosure, the determining module further comprises a third determining unit. The third determining unit is configured to determine, based on the attribute identification information, a target to-be-tested experiment version in each of a plurality of to-be-tested experiment modules in a target to-be-tested experiment.
According to an embodiment of the present disclosure, a construction module includes an acquisition unit and a construction unit.
The device comprises an acquisition unit, a test unit and a test unit, wherein the acquisition unit is used for acquiring the experiment configuration information of an experiment to be tested according to a preset time interval; and
and the construction unit is used for constructing a Hash ring based on the experimental configuration information of the experiment to be tested.
According to an embodiment of the present disclosure, the experimental testing device 700 further includes a second determination module and a saving module.
The second determining module is used for determining target experiment information to be tested, which is matched with the target experiment identification node; and
and the storage module is used for storing the target experimental information to be tested into a cache.
According to an embodiment of the present disclosure, the execution module includes a second response unit and an execution unit.
The second response unit is used for responding to the user request information and acquiring target to-be-tested experimental information corresponding to the attribute identification information in the cache; and
and the execution unit is used for dynamically loading and executing the running file of the target experiment to be tested by utilizing the remote procedure call request based on the information of the target experiment to be tested, wherein the running file of the target experiment to be tested comprises at least one packaging running file.
According to an embodiment of the present disclosure, the experimental test device 700 further comprises a third determination module. The third determining module is used for determining the target experiment information to be tested according to a preset testing rule under the condition that the target experiment information to be tested in the cache is not obtained; and the remote procedure call request is utilized to dynamically load and execute the operation file of the target experiment to be tested based on the target experiment information to be tested.
According to an embodiment of the present disclosure, the experimental testing device 700 further comprises a data burial point module. The data point burying module is used for performing data point burying on data associated with the experiment to be tested on the hash ring.
Any number of modules, sub-modules, units, sub-units, or at least part of the functionality of any number thereof according to embodiments of the present disclosure may be implemented in one module. Any one or more of the modules, sub-modules, units, and sub-units according to the embodiments of the present disclosure may be implemented by being split into a plurality of modules. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in any other reasonable manner of hardware or firmware by integrating or packaging a circuit, or in any one of or a suitable combination of software, hardware, and firmware implementations. Alternatively, one or more of the modules, sub-modules, units, sub-units according to embodiments of the disclosure may be at least partially implemented as a computer program module, which when executed may perform the corresponding functions.
For example, any number of the building module 710, the determining module 720 and the executing module 730 may be combined and implemented in one module/unit/sub-unit, or any one of the modules/units/sub-units may be split into a plurality of modules/units/sub-units. Alternatively, at least part of the functionality of one or more of these modules/units/sub-units may be combined with at least part of the functionality of other modules/units/sub-units and implemented in one module/unit/sub-unit. According to an embodiment of the present disclosure, at least one of the building module 710, the determining module 720, and the executing module 730 may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in hardware or firmware by any other reasonable manner of integrating or packaging a circuit, or in any one of or a suitable combination of software, hardware, and firmware. Alternatively, at least one of the building module 710, the determining module 720 and the executing module 730 may be at least partially implemented as a computer program module, which when executed may perform a corresponding function.
It should be noted that the experimental testing device part in the embodiments of the present disclosure corresponds to the experimental testing method part in the embodiments of the present disclosure, and the description of the experimental testing device part specifically refers to the experimental testing method part, and is not repeated herein.
FIG. 8 schematically illustrates a block diagram of a computer system suitable for implementing the above-described method, according to an embodiment of the present disclosure. The computer system illustrated in FIG. 8 is only one example and should not impose any limitations on the scope of use or functionality of embodiments of the disclosure.
As shown in fig. 8, a computer system 800 according to an embodiment of the present disclosure includes a processor 801 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)802 or a program loaded from a storage section 808 into a Random Access Memory (RAM) 803. The processor 801 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or associated chipset, and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), among others. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flows according to embodiments of the present disclosure.
In the RAM 803, various programs and data necessary for the operation of the system 800 are stored. The processor 801, the ROM 802, and the RAM 803 are connected to each other by a bus 804. The processor 801 performs various operations of the method flows according to the embodiments of the present disclosure by executing programs in the ROM 802 and/or RAM 803. Note that the programs may also be stored in one or more memories other than the ROM 802 and RAM 803. The processor 801 may also perform various operations of method flows according to embodiments of the present disclosure by executing programs stored in the one or more memories.
According to embodiments of the present disclosure, method flows according to embodiments of the present disclosure may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable storage medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 809 and/or installed from the removable medium 811. The computer program, when executed by the processor 801, performs the above-described functions defined in the system of the embodiments of the present disclosure. The systems, devices, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.
The present disclosure also provides a computer-readable storage medium, which may be contained in the apparatus/device/system described in the above embodiments; or may exist separately and not be assembled into the device/apparatus/system. The computer-readable storage medium carries one or more programs which, when executed, implement the method according to an embodiment of the disclosure.
According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. Examples may include, but are not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
For example, according to embodiments of the present disclosure, a computer-readable storage medium may include the ROM 802 and/or RAM 803 described above and/or one or more memories other than the ROM 802 and RAM 803.
Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for performing the method provided by the embodiments of the present disclosure, when the computer program product is run on an electronic device, the program code being adapted to cause the electronic device to carry out the experimental testing method provided by the embodiments of the present disclosure.
The computer program, when executed by the processor 801, performs the above-described functions defined in the system/apparatus of the embodiments of the present disclosure. The systems, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.
In one embodiment, the computer program may be hosted on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted in the form of a signal on a network medium, distributed, downloaded and installed via communication section 809, and/or installed from removable media 811. The computer program containing program code may be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.
In accordance with embodiments of the present disclosure, program code for executing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, these computer programs may be implemented using high level procedural and/or object oriented programming languages, and/or assembly/machine languages. The programming language includes, but is not limited to, programming languages such as Java, C + +, python, the "C" language, or the like. The program code may execute entirely on the user computing device, partly on the user device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. Those skilled in the art will appreciate that various combinations and/or combinations of features recited in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not expressly recited in the present disclosure. In particular, various combinations and/or combinations of the features recited in the various embodiments and/or claims of the present disclosure may be made without departing from the spirit or teaching of the present disclosure. All such combinations and/or associations are within the scope of the present disclosure.
The embodiments of the present disclosure have been described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described separately above, this does not mean that the measures in the embodiments cannot be used in advantageous combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be devised by those skilled in the art without departing from the scope of the present disclosure, and such alternatives and modifications are intended to be within the scope of the present disclosure.
Claims (11)
1. An experimental testing method comprising:
constructing a hash ring, wherein the hash ring comprises at least one experiment identification node, and each experiment identification node corresponds to an experiment to be tested;
determining a target experiment identification node on the hash ring, wherein the target experiment identification node is an experiment identification node corresponding to attribute identification information, and the attribute identification information is information carried in user request information; and
and dynamically loading and executing the running file of the target experiment to be tested, which is matched with the target experiment identification node.
2. The method of claim 1, wherein the determining a target experiment identification node on the hash ring comprises:
responding to the user request information, and acquiring the hash ring;
processing the attribute identification information by using a consistent hash algorithm to obtain a hash result;
determining, based on the hash result, a target location on the hash ring corresponding to the hash result; and
and determining a target experiment identification node on the hash ring according to a preset search rule based on the target position.
3. The method of claim 2, wherein the target experiment to be tested matched with the target experiment identification node comprises a plurality of experiment modules to be tested, wherein each experiment module to be tested in the plurality of experiment modules to be tested comprises a first experiment version to be tested and a second experiment version to be tested;
wherein the determining the target experiment identification node on the hash ring further comprises:
and determining a target experiment version to be tested in each experiment module to be tested in the plurality of experiment modules to be tested in the target experiment to be tested based on the attribute identification information.
4. The method of claim 2, wherein the constructing a hash ring comprises:
acquiring experiment configuration information of an experiment to be tested according to a preset time interval; and
and constructing the Hash ring based on the experimental configuration information of the experiment to be tested.
5. The method of claim 1, further comprising:
determining target experiment information to be tested matched with the target experiment identification node; and
storing the target experimental information to be tested into a cache;
the dynamic loading and execution of the running file of the target experiment to be tested matched with the target experiment identification node comprises the following steps:
responding to the user request information, and acquiring the target experiment information to be tested corresponding to the attribute identification information in the cache; and
and dynamically loading and executing the running file of the target experiment to be tested by utilizing a remote procedure call request based on the information of the target experiment to be tested, wherein the running file of the target experiment to be tested comprises at least one packaging running file.
6. The method of claim 5, further comprising:
under the condition that the target experiment information to be tested in the cache is not obtained, determining the target experiment information to be tested according to a preset test rule; and the operation of dynamically loading and executing the running file of the target experiment to be tested by utilizing the remote procedure call request based on the target experiment information to be tested is conveniently executed.
7. The method of claim 1, further comprising:
and carrying out data point burying on the data associated with the experiment to be tested on the hash ring.
8. An experimental test device comprising:
the system comprises a construction module, a test module and a test module, wherein the construction module is used for constructing a hash ring, the hash ring comprises at least one experiment identification node, and each experiment identification node corresponds to an experiment to be tested;
a determining module, configured to determine a target experiment identification node on the hash ring, where the target experiment identification node is an experiment identification node corresponding to attribute identification information, and the attribute identification information is information carried in user request information; and
and the execution module is used for dynamically loading and executing the running file of the target experiment to be tested, which is matched with the target experiment identification node.
9. A computer system, comprising:
one or more processors;
a memory for storing one or more programs,
wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1-7.
10. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to carry out the method of any one of claims 1 to 7.
11. A computer program product, comprising:
computer executable instructions for use when executed to implement the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110304767.3A CN113051162A (en) | 2021-03-22 | 2021-03-22 | Experiment testing method, device, computer system and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110304767.3A CN113051162A (en) | 2021-03-22 | 2021-03-22 | Experiment testing method, device, computer system and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113051162A true CN113051162A (en) | 2021-06-29 |
Family
ID=76514509
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110304767.3A Pending CN113051162A (en) | 2021-03-22 | 2021-03-22 | Experiment testing method, device, computer system and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113051162A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114942881A (en) * | 2022-05-10 | 2022-08-26 | 北京百度网讯科技有限公司 | Method and device for A/B test experiment, electronic device and medium |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140310390A1 (en) * | 2013-04-16 | 2014-10-16 | Amazon Technologies, Inc. | Asymmetric packet flow in a distributed load balancer |
| US20150081243A1 (en) * | 2013-03-20 | 2015-03-19 | Nec Laboratories America, Inc. | Setsudo: Pertubation-based Testing Framework for Scalable Distributed Systems |
| US20170091075A1 (en) * | 2015-09-30 | 2017-03-30 | Red Hat Inc. | Distributed system test automation framework |
| US20190243751A1 (en) * | 2018-02-02 | 2019-08-08 | Ca, Inc. | Automated selection of test cases for regression testing |
| CN110262964A (en) * | 2019-05-21 | 2019-09-20 | 深圳壹账通智能科技有限公司 | Test method, device, equipment and computer readable storage medium |
| US20190294532A1 (en) * | 2018-03-26 | 2019-09-26 | Hyperconnect, Inc. | Computer program and server for executing test group distribution method |
| CN111338967A (en) * | 2020-03-09 | 2020-06-26 | 京东数字科技控股有限公司 | Shunting test method and device, electronic equipment and storage medium |
| CN111459801A (en) * | 2020-03-09 | 2020-07-28 | 江苏苏宁物流有限公司 | RSF remote service interface function test method, module, computer equipment and storage medium |
| US10733088B1 (en) * | 2019-03-01 | 2020-08-04 | Keysight Technologies, Inc. | Methods, systems, and computer readable media for testing a network node or a related application programming interface using source code metadata |
| CN111988195A (en) * | 2019-05-24 | 2020-11-24 | 北京京东尚科信息技术有限公司 | Response scheme determination method, device, equipment and medium for packet test |
| CN112269743A (en) * | 2020-10-30 | 2021-01-26 | 湖南快乐阳光互动娱乐传媒有限公司 | A/B test method and device |
-
2021
- 2021-03-22 CN CN202110304767.3A patent/CN113051162A/en active Pending
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150081243A1 (en) * | 2013-03-20 | 2015-03-19 | Nec Laboratories America, Inc. | Setsudo: Pertubation-based Testing Framework for Scalable Distributed Systems |
| US20140310390A1 (en) * | 2013-04-16 | 2014-10-16 | Amazon Technologies, Inc. | Asymmetric packet flow in a distributed load balancer |
| US20170091075A1 (en) * | 2015-09-30 | 2017-03-30 | Red Hat Inc. | Distributed system test automation framework |
| US20190243751A1 (en) * | 2018-02-02 | 2019-08-08 | Ca, Inc. | Automated selection of test cases for regression testing |
| US20190294532A1 (en) * | 2018-03-26 | 2019-09-26 | Hyperconnect, Inc. | Computer program and server for executing test group distribution method |
| US10733088B1 (en) * | 2019-03-01 | 2020-08-04 | Keysight Technologies, Inc. | Methods, systems, and computer readable media for testing a network node or a related application programming interface using source code metadata |
| CN110262964A (en) * | 2019-05-21 | 2019-09-20 | 深圳壹账通智能科技有限公司 | Test method, device, equipment and computer readable storage medium |
| CN111988195A (en) * | 2019-05-24 | 2020-11-24 | 北京京东尚科信息技术有限公司 | Response scheme determination method, device, equipment and medium for packet test |
| CN111338967A (en) * | 2020-03-09 | 2020-06-26 | 京东数字科技控股有限公司 | Shunting test method and device, electronic equipment and storage medium |
| CN111459801A (en) * | 2020-03-09 | 2020-07-28 | 江苏苏宁物流有限公司 | RSF remote service interface function test method, module, computer equipment and storage medium |
| CN112269743A (en) * | 2020-10-30 | 2021-01-26 | 湖南快乐阳光互动娱乐传媒有限公司 | A/B test method and device |
Non-Patent Citations (1)
| Title |
|---|
| 安德鲁·凯莱赫 等: "《机器学习实践》", 30 April 2020, 机械工业出版社, pages: 154 - 155 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114942881A (en) * | 2022-05-10 | 2022-08-26 | 北京百度网讯科技有限公司 | Method and device for A/B test experiment, electronic device and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20110179398A1 (en) | Systems and methods for per-action compiling in contact handling systems | |
| US20110178946A1 (en) | Systems and methods for redundancy using snapshots and check pointing in contact handling systems | |
| CN110765024A (en) | Simulation test method, simulation test device, electronic equipment and computer-readable storage medium | |
| CN108255708B (en) | Method, device, storage medium and equipment for accessing production file in test environment | |
| CN113094081B (en) | Software release method, device, computer system and computer readable storage medium | |
| CN107644075B (en) | Method and device for collecting page information | |
| CN106941420A (en) | A kind of cluster application environment upgrade method and device | |
| US20190082050A1 (en) | Data structure driven interactive voice response (ivr) framework | |
| US10824446B2 (en) | Methods and systems for autocompletion | |
| CN113051162A (en) | Experiment testing method, device, computer system and readable storage medium | |
| CN113420010B (en) | Database management method, device, electronic equipment and storage medium | |
| US20200304434A1 (en) | Virtual agent portal integration of two frameworks | |
| CN109408256A (en) | Using call method and device | |
| US20210165681A1 (en) | Method and apparatus for processing a service of an abnormal server | |
| CN113191889A (en) | Wind control configuration method, configuration system, electronic device and readable storage medium | |
| US20140149488A1 (en) | System and method for engaging a mobile device | |
| CN111414154A (en) | Method and device for front-end development, electronic equipment and storage medium | |
| CN114640614B (en) | RPC interface automatic test method, system, electronic equipment, medium and product | |
| CN111338928A (en) | Chrome-based browser testing method and device | |
| CN112306723B (en) | Operation information acquisition method and device applied to applet | |
| CN107766212A (en) | Determine the method and device of the installment state of application program | |
| CN113760262A (en) | Task processing method, device, computer system and computer readable storage medium | |
| CN113542506A (en) | Outbound method, device, equipment and medium | |
| CN113448602A (en) | Version updating method and device | |
| CN114301660B (en) | Multi-server authentication method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: Room 221, 2 / F, block C, 18 Kechuang 11th Street, Daxing District, Beijing, 100176 Applicant after: Jingdong Technology Holding Co.,Ltd. Address before: Room 221, 2 / F, block C, 18 Kechuang 11th Street, Daxing District, Beijing, 100176 Applicant before: Jingdong Digital Technology Holding Co., Ltd |
|
| CB02 | Change of applicant information |