CN113448876A

CN113448876A - Service testing method, device, computer equipment and storage medium

Info

Publication number: CN113448876A
Application number: CN202111012950.2A
Authority: CN
Inventors: 门聪
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2021-08-31
Filing date: 2021-08-31
Publication date: 2021-09-28
Anticipated expiration: 2041-08-31
Also published as: CN113448876B

Abstract

The embodiment of the application provides a service testing method, a service testing device, computer equipment and a storage medium, which can be used for testing services in the fields of maps, traffic and the like. The service testing method comprises the following steps: according to the historical attribute data of each experimental object in the experimental group, carrying out layering processing on each experimental object in the experimental group to obtain M experimental attribute layers; according to the historical attribute data of each comparison object in the comparison group, carrying out layering processing on each comparison object in the comparison group to obtain M comparison attribute layers; performing index aggregation calculation on the M experimental attribute layers by adopting the test indexes, and performing index aggregation calculation on the M comparison attribute layers by adopting the test indexes to obtain index information of the experimental group and the comparison group under the test indexes; and comparing the difference between the index information of the experimental group and the index information of the control group under the test indexes, and determining the test result of the target service according to the difference. The accuracy of the test result can be improved.

Description

Service testing method, device, computer equipment and storage medium

Technical Field

The present application relates to the field of computer technologies, and in particular, to a method and an apparatus for testing a service, a computer device, and a storage medium.

Background

With the rapid development and progress of computer technology, various services (such as products, algorithms, models and the like) are continuously emerged in internet scenes, and before the services are officially released and popularized, the services need to be tested. At present, for any service in the internet scenario, for example, the service may be a target service, and the service testing scheme adopted is an AB experiment, that is, two batches of objects are randomly extracted as an experimental group and a control group of the target service, and the target service is tested by comparing and analyzing object index data generated in the experimental group and object index data generated in the control group.

In the service testing scheme based on the AB experiment, the experimental group and the control group are randomly extracted, and the random extraction mode easily causes the unbalanced distribution of objects in the experimental group and the control group; taking the statistical table shown in fig. 1 as an example, the proportion of the high-activity subjects in the control group is higher than that in the experimental group, although the average subject durations of the high-activity subjects in the experimental group and the control group are the same, and the average subject durations of the low-activity subjects in the experimental group and the control group are also the same, a deviation of 4% is generated between the average subject duration of the experimental group and the average subject duration of the control group; therefore, when the AB experiments of the random extraction experiment group and the control group are adopted to test the target service, the accuracy of the obtained test result is lower; therefore, how to improve the accuracy of the test result becomes a current research hotspot.

Disclosure of Invention

The embodiment of the application provides a service testing method, a service testing device, computer equipment and a storage medium, and the accuracy of a testing result can be improved.

In one aspect, an embodiment of the present application provides a service testing method, where the service testing method includes:

acquiring an experimental group and a control group of target services;

according to the historical attribute data of each experimental object in the experimental group, carrying out layering processing on each experimental object in the experimental group to obtain M experimental attribute layers; according to the historical attribute data of each comparison object in the comparison group, carrying out layering processing on each comparison object in the comparison group to obtain M comparison attribute layers, wherein M is a positive integer;

performing index aggregation calculation on the M experimental attribute layers by adopting the test indexes to obtain index information of the experimental group under the test indexes; performing index aggregation calculation on the M comparison attribute layers by adopting the test indexes to obtain index information of the comparison group under the test indexes;

and comparing the difference between the index information of the experimental group under the test index and the index information of the comparison group under the test index, and determining the test result of the target service according to the difference.

In one aspect, an embodiment of the present application provides a service testing apparatus, where the service testing apparatus includes:

the acquisition unit is used for acquiring an experimental group and a control group of the target service;

the processing unit is used for carrying out layering processing on each experimental object in the experimental group according to the historical attribute data of each experimental object in the experimental group to obtain M experimental attribute layers; according to the historical attribute data of each comparison object in the comparison group, carrying out layering processing on each comparison object in the comparison group to obtain M comparison attribute layers, wherein M is a positive integer;

the processing unit is also used for performing index aggregation calculation on the M experimental attribute layers by adopting the test indexes to obtain index information of the experimental group under the test indexes; performing index aggregation calculation on the M comparison attribute layers by adopting the test indexes to obtain index information of the comparison group under the test indexes;

and the processing unit is also used for comparing the difference between the index information of the experimental group under the test index and the index information of the comparison group under the test index, and determining the test result of the target service according to the difference.

In one implementation, the experimental group includes P experimental subjects, the number of the test indexes is R, and P and R are positive integers; the processing unit is configured to perform index aggregation calculation on the M experiment attribute hierarchies by using the test indexes, and when obtaining index information of the experiment group under the test indexes, the processing unit is specifically configured to execute the following steps:

randomly dividing P experimental objects into N random layers, wherein N is a positive integer;

combining the N random hierarchies with the M experiment attribute hierarchies to obtain N multiplied by M experiment comprehensive hierarchies, and determining an experiment object belonging to each experiment comprehensive hierarchy in the N multiplied by M experiment comprehensive hierarchies;

and performing index aggregation calculation on each experiment comprehensive layer in the N multiplied by M experiment comprehensive layers by adopting R test indexes, and determining the index information of each experiment comprehensive layer in the N multiplied by M experiment comprehensive layers under the R test indexes.

In one implementation, any one of the nxm experimental synthesis hierarchies is represented as an experimental synthesis hierarchy X_i*Y_j，X_iDenotes the ith random hierarchy, Y, of the N random hierarchies_jRepresenting the jth experiment attribute hierarchy in the M experiment attribute hierarchies, wherein i is a positive integer less than or equal to N, and j is a positive integer less than or equal to M;

a processing unit, configured to combine the N random hierarchies with the M experiment attribute hierarchies to obtain N × M experiment integrated hierarchies, and when determining an experiment object belonging to each experiment integrated hierarchy of the N × M experiment integrated hierarchies, specifically configured to execute the following steps:

combining the ith random hierarchy with the jth experiment attribute hierarchy to obtain an experiment comprehensive hierarchy X_i*Y_j；

Determining the experimental objects existing in the ith random layer and the jth experimental attribute layer as belonging to the experimental comprehensive layer X_i*Y_jThe subject of (1).

In one implementation, any one of the R test indexes is represented as the R-th test index, and R is a positive integer less than or equal to R; the processing unit is configured to perform index aggregation calculation on each of the N × M experiment integrated tiers by using R test indexes, and specifically, when determining index information of each of the N × M experiment integrated tiers under R test indexes, perform the following steps:

obtaining experimental comprehensive layering X_i*Y_jSubject index data for each subject in (a);

determining index information of a correlation index of an r-th test index from the object index data;

according to the determined index information of the associated index, calculating an experiment comprehensive hierarchy X_i*Y_jIndex information under the r-th test index.

In one implementation, the processing unit, when randomly dividing the P experimental subjects into N random hierarchies, is specifically configured to perform the following steps:

acquiring an object identifier of each experimental object in the P experimental objects;

performing hash calculation on the obtained object identifications of the P experimental objects to obtain object hashes of the P experimental objects;

and randomly dividing the P experimental objects into N random layers according to the object hash of the P experimental objects.

In one implementation, any one of the R test indexes is represented as the R-th test index, and R is a positive integer less than or equal to R; the processing unit is used for comparing the difference between the index information of the experimental group under the test index and the index information of the comparison group under the test index, and is specifically used for executing the following steps:

performing variance reduction calculation on the index information of each experimental comprehensive layer in the NxM experimental comprehensive layers under the r-th test index and the index information of each contrast comprehensive layer in the NxM contrast comprehensive layers under the r-th test index to determine the polymerization relative difference value of each random layer in the N random layers under the r-th test index;

wherein the NxM comparison integrated hierarchies are obtained by combining the N random hierarchies and the M comparison attribute hierarchies.

In one implementation, any one of the N random hierarchies is represented as the ith random hierarchy, i being a positive integer less than or equal to N; a processing unit, configured to perform variance reduction calculation on index information of each experimental integrated hierarchy in the N × M experimental integrated hierarchies under an r-th test index and index information of each comparative integrated hierarchy in the N × M comparative integrated hierarchies under the r-th test index, and when determining an aggregate relative difference value of each random hierarchy in the N random hierarchies under the r-th test index, specifically perform the following steps:

acquiring M experimental comprehensive hierarchies obtained by combining the ith random hierarchy and the M experimental attribute hierarchies, and acquiring M comparison comprehensive hierarchies obtained by combining the ith random hierarchy and the M comparison attribute hierarchies;

calculating M initial relative differences between the index information of each experiment comprehensive layer in the M experiment comprehensive layers under the r-th test index and the index information of each comparison comprehensive layer in the M comparison comprehensive layers under the r-th test index; wherein, the jth initial relative difference value in the M initial relative difference values is obtained by calculation according to the index information of the jth experiment comprehensive layer in the M experiment comprehensive layers under the r-th test index and the index information of the jth contrast comprehensive layer in the M contrast comprehensive layers under the r-th test index, and j is a positive integer less than or equal to M;

and performing weighted aggregation on the M initial relative difference values to obtain an aggregate relative difference value of the ith random layer under the r test index.

In an implementation manner, the processing unit, when determining the test result of the target service according to the difference, is specifically configured to execute the following steps:

performing significance calculation on the polymerization relative difference value of each random layer in the N random layers under the r test index to obtain a significance result of the target service under the r test index;

and determining the test result of the target service under the r test index according to the significance result of the target service under the r test index.

In one implementation, the experimental group includes P experimental subjects, P being a positive integer; one experimental attribute hierarchy in the M experimental attribute hierarchies corresponds to one attribute partition space; the processing unit is configured to perform layering processing on each experimental object in the experimental group according to the historical attribute data of each experimental object in the experimental group to obtain M experimental attribute layers, and is specifically configured to perform the following steps:

determining attribute partition areas to which historical attribute data of any one of the P experimental subjects belong;

and dividing any experimental object into the experimental attribute layers corresponding to the determined attribute division areas.

In one implementation, the processing unit is further configured to perform the following steps:

and screening the experimental objects in the M experimental attribute layers by adopting the attribute screening threshold value, and determining the experimental objects of which the historical attribute data in the M experimental attribute layers meet the attribute screening threshold value.

In one implementation, the number of the test indexes is R, any one of the R test indexes is represented as the R-th test index, R and R are positive integers, and R is less than or equal to R; the experimental group corresponds to a first business strategy of the target business, and the comparison group corresponds to a second business strategy of the target business; a processing unit further configured to perform the steps of:

if the test result of the target service under the r test index indicates that the first service strategy is superior to the second service strategy, optimizing the target service by adopting the first service strategy;

and if the test result of the target service under the r test index indicates that the second service strategy is superior to the first service strategy, maintaining the second service strategy in the target service.

In one aspect, an embodiment of the present application provides a computer device, which includes a processor and a computer-readable storage medium, wherein:

a processor adapted to implement a computer program; and a computer readable storage medium storing a computer program adapted to be loaded by a processor and to execute the above-mentioned service testing method.

Accordingly, an embodiment of the present application provides a computer-readable storage medium, where a computer program is stored, and when the computer program is read and executed by a processor of a computer device, the computer device is caused to execute the service testing method described above.

Accordingly, embodiments of the present application provide a computer program product or a computer program comprising computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to execute the service testing method.

In the embodiment of the application, after the experimental group and the control group of the target service are obtained, each experimental object in the experimental group can be subjected to layering treatment; similarly, each control object in the control group may be subjected to a stratification process; secondly, index aggregation calculation can be carried out on each layer in the experimental group to obtain index information of the experimental group under the test index, and index aggregation calculation can be carried out on each layer in the comparison group to obtain index information of the comparison group under the test index; then, by comparing the difference between the index information of the experimental group under the test index and the index information of the control group under the test index, the test result of the target service can be determined according to the difference. Therefore, the experimental group and the control group are subjected to layering treatment, so that the object distribution of the experimental group and the control group in each layer can be forcedly kept to be balanced, the interclass variance between the experimental group and the control group can be reduced, and the accuracy of a test result can be improved; moreover, index aggregation calculation is carried out on each layer in the experimental group and the comparison group, so that extra deviation caused by introduction of layers when index information of the test indexes is calculated can be avoided, and the accuracy of the test result can be further improved.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

FIG. 1 is a table illustrating test results statistics for a prior art business test scenario provided by an exemplary embodiment of the present application;

FIG. 2a is an interface diagram illustrating a display scheme for a hotspot card provided by an exemplary embodiment of the present application;

FIG. 2b is an interface diagram illustrating a display scheme for a hotspot card provided by another exemplary embodiment of the present application;

FIG. 2c is a schematic interface diagram illustrating a video playback scheme provided by an exemplary embodiment of the present application;

FIG. 2d is an interface diagram illustrating a video playback scheme according to another exemplary embodiment of the present application;

FIG. 3 is a block diagram illustrating an architecture of a business testing system according to an exemplary embodiment of the present application;

FIG. 4a is a flow chart illustrating a business testing scenario provided by an exemplary embodiment of the present application;

FIG. 4b illustrates an interface diagram of a test result provided by an exemplary embodiment of the present application;

FIG. 5 is a flow chart illustrating a method for testing services according to an exemplary embodiment of the present application;

FIG. 6 is a flow chart illustrating a method for testing services according to another exemplary embodiment of the present application;

FIG. 7 is a schematic structural diagram of a service test apparatus according to an exemplary embodiment of the present application;

fig. 8 shows a schematic structural diagram of a computer device according to an exemplary embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

In order to clearly understand the technical solutions provided in the embodiments of the present application, the following first introduces related terms related to the embodiments of the present application:

(1) the embodiment of the application relates to services. The services mentioned in the embodiment of the application refer to products, algorithms, models and the like which need to be released and promoted; any service (taking a target service as an example) may correspond to multiple (for example, two or more) service policies, a service policy refers to execution logic of a service, and a target service corresponding to multiple service policies may be understood as execution logic of multiple sets of services existing in the target service. For example, the target service is a product to be released and promoted, the first service policy of the target service may be a new version of the product, and the second service policy of the target service may be an old version of the product; the product can be map application, intelligent traffic application, audio and video playing application, news information application and the like; the product may also be, for example, a certain service function in the above application (e.g., a route planning function, a route recommendation function in a map application, a public transportation real-time location reminding function, an intelligent parking recommendation function in an intelligent transportation application, an audio/video playing function of an audio/video playing application, etc.). For another example, the target business is an artificial intelligence model, the first business strategy of the target business can be a new algorithm for implementing the artificial intelligence model, and the second business strategy of the target business can be an old algorithm for implementing the artificial intelligence model; the artificial intelligence model can be, for example, an artificial intelligence model applied to the fields of smart home, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned, autonomous, unmanned aerial vehicles, robots, smart medical care, smart customer service, internet of vehicles, autonomous, smart traffic, and the like.

(2) The embodiments of the present application also relate to AB experiments. The AB experiment mentioned in the embodiment of the application is a testing method and can be used for testing services; in the service test process adopting the AB experiment, taking the test process of the target service as an example, two batches of objects are randomly extracted as an experimental group of the target service and a comparison group of the target service, wherein the experimental group corresponds to a first service strategy of the target service, the comparison group corresponds to a second service strategy of the target service, and the test result of the target service can be determined by comparing and analyzing the index information of the test index generated by the experimental group under the first service strategy and the difference between the index information of the test index generated by the comparison group under the second service strategy; if the test result indicates that the first service strategy is superior to the second service strategy, the target service under the first service strategy can be released and promoted; if the test result indicates that the second service strategy is superior to the first service strategy, the target service under the second service strategy can be released and promoted.

(3) The embodiment of the application also relates to a test index. The test indexes mentioned in the embodiment of the application refer to units or methods for measuring object operations in an experimental group or a control group, the object operations refer to access operations of an object to a target item, the target item may include an application program, a website, a link, a picture, a text and the like, and the access operations may include login operations, view operations, click operations, download operations and the like; for example, the login operation of the object to the application may belong to an object operation, the view operation of the object to the link may belong to an object operation, and the download operation of the object to the picture or the text may belong to an object operation. The test indexes mentioned in the embodiments of the present application may include, but are not limited to, at least one of the following: average stay time of the object, image-text click through rate, retention rate, average access times of the object and the like. Wherein:

the average stay time of the objects can mean the average time of the multiple objects accessing the target item in the statistical period, and the average stay time of the objects can be obtained by calculation according to the total time of the multiple objects accessing the target item in the statistical period and the total number of the objects; the statistical period refers to a period for performing statistics on the operation of the object, and may be, for example, two days, three days, half a month, one month, or the like; for example, if object a accessed the target item for 5000 seconds and object B accessed the target item for 3000 seconds within the statistical period, then the average dwell time for the objects may be (5000 + 3000)/2 =4000 seconds. The image-text CTR (Click-Through-Rate) can refer to the ratio of the actual access times of the image-text (such as pictures, texts, links and the like) to the display times of the image-text; for example, if a link is shown 1000 times in a statistical period, but the actual number of accesses to the link is 200 times, the teletext click through rate may be 200/1000= 0.2. The retention rate can be the proportion of the object which accesses the target item at the beginning of the statistical period and still accesses the target item at the end of the statistical period in the object which accesses the target item at the beginning of the statistical period; for example, if 30 objects are accessed to the target item at the beginning of the statistical period and 12 objects are still accessed to the target item at the end of the statistical period, the retention rate may be 12/30= 0.4. The average access times of the objects can mean the average times of the multiple objects accessing the target item in the statistical period, and the average access times of the objects can be obtained by calculation according to the total times of the multiple objects accessing the target item in the statistical period and the total number of the objects; for example, if object a accesses the target item 2000 times and object B accesses the target item 1000 times within the statistical period, the average number of accesses for the object may be (2000 + 1000)/2 =1500 times.

The index information of the test index, that is, the index value of the test index, may be used to reflect the object operation attribute in the experimental group or the comparison group, that is, the index information of the experimental group under the test index may be used to reflect the object operation attribute in the experimental group, and the index information of the comparison group under the test index may be used to reflect the object operation attribute in the comparison group; for example, the test index is an average stay time of the objects of 100 experimental objects in the experimental group, when an index value of the average stay time of the objects is higher than a certain threshold, it may be determined that the operations of the objects in the experimental group are biased to the operations of the objects of the high-activity objects, and when the index value of the average stay time of the objects is lower than a certain threshold, it may be determined that the operations of the objects in the experimental group are biased to the operations of the objects of the low-activity objects.

It should be noted that the object mentioned in the embodiment of the present application may be, for example, a user, and the data related to the object (for example, the object operation, the index information, and the like) in the embodiment of the present application are all data acquired after the approval and authorization of the object.

Based on the business, the AB experiment and the relevant description of the test indexes, the embodiment of the application provides a business test scheme based on the AB experiment, and the business test scheme can solve the problem that the test result is inaccurate due to random extraction of an experiment group and a comparison group in the existing AB experiment. The service test scheme may be described as follows: when any task (taking a target service as an example) needs to be tested, an experimental group and a comparison group of the target service can be obtained, wherein the experimental group corresponds to a first service strategy of the target service, and the comparison group corresponds to a second service strategy of the target service; then, layering each experimental object in the experimental group, and performing index aggregation calculation on each layer in the experimental group to obtain index information of the experimental group under the first business strategy under the test index; similarly, each comparison object in the comparison group can be subjected to layering processing, and index aggregation calculation is performed on each layering in the comparison group, so that index information of the comparison group under the second business strategy under the test index is obtained; then, the index information of the experimental group under the test index and the index information of the comparison group under the test index are compared and analyzed, so that the test result of the target service can be determined. In the service testing scheme, the experimental group and the control group are subjected to layering treatment, so that the distribution of objects in the same layer of the experimental group and the control group is balanced, for example, the first experimental attribute layer of the experimental group and the first control attribute layer of the control group are all objects in the same attribute (such as activity) range, so that the interclass variance between the experimental group and the control group can be reduced, and the accuracy of the testing result of the target service is improved; moreover, index aggregation calculation is performed among the layers in the experimental group, and index aggregation calculation is performed among the layers in the comparison group, so that index information in the layers can be aggregated, deviation of the index information among the layers is avoided, and accuracy of a test result is further improved.

The service test scheme provided by the embodiment of the present application is applicable to various service test scenarios, and the service test scenario applicable to the embodiment of the present application is introduced below with reference to fig. 2a to 2 d:

(1) the target service to be tested is to determine the display position of the hot card in the news interface of the news application. Fig. 2a shows a first business strategy of a target business corresponding to a hot card display scheme, fig. 2a shows an interface schematic diagram of a hot card display scheme provided in an exemplary embodiment of the present application, a news interface 20 includes 5 display positions, which are a first display position 201, a second display position 202, a third display position 203, a fourth display position 204, and a fifth display position 205, respectively, and a hot card 206 is displayed in the third display position 203; fig. 2b shows a hot spot card display scheme corresponding to a second business policy of a target business, fig. 2b shows an interface schematic diagram of a hot spot card display scheme provided in another exemplary embodiment of the present application, and a hot spot card 206 is displayed in a fourth display position 204. The service test scheme provided by the embodiment of the application can be used for testing the target service, and if the test result indicates that the first service strategy is superior to the second service strategy, that is, the index performance of the test index when the hot spot card 206 is displayed at the third display position 203 is superior to the index performance of the test index when the hot spot card 206 is displayed at the fourth display position 204, the news application can be issued according to the first service strategy; if the test result indicates that the second business strategy is better than the first business strategy, that is, the index performance of the test index when the hot spot card 206 is displayed at the fourth display position 204 is better than the index performance of the test index when the hot spot card 206 is displayed at the third display position 203, the news application can be released according to the second business strategy.

(2) The target service to be tested is to determine a video playing strategy in a video playing interface of the video playing application. Fig. 2c shows a first service policy of a target service corresponding to a video playing scheme, and fig. 2c shows an interface schematic diagram of a video playing scheme provided by an exemplary embodiment of the present application, where a video playing interface 21 includes a plurality of video playing regions, and in any video playing region, for example, in a video playing region 211, when a playing progress of a currently played video (for example, the video 2 shown in fig. 2 c) reaches 80%, a next video (for example, the video 4 shown in fig. 2 c) may be switched to for playing; fig. 2d shows a second service policy of the video playing scheme corresponding to the target service, and fig. 2d shows an interface schematic diagram of a video playing scheme provided by another exemplary embodiment of the present application, where after the video currently playing in the video playing area 211 (for example, the video 2 shown in fig. 2 d) is finished playing (that is, the playing progress reaches 100%), the video may be switched to the next video (for example, the video 4 shown in fig. 2 d) for playing; the service test scheme provided by the embodiment of the application can be used for testing the target service, and if the test result indicates that the first service strategy is superior to the second service strategy, namely the index performance of the next video is switched and played when the playing progress of the current video reaches 80%, and the index performance of the next video is switched and played after the playing of the current video is finished, the video playing application can be issued according to the first service strategy; if the test result indicates that the second service strategy is superior to the first service strategy, that is, the index performance of the next video is switched and played after the current video is played, and the index performance of the next video is switched and played when the playing progress of the current video reaches 80%, the video playing application can be issued according to the second service strategy.

Referring to fig. 3, a service testing system suitable for implementing a service testing scheme provided in an embodiment of the present application is described below, where fig. 3 illustrates an architecture schematic diagram of a service testing system provided in an exemplary embodiment of the present application, where the service testing system may include a testing terminal 301, a testing server 302, and a plurality of (for example, two or more) object terminals 303, and the testing terminal 301, the testing server 302, and the object terminals 303 may establish direct or indirect connections through a wired communication manner or a wireless communication manner. The test terminal 301 may be, but is not limited to, a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, a smart voice interaction device, a smart home appliance (for example, a smart television), and the like; any one of the object terminals 303 may be, but is not limited to, a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, a smart voice interaction device, a smart home appliance (for example, a smart television), and the like; the test server 302 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud computing services, which is not limited herein.

The object terminal 303 corresponds to a measured object, and the object terminal 303 may generate object index data according to an object operation (i.e., an access operation) of the measured object for a service policy of a target service; the object index data is data reflecting an operation of an object of the object to be measured, and index information of an object-level index, which may be understood as an index for each object, may be included in the object index data, for example, the object-level index may include a stay time of the object per day in the statistical period, a number of times of access of the object per day in the statistical period, a total stay time of the object in the statistical period, a total number of times of access of the object in the statistical period, and the like. The test terminal 301 corresponds to a test object, and a test client, which may be, for example, a test application, a test website, or the like, runs in the test terminal 301, and provides a test interface for the test object, and the test object may interact with the test server 302 by operating in the test interface provided by the test client. The test client and the test server 302 can form a service test platform; it should be noted that, in the service test system shown in fig. 3, for example, the test client and the test server 302 are distributed in different devices, in an actual service test scenario, the test client and the test server 302 may also be integrated in the same device, for example, the test client and the test server 302 may be integrated in the test server 302, and for example, the test client and the test server 302 may be integrated in the test terminal 301, which is not limited in this embodiment of the present application. It should be noted that the object index data of each measured object is data acquired after approval and authorization of each measured object.

In the service test system composed of the test terminal 301, the test server 302 and the plurality of object terminals 303, the service test scheme may include a test preparation stage, a service test stage and a test result display stage, which are described below:

(1) and (5) a test preparation stage.

In order to test the target service, a plurality of object terminals are extracted as an experimental group and a comparison group by a test server, that is, a plurality of objects to be tested are extracted as an experimental object and a comparison object by the test server; the object terminals of the experiment group correspond to a first business strategy of the target business, the object terminals of the experiment group can operate according to the object of the experiment object aiming at the first business strategy to generate object index data under the first business strategy, and then the object terminals of the experiment group can send the generated object index data to the test server; similarly, the object terminal of the comparison group corresponds to a second service policy of the target service, the object terminal of the comparison group may generate object index data under the second service policy according to the object operation of the comparison object to the second service policy, and then the object terminal of the comparison group may send the generated object index data to the test server. It should be noted that, in the embodiment of the present application, there are various ways for the test server to extract the experimental group and the control group, which may be random extraction ways or extraction ways according to extraction rules; an exemplary extraction rule may be described as extracting object terminals with even terminal identifications as experimental groups and extracting object terminals with odd terminal identifications as control groups; in the process of extracting the experimental group and the control group, any one of the extraction methods can be adopted for extraction, and the extraction method is not limited in the embodiment of the application.

Then, the test server can carry out layering processing on the experimental groups, and carry out index aggregation calculation on object index data among all the layers of the experimental groups to obtain index information of the experimental groups under the test indexes; and the test server can carry out layering processing on the comparison group and carry out index aggregation calculation on the object index data among all the layers of the comparison group to obtain the index information of the comparison group under the test index. The test server can also store the processed index information in a database; the database can be a local database of the test server, and the index information obtained by processing is stored in the local database of the test server, so that when the index information needs to be used in a service test stage, the index information can be quickly inquired and loaded from the local database, and the inquiry efficiency and the loading speed of the index information are improved; the database can also be a remote database independent of the test server, the database can also be a database storage service deployed in the cloud server, the cloud server exists independent of the test server, and the requirement of the test server on the storage and calculation capacity of the database can be reduced by storing the index information by means of the storage service except the test server; in addition, the data size of the object-level object index data is very large, and the index information obtained by performing index aggregation calculation on the object index data is stored in the database without storing the object-level object index data in the database, so that the storage pressure of the database can be reduced.

(2) And (5) a service test stage.

In one implementation, when the test client detects a test operation for the target service, for example, when a test object selects a test control for the target service in the test client, the test client may send a test request for the target service to the test server. When the test server receives a test request of the target service, the test server can inquire index information of an experimental group of the target service under a test index and index information of a comparison group of the target service under the test index from a database; then, the test server can compare the difference between the index information of the experimental group under the test index and the index information of the comparison group under the test index, and determine the test result of the target service according to the difference, and the test server can also return the test result of the target service to the test client.

In another implementation manner, as shown in fig. 4a, fig. 4a is a schematic flowchart illustrating a service testing scheme provided in an exemplary embodiment of the present application, where when a testing server receives a testing request of a target service, the testing server may query, from a database, index information of an experimental group of the target service under a testing index and index information of a comparison group of the target service under the testing index; then, the test server can call a variance reduction service deployed in the cloud server to compare the difference between the index information of the experimental group under the test index and the index information of the comparison group under the test index, determine the test result of the target service according to the difference, obtain the test result of the target service from the variance reduction service, and then return the test result of the target service to the test client. In the implementation mode, the variance reduction service is deployed in the cloud server and can be used for multiplexing a plurality of business test processes, so that the business test efficiency can be improved.

The variance reduction service may be deployed in the cloud server through a service module (for example, a Python flash module), and the process of the test server invoking the variance reduction service may include: firstly, the test server can package the index information of the experimental group under the test index and the index information of the comparison group under the test index by adopting a target format; the target format can be json format, for example, and convenience in index information transmission can be improved by packaging the index information; the test server generates an access request according to the encapsulated index information and sends the access request to a service interface of the variance reduction service in the cloud server; the access request may be, for example, a post request in HTTP (Hypertext Transfer Protocol); and thirdly, after the variance reduction service in the cloud server determines the test result of the target service, the test server can obtain the test result of the target service through a service interface of the variance reduction service.

(3) And a test result display stage.

After the test client receives the test result of the target service returned by the test server, the test client can render the test result of the target service to a test interface of the test client for display; fig. 4b is an interface diagram of a test result provided in an exemplary embodiment of the present application, and as shown in fig. 4b, the test result shows index information of an experimental group (for example, comparison 1 shown in fig. 4 b) under 7 test indexes, index information of a control group (for example, comparison 2 shown in fig. 4 b) under 7 test indexes, a difference between the index information of the experimental group under each test index and the index information of the control group under each test index (for example, an absolute difference and a relative difference shown in fig. 4 b), and the like.

It can be understood that the service test system and the three stages of the service test scheme described in the embodiment of the present application are for more clearly illustrating the technical scheme of the embodiment of the present application, and do not constitute a limitation to the technical scheme provided in the embodiment of the present application, and it can be known by a person skilled in the art that the technical scheme provided in the embodiment of the present application is also applicable to similar technical problems along with the evolution of the system architecture and the appearance of new service scenarios.

Based on the above-mentioned service test scheme and the related description of the service test system, the service test scheme provided by the embodiment of the present application is described in more detail below with reference to fig. 5 and 6. Referring to fig. 5, fig. 5 is a flowchart illustrating a service testing method according to an exemplary embodiment of the present application, where the service testing method may be executed by a computer device provided in an exemplary embodiment of the present application, where the computer device may be the testing server 302 in the service testing system, and the service testing method may include the following steps S501 to S504:

s501, acquiring an experimental group and a control group of the target service.

When a target service needs to be tested, an experimental group of the target service and a comparison group of the target service can be obtained, wherein the experimental group of the target service corresponds to a first service strategy of the target service, and the comparison group of the target service corresponds to a second service strategy of the target service; p subjects can be included in the experimental group, Q control subjects can be included in the control group, and P and Q are positive integers.

S502, according to the historical attribute data of each experimental object in the experimental group, carrying out layering processing on each experimental object in the experimental group to obtain M experimental attribute layers; and according to the historical attribute data of each comparison object in the comparison group, carrying out layering processing on each comparison object in the comparison group to obtain M comparison attribute layers.

The layering processing process of the experimental objects in the experimental group is carried out based on the historical attribute data of the experimental objects, the layering processing process of the control objects in the control group is carried out based on the historical attribute data of the control objects, and the historical attribute data is introduced firstly: the historical attribute data refers to the operation attribute of the object in a historical time interval; the historical time interval refers to a time interval formed from the nth-m-T day to the nth-m day, namely the historical time interval can be expressed as [ n-m-T, n-m ]; wherein n represents that the service test method is executed on the nth day; the value of m is larger than the experimental period of the AB experiment, so that the interference of the experimental result of the AB experiment on the historical attribute data can be avoided; t denotes a statistical period, which may be two days, three days, half a month, one month, etc., as can be appreciated from the foregoing. The historical attribute data may include a total dwell time of the object in the historical time interval or a total number of accesses of the object in the historical time interval. For example, the service test method is performed on day 60, the test period of the AB test is 14 days, m is 20, and T is 30, so the historical time interval is the time interval formed from day 10 to day 40, that is, the historical attribute data may include the total staying time of the object in the historical time interval [10, 40] or the total number of visits of the object in the historical time interval [10, 40 ]. It should be noted that the historical attribute data of each subject (including the experimental subject and the control subject) is data collected after approval and authorization of each subject.

In addition, historical attribute data can be used to indicate liveness of objects. For example, the longer the total stay time of the object in the historical time interval, the higher the activity of the object is represented; the shorter the total stay time of the object in the historical time interval is, the lower the activity of the object is; for another example, the more total access times of the object in the historical time interval, the higher the activity of the object; the smaller the total number of accesses of the object in the historical time interval, the lower the activity of the object.

For P experimental objects in the experimental group, each experimental object in the experimental group may be layered according to the historical attribute data of each experimental object in the experimental group, to obtain M experimental attribute layers, where M is a positive integer. Specifically, one of the M experiment attribute hierarchies corresponds to one attribute partition space of the historical attribute data, and for any one of the P experiment objects, the attribute partition space to which the historical attribute data of the experiment object belongs can be determined, so that the experiment object can be partitioned into the experiment attribute hierarchy corresponding to the determined attribute partition space. For example, the historical attribute data refers to a total staying time of an object in a historical time interval, the experimental objects in an experimental group need to be divided into 5 experimental attribute layers, and the 5 experimental attribute layers are respectively a first experimental attribute layer, a second experimental attribute layer, a third experimental attribute layer, a fourth experimental attribute layer and a fifth experimental attribute layer, where the first experimental attribute layer corresponds to a total staying time interval [0, 5000 ], the second experimental attribute layer corresponds to a total staying time interval [5000, 20000 ], the third experimental attribute layer corresponds to a total staying time interval [20000, 50000 ], the fourth experimental attribute layer corresponds to a total staying time interval [50000, 120000 ], the fifth experimental attribute layer corresponds to a total staying time interval [120000, 600000], and the unit is second; if the total staying time of the experimental object a in the historical time interval is 60000 seconds, the experimental object a may be classified into a fourth experimental attribute hierarchy.

It should be noted that, the hierarchical attribute dimension adopted in the hierarchical processing process is one-dimensional, and in order to improve the diversity of the hierarchy and further improve the accuracy of the test result, the hierarchical attribute dimension may be expanded, and the experimental objects in the experimental group are hierarchically processed based on a plurality of (e.g., two or more) historical attribute data. In the single-dimension hierarchical processing process, the number of the experimental attribute hierarchies is determined according to the number of the attribute partition areas of the historical attribute data, the number of the experimental attribute hierarchies is equal to the number of the attribute partition areas of the historical attribute data, and one experimental attribute hierarchy corresponds to one attribute partition area of the historical attribute data. For the multi-dimensional hierarchical processing procedure, the following two cases can be included but not limited: (1) taking a two-dimensional hierarchical processing process as an example, for example, the experimental objects in the experimental group are hierarchically processed based on the first historical attribute data and the second historical attribute data; the first historical attribute data comprises a plurality of attribute division areas, the second historical attribute data comprises a plurality of attribute division areas, and the number of the experimental attribute hierarchies is determined according to the number of the attribute division areas of the first historical attribute data and the number of the attribute division areas of the second historical attribute data; the number of the experimental attribute hierarchies is equal to the product of the number of the attribute partitions of the first historical attribute data and the number of the attribute partitions of the second historical attribute data; and the attribute division areas corresponding to the experimental attribute layers are determined according to the attribute division areas of the first historical attribute data and the attribute division areas of the second historical attribute data. (2) Taking a two-dimensional hierarchical processing process as an example, for example, the experimental objects in the experimental group are hierarchically processed based on the first historical attribute data and the second historical attribute data; the first historical attribute data comprises a plurality of attribute division areas, the second historical attribute data comprises a plurality of attribute values, the number of the experimental attribute hierarchies is determined according to the number of the attribute division areas of the first historical attribute data and the number of the attribute values of the second historical attribute data, and the number of the experimental attribute hierarchies is equal to the product of the number of the attribute division areas of the first historical attribute data and the number of the attribute values of the second historical attribute data; and the attribute partition intervals corresponding to the experimental attribute hierarchies are determined according to the attribute partition intervals of the first historical attribute data and the attribute values of the second historical attribute data. For example, the first historical attribute data includes two attribute divisions, [ a, b ] and [ c, d ], respectively; the second historical attribute data includes two attribute values, male and female respectively; then it can be determined that the number of the experimental attribute hierarchies is 4, and the attribute partition regions corresponding to the 4 experimental attribute hierarchies are { [ a, b ], male }, { [ a, b ], female }, { [ c, d ], male } and { [ c, d ], female }, respectively.

After dividing the experimental objects in the experimental group into M experimental attribute layers, screening the experimental objects in the experimental group. The specific screening process can be described as follows, screening the experimental objects in the M experimental attribute layers by adopting an attribute screening threshold value, and determining the experimental objects of which the historical attribute data in the M experimental attribute layers meet the attribute screening threshold value; the experimental object with the historical attribute data meeting the attribute screening threshold value is the experimental object with the historical attribute data smaller than or equal to the attribute screening threshold value; that is, the experimental objects with the historical attribute data larger than the attribute screening threshold in the M experimental attribute layers are taken as abnormal objects to be removed, and the experimental objects with the historical attribute data smaller than or equal to the attribute screening threshold in the M experimental attribute layers are reserved; the attribute filtering threshold may be set based on empirical values of the test object. According to the foregoing, the historical attribute data of the experimental objects is related to the liveness, and although the proportion of the high-liveness objects occupied in the experimental group is small, additional test deviations are easily introduced, so that the high-liveness objects are filtered by using the attribute screening threshold, and the influence of the existence of the high-liveness objects on the accuracy of the test result can be avoided.

The hierarchical processing process of the comparison object in the comparison group is similar to that of the experimental object in the experimental group, namely, one comparison attribute hierarchy in the M comparison attribute hierarchies corresponds to one attribute partition space of the historical attribute data, and for any comparison object in the Q comparison objects, the attribute partition space to which the historical attribute data of the comparison object belongs can be determined, so that the comparison object can be partitioned into the comparison attribute hierarchy corresponding to the determined attribute partition space. It should be noted that, in the process of hierarchical processing, the attribute partition space of the historical attribute data in the experimental group is the same as the attribute partition space of the historical attribute data in the control group, so that the proportion occupied by each activity object in the experimental group and the control group can be kept the same, thereby avoiding the test deviation caused by the uneven proportion occupied by each activity object in the experimental group and the control group, and being beneficial to improving the accuracy of the test result. And after dividing the comparison objects in the comparison group into M comparison attribute hierarchies, screening the experimental objects in the M comparison attribute hierarchies by using an attribute screening threshold, and determining the comparison objects of which the historical attribute data in the M comparison attribute hierarchies meet the attribute screening threshold, that is, the high-activity objects in the experimental group and the comparison group can be removed as abnormal objects by using the attribute screening threshold, so that the influence of the existence of the high-activity objects on the accuracy of the test result can be avoided.

S503, performing index aggregation calculation on the M experiment attribute layers by adopting the test indexes to obtain index information of the experiment group under the test indexes; and performing index aggregation calculation on the M comparison attribute layers by adopting the test indexes to obtain index information of the comparison group under the test indexes.

The number of the test indexes can be R, and index aggregation calculation can be performed on M experiment attribute layers by adopting the R test indexes to obtain index information of an experiment group under the R test indexes; and performing index aggregation calculation on the M comparison attribute layers by adopting R test indexes to obtain index information of the comparison group under the R test indexes, wherein R is a positive integer. The index aggregation calculation process of the experimental group and the index aggregation calculation process of the control group are similar, and the index aggregation calculation process of the control group is described by taking the experimental group as an example, the index aggregation calculation process of the control group can be referred to the index aggregation calculation process of the experimental group, and the process of performing index aggregation calculation on M experimental attribute layers by using R test indexes can include the following (1) to (3):

(1) and randomly dividing the P experimental objects into N random layers, wherein N is a positive integer. The random stratification process for P subjects can be described as follows: the object identification of each experimental object in the P experimental objects can be obtained; then, performing hash calculation on the obtained object identifications of the P experimental objects to obtain object hashes of the P experimental objects; further, the P experimental objects can be randomly divided into N random hierarchies according to the object hash of the P experimental objects. In the specific implementation, the object hash is a character string consisting of random letters and numbers and has randomness; the number of different tail characters in the object hashes of P experimental objects can be counted, the number of different tail characters in the object hashes of P experimental objects is determined as the number of random hierarchies, one random hierarchy corresponds to one tail character, then the P experimental objects can be divided into corresponding random hierarchies according to the tail characters of the object hashes, and the tail characters of the object hashes of the experimental objects belonging to the same random hierarchy are the same; for example, if there are 3 different tail characters in the object hashes of P experimental objects, the number of random hierarchies may be determined to be 3, and the P experimental objects may be divided into 3 random hierarchies according to the tail characters of the object hashes. It should be noted that the above determining the number of random hierarchies according to the number of different tail characters in the object hash of P experimental objects is only used as an example; the number of random hierarchies may also be determined according to the number of different first characters in the object hash, where one random hierarchy corresponds to one first character, in which case P experimental objects may be divided into corresponding random hierarchies according to the first characters of the object hash, which is not limited in this application embodiment.

(2) And combining the N random hierarchies with the M experiment attribute hierarchies to obtain N multiplied by M experiment comprehensive hierarchies, and determining an experiment object belonging to each experiment comprehensive hierarchy in the N multiplied by M experiment comprehensive hierarchies. The comprehensive N × M experimental hierarchies obtained by combining the N random hierarchies and the M experimental attribute hierarchies can be referred to the following table 1:

TABLE 1

As shown in Table 1 above, the N random hierarchies may be represented as X₁,…,X_i,…,X_N(ii) a The M experimental attribute hierarchies may be represented as Y₁,…,Y_j,…,Y_M；X_iDenotes the ith random hierarchy, Y, of the N random hierarchies_jRepresenting j test attribute hierarchy of M test attribute hierarchies, i is a positive integer less than or equal to N, j is a positive integer less than or equal to M, and any test comprehensive hierarchy of NxM test comprehensive hierarchies can be represented as a test comprehensive hierarchy X_i*Y_j。

Here, the layers X are combined by means of experiments_i*Y_jFor example, comprehensive stratification for certain experiments X_i*Y_jAnd determining that it belongs to the experimental integrated stratification X_i*Y_jThe procedure of the subject of (1) is described. Specifically, the ith random hierarchy may be combined with the jth experimental attribute hierarchy,obtaining an experimental comprehensive layer X_i*Y_j(ii) a And determining the experimental objects existing in the ith random layer and the jth experimental attribute layer as belonging to the experimental comprehensive layer X_i*Y_jThe subject of (1). For example, the ith random hierarchy includes an experimental object a and an experimental object B, and the jth experimental attribute hierarchy includes an experimental object a, an experimental object B and an experimental object C; combining the ith random hierarchy and the jth experiment attribute hierarchy to obtain an experiment comprehensive hierarchy X_i*Y_jThen belongs to the experimental comprehensive layering X_i*Y_jThe subjects of (1) are subject a and subject B.

(3) And performing index aggregation calculation on each experiment comprehensive layer in the N multiplied by M experiment comprehensive layers by adopting R test indexes, and determining the index information of each experiment comprehensive layer in the N multiplied by M experiment comprehensive layers under the R test indexes.

Any reference index in the R test indexes can be expressed as the R-th test index, and any experiment comprehensive layering in the N multiplied by M experiment comprehensive layering can be expressed as an experiment comprehensive layering X_i*Y_jThe r test index and the experiment are used for integrating the layering X_i*Y_jFor example, the r test index is adopted to comprehensively layer the experiment X_i*Y_jPerforming index aggregation calculation to determine experimental comprehensive layering X_i*Y_jThe process of index information under the r-th test index is introduced. Specifically, an experimental integrated hierarchy X may be obtained_i*Y_jSubject index data for each subject in (a); then, the index information of the relevant index of the r-th test index can be determined from the target index data, and the experiment comprehensive hierarchy X can be calculated and obtained according to the determined index information of the relevant index_i*Y_jIndex information under the r-th test index. Here, the correlation index of the r-th test index is an index related to the calculation of the r-th test index; for example, if the r-th test index is the average stay time of the subject, and the average stay time of the subject is calculated in relation to the total stay time of the subject in the statistical period, then the correlation index may be the average stay time of the subject in the statistical periodTotal dwell time in the cycle; for another example, the r-th test index is the average staying time of the subject, and the calculation of the average staying time of the subject is related to the staying time of the subject per day in the statistical period, so the related index may also be the staying time of the subject per day in the statistical period. When the correlation index is the total stay time of the experimental object in the statistical period, the comprehensive layering X can be performed according to the experiment_i*Y_jThe total stay time of each experimental object in the statistical period and the comprehensive layering X of the experiment_i*Y_jThe total number of the experimental objects in (1) is calculated to obtain an experimental comprehensive layering X_i*Y_jAverage dwell time of the subject in (1).

For the comparison group, R test indexes may also be used to perform index aggregation calculation on the M comparison attribute layers to obtain index information of the comparison group under the R test indexes. The process of performing index aggregation calculation on the M control attribute hierarchies using the R test indexes may include: randomly dividing the Q comparison objects into N random layers; combining the N random hierarchies with the M comparison attribute hierarchies to obtain N × M comparison comprehensive hierarchies, and determining a comparison object belonging to each comparison comprehensive hierarchy of the N × M comparison comprehensive hierarchies; and performing index aggregation calculation on each of the NxM control comprehensive hierarchies by adopting the R test indexes to determine the index information of each of the NxM control comprehensive hierarchies under the R test indexes. It should be noted that, the process of performing index aggregation calculation on M comparison attribute hierarchies by using R test indexes is similar to the process of performing index aggregation calculation on M experiment attribute hierarchies by using R test indexes, and specific reference may be made to the description of the index aggregation calculation process on M experiment attribute hierarchies in (1) to (3), which is not described herein again.

S504, comparing the difference between the index information of the experimental group under the test index and the index information of the comparison group under the test index, and determining the test result of the target service according to the difference.

As can be seen from the above, the number of the test indexes may be R, so that the differences between the index information of the experimental group under each test index and the index information of the comparison group under the corresponding test index may be respectively compared, and the test result of the target service is determined according to the differences of each test index. Taking the R-th test index of the R test indexes as an example for description, the difference between the index information of the experimental group under the R-th test index and the index information of the comparison group under the R-th test index can be compared, and the test result of the target service under the R-th test index can be determined according to the difference under the R-th test index.

In addition, for a core index (the core index is an index which has higher sensitivity and larger contribution to the test result of the target service) in the R test indexes, the service policy of the target service can be determined directly according to the test result of the target service under the core index. Specifically, the experimental group corresponds to a first business strategy of the target business, and the comparison group corresponds to a second business strategy of the target business, wherein the first business strategy is a new business strategy capable of achieving the target business compared with the second business strategy, and the second business strategy can be originally existed in the target business; if the R-th test index is a core index of the R test indexes, after the test result of the target service under the R-th test index is determined according to the difference of the R-th test index, the service policy of the target service can be determined according to the test result of the target service under the R-th test index. If the test result of the target service under the r test index indicates that the first service strategy is superior to the second service strategy, the target service can be optimized by adopting the first service strategy, namely, the first service strategy is adopted to replace the original second service strategy in the target service; or, if the test result of the target service under the r-th test index indicates that the second service policy is better than the first service policy, the second service policy may be maintained in the target service, that is, the original second service policy in the target service may be maintained unchanged. By the method, the service strategy of the target service can be determined based on the test result of the target service under the core index, the test result of the target service under each test index does not need to be considered, the core index is an index which greatly contributes to the test result of the target service, and the service test efficiency can be improved on the premise of ensuring the accuracy of the test result.

In the embodiment of the application, after the experimental group is subjected to layering processing and index aggregation calculation, index information of the experimental group under the test index can be obtained; after the control group is subjected to layering treatment and index aggregation calculation, index information of the control group under the test index can be obtained; then, by comparing the difference between the index information of the experimental group under the test index and the index information of the comparison group under the test index, the test result of the target service can be determined, and the determined test result has higher accuracy. In addition, the high-activity objects in the experimental group and the control group can be filtered by adopting the attribute screening threshold value, and the high-activity objects are taken as abnormal objects to be removed, so that the influence of the existence of the high-activity objects on the accuracy of the test result can be avoided. In addition, in the layering treatment process of the experimental group and the control group, the layering attribute dimensionality can be expanded, the more layering attribute dimensionalities are, the more detailed and rich layering schemes of the experimental group and the control group are, and therefore the accuracy of the test result can be further improved.

Referring to fig. 6, fig. 6 is a flowchart illustrating a service testing method according to another exemplary embodiment of the present application, where the service testing method may be executed by a computer device provided in an embodiment of the present application, where the computer device may be the testing server 302 in the service testing system, and the service testing method may include the following steps S601 to S606:

s601, acquiring an experimental group and a control group of the target service.

The execution process of step S601 in the embodiment of the present application is the same as the execution process of step S501 in the embodiment shown in fig. 5, and the execution process of step S601 may refer to the specific description of step S501 in the embodiment shown in fig. 5, and is not described herein again.

S602, carrying out layering processing on each experimental object in the experimental group according to the historical attribute data of each experimental object in the experimental group to obtain M experimental attribute layers; and according to the historical attribute data of each comparison object in the comparison group, carrying out layering processing on each comparison object in the comparison group to obtain M comparison attribute layers.

In the embodiment of the present application, the execution process of step S602 is the same as the execution process of step S502 in the embodiment shown in fig. 5, and the execution process of step S602 may refer to the specific description of step S502 in the embodiment shown in fig. 5, and is not described herein again.

S603, performing index aggregation calculation on the M experiment attribute layers by adopting the R test indexes, and determining index information of each experiment comprehensive layer in the NxM experiment comprehensive layers under the R test indexes; and performing index aggregation calculation on the M comparison attribute layers by adopting the R test indexes to determine the index information of each comparison comprehensive layer in the NxM comparison comprehensive layers under the R test indexes.

The execution process of step S603 in this embodiment is the same as the execution process of step S503 in the embodiment shown in fig. 5, and the execution process of step S603 may refer to the specific description of step S503 in the embodiment shown in fig. 5, and is not described herein again.

S604, performing variance reduction calculation on the index information of each experiment comprehensive layer in the N multiplied by M experiment comprehensive layers under the r-th test index and the index information of each comparison comprehensive layer in the N multiplied by M comparison comprehensive layers under the r-th test index, and determining the polymerization relative difference value of each random layer in the N random layers under the r-th test index.

Step S603 may find that, after performing index aggregation calculation on the M experiment attribute hierarchies by using the R test indexes, index information of each experiment comprehensive hierarchy of the N × M experiment comprehensive hierarchies under the R test indexes may be obtained; and after index aggregation calculation is carried out on the M comparison attribute layers by adopting the R test indexes, index information of each comparison comprehensive layer in the NxM comparison comprehensive layers under the R test indexes can be obtained. Based on this, for any one of the R test indexes (i.e., the R-th test index), a variance reduction calculation may be performed on the index information of each of the N × M experimental integrated tiers under the R-th test index and the index information of each of the N × M control integrated tiers under the R-th test index to determine an aggregate relative difference value of each of the N random tiers under the R-th test index. Taking any one of the R test indexes (i.e., the R-th test index) and any one of the N random hierarchies (i.e., the i-th random hierarchy) as an example, a process for determining the aggregated relative difference value of the i-th random hierarchy of the N random hierarchies under the R-th test index will be described, and the process may include the following (1) to (3):

(1) obtaining M experimental comprehensive hierarchies obtained by combining the ith random hierarchy and the M experimental attribute hierarchies, and obtaining M comparison comprehensive hierarchies obtained by combining the ith random hierarchy and the M comparison attribute hierarchies.

(2) And calculating M initial relative difference values between the index information of each experiment comprehensive layer in the M experiment comprehensive layers under the r-th test index and the index information of each comparison comprehensive layer in the M comparison comprehensive layers under the r-th test index.

Wherein, the jth initial relative difference value in the M initial relative difference values is obtained by calculation according to the index information of the jth experiment comprehensive layer in the M experiment comprehensive layers under the r-th test index and the index information of the jth contrast comprehensive layer in the M contrast comprehensive layers under the r-th test index, and j is a positive integer less than or equal to M; specifically, the jth initial relative difference is equal to a difference between a first value and a second value, the first value is equal to a ratio between the index information of the jth experimental integrated hierarchy under the r-th test index and the index information of the jth comparison integrated hierarchy under the r-th test index, and the second value may be a constant, and may be 1, for example.

(3) And performing weighted aggregation on the M initial relative difference values to obtain an aggregate relative difference value of the ith random layer under the r test index.

Before performing weighted aggregation on the M initial relative differences, M weights corresponding to the M initial relative differences need to be determined, where one initial relative difference corresponds to one weight, the jth initial relative difference corresponds to the jth weight, and the jth weight is related to a calculation manner of index information of the r-th test index in the jth comparison comprehensive hierarchy, and here, taking the r-th test index and the jth weight as an example, a determination process of the jth weight is introduced: firstly, a calculation mode of index information of a jth contrast comprehensive layer under an r test index can be determined, wherein the jth contrast comprehensive layer is obtained by combining an ith random layer and a jth contrast attribute layer; then, the jth weight can be determined according to the calculation mode of the index information of the jth contrast comprehensive hierarchy under the r-th test index.

For example, the r-th test index is the average stay time of the object, and the calculation mode of the j-th comparison comprehensive hierarchy index information under the average stay time of the object is as follows: taking the ratio of the total staying time of each contrast object in the jth contrast comprehensive layer to the total number of the contrast objects in the jth contrast comprehensive layer as index information of the jth contrast comprehensive layer under the average staying time of the objects; the jth weight is determined according to the total number of the control objects in the jth control comprehensive layer and the total number of the control objects under the ith random layer, and the jth weight is equal to the ratio of the total number of the control objects in the jth control comprehensive layer to the total number of the control objects under the ith random layer. For another example, the r-th test index is the day-average staying time length, and since the statistical period is the same for each control object, when the r-th test index is the day-average staying time length, the weights corresponding to the initial relative difference values are the same and are all 1/M.

To sum up from (1) to (3), the calculation process of the aggregate relative difference value of the ith random hierarchy of the N random hierarchies under the r test index can be seen in the following formula 1:

equation 1

As can be seen from the above-mentioned formula 1,

representing the relative difference of aggregation of the ith random layering under the r test index;

represents the jth initial relative difference;

representing the jth weight.

S605, performing significance calculation on the polymerization relative difference value of each random layer in the N random layers under the r test index to obtain a significance result of the target service under the r test index.

The process of calculating the significance of the aggregation relative difference of each random hierarchy of the N random hierarchies under the r-th test index to obtain the significance result of the target service under the r-th test index may specifically include: and inputting the polymerization relative difference value of each random layer in the N random layers under the r test index into the significance calculation model, and acquiring the significance result of the target service output by the significance calculation model under the r test index. The significance calculation model can be a double-population T test model, for example, the double-population T test model is a hypothesis test model, and original hypotheses and alternative hypotheses exist in the double-population T test model; the method comprises the following steps that in an original hypothesis, no difference exists between index information of a hypothesis experiment group under an r-th test index and index information of a comparison group under the r-th test index, and in a preparation hypothesis, significant difference exists between the index information of the hypothesis experiment group under the r-th test index and the index information of the comparison group under the r-th test index; the significance result output by the significance calculation model can be seen in the test interface shown in fig. 4b, and the significance result may include significance or no significance; if the significance result is significant, the original hypothesis is rejected, and an alternative hypothesis is selected, namely the index information of the experimental group under the r test index is significantly different from the index information of the comparison group under the r test index; and if the significance result is not significant, the original hypothesis is selected, and the alternative hypothesis is rejected, namely, the index information of the experimental group under the r test index is not different from the index information of the control group under the r test index.

S606, according to the significance result of the target service under the r test index, determining the test result of the target service under the r test index.

After the aggregated relative difference of each random hierarchy of the N random hierarchies under the r-th test index is input into the significance calculation model, the significance calculation model may output, in addition to the significance result of the target service under the r-th test index, an overall relative difference between index information of the experimental group under the r-th test index and index information of the control group under the r-th test index, for example, a relative difference shown in the test interface shown in fig. 4 b; therefore, the test result of the target service under the r test index can be determined according to the significance result of the target service under the r test index and the total relative difference value between the index information of the experimental group under the r test index and the index information of the comparison group under the r test index.

In a specific implementation, the test result may include a first test result and a second test result, the first test result indicates that the first service policy corresponding to the experimental group is better than the second service policy corresponding to the comparison group, and the second test result indicates that the second service policy corresponding to the comparison group is better than the first service policy corresponding to the experimental group; if the significance result of the target service under the r-th test index is significant, and the total relative difference value between the index information of the experimental group under the r-th test index and the index information of the comparison group under the r-th test index is a positive value, determining that the test result of the target service is a first test result; if the significance result of the target service under the r-th test index is significant, and the total relative difference value between the index information of the experimental group under the r-th test index and the index information of the comparison group under the r-th test index is a negative value, the test result of the target service can be determined to be a second test result.

In the embodiment of the application, after the experimental group is subjected to layering processing and index aggregation calculation, index information of the experimental group under the test index can be obtained; after the control group is subjected to layering treatment and index aggregation calculation, index information of the control group under the test index can be obtained; then, by comparing the difference between the index information of the experimental group under the test index and the index information of the comparison group under the test index, the test result of the target service under the corresponding index can be determined, and the determined test result has higher accuracy. In addition, in the process of comparing the difference between the index information of the experimental group under the test index and the index information of the comparison group under the test index, the false positive proportion of the test result can be reduced, the fluctuation of the test index is reduced, and the sensitivity of the test index is improved through the variance reduction calculation and the significance calculation, so that the minimum experimental flow (namely the number of the objects of the experimental group and the comparison group required by the test) required by the test can be reduced, and the business test cost is saved.

Describing a verification process of the test effect of the service test method, wherein the verification process aims at a first type of error rate and a second type of error rate; the first type error rate refers to the probability of the first type error, and the first type error refers to the error that rejects the true error, i.e. rejects the correct hypothesis; the second type error rate refers to the probability of the second type of error, and the second type of error refers to the error that the original hypothesis is received when the original hypothesis is incorrect during hypothesis test; in the actual test process, the lower the first type error rate and the second type error rate are, the more accurate the test result of the service test method can be shown. The specific verification method comprises the following steps: acquiring real object index data of a plurality of objects (for example, 100 ten thousand objects), simulating a plurality of times (for example, 1000 times) of the service test scheme mentioned in the embodiment of the present application and the service test scheme mentioned in the prior art by using three representative test indexes, namely, an average stay time of the objects, a graph-text click through rate, and a retention rate, and referring to the following tables 2 to 4 for verification results of a first type of error rate:

TABLE 2

TABLE 3

TABLE 4

The table 2 is a verification result for the first type of error rate generated after the service test scheme provided in the embodiment of the present application is verified, the table 3 is a verification result for the first type of error rate generated after the existing service test scheme is verified, and the table 4 is a ratio between actual non-significance and prediction non-significance in the existing service test scheme; as can be seen from the comparison between table 2 and table 3, the first-class error rate of the service test scheme provided in the embodiment of the present application under each test index is lower than that of the existing service test scheme, and it can be seen that the accuracy of the test result is higher when the service test scheme provided in the embodiment of the present application is used for testing.

The results of the verification for the second type of error rate can be seen in tables 5 to 10 below, where tables 5 to 7 correspond to the case of smaller gains given and tables 8 to 10 correspond to the case of larger gains given:

TABLE 5

TABLE 6

TABLE 7

TABLE 8

TABLE 9

Watch 10

Table 5 above shows the second type of error rate verification results generated after the service test scheme provided in the embodiment of the present application is verified given a smaller profit; table 6 is the validation results for the second type of error rate generated after validating the existing business test scheme given the smaller revenue; table 7 is the ratio between actual and predicted significance in the existing traffic test scheme given the smaller gains; table 8 above shows the second type of error rate verification results generated after the service test scheme provided in the embodiment of the present application is verified under the condition of a large profit; table 9 is the validation results for the second type of error rate generated after validating the existing business test scheme given the greater profit; table 10 is the ratio between actual and predicted significance in the existing traffic test scheme given the greater gain; as can be seen from the comparison between tables 5 and 6 and the comparison between tables 8 and 9, the second error rate of the service test scheme provided in the embodiment of the present application under each test index is lower than that of the existing service test scheme, and it can be seen that the accuracy of the test result is higher when the service test scheme provided in the embodiment of the present application is used for testing; in addition, compared with each test index in each table in the longitudinal direction, it is obvious that the service test scheme provided by the embodiment of the application has a more obvious improvement on the index sensitivity of the target average index (for example, the target average stay time), and has a certain effect but is limited on the indexes of the proportion type or the ratio type (for example, the image-text click through rate and the retention rate).

While the method of the embodiments of the present application has been described in detail above, to facilitate better implementation of the above-described aspects of the embodiments of the present application, the apparatus of the embodiments of the present application is provided below accordingly.

Referring to fig. 7, fig. 7 is a schematic structural diagram illustrating a service testing apparatus according to an exemplary embodiment of the present application, where the service testing apparatus may be disposed in a computer device provided in an embodiment of the present application, and the computer device may be a testing server mentioned in the foregoing method embodiment; in some embodiments, the traffic testing apparatus may be a computer program (comprising program code) running in a computer device, which may be used to perform the respective steps in the method embodiments shown in fig. 5 or fig. 6. Referring to fig. 7, the service test apparatus may include the following units:

an obtaining unit 701, configured to obtain an experimental group and a control group of a target service;

the processing unit 702 is configured to perform layering processing on each experimental object in the experimental group according to the historical attribute data of each experimental object in the experimental group, so as to obtain M experimental attribute layers; according to the historical attribute data of each comparison object in the comparison group, carrying out layering processing on each comparison object in the comparison group to obtain M comparison attribute layers, wherein M is a positive integer;

the processing unit 702 is further configured to perform index aggregation calculation on the M experiment attribute hierarchies by using the test indexes to obtain index information of the experiment group under the test indexes; performing index aggregation calculation on the M comparison attribute layers by adopting the test indexes to obtain index information of the comparison group under the test indexes;

the processing unit 702 is further configured to compare a difference between the index information of the experimental group under the test index and the index information of the comparison group under the test index, and determine a test result of the target service according to the difference.

In one implementation, the experimental group includes P experimental subjects, the number of the test indexes is R, and P and R are positive integers; the processing unit 702 is configured to perform index aggregation calculation on the M experiment attribute hierarchies by using the test indexes, and when obtaining index information of the experiment group under the test indexes, specifically configured to execute the following steps:

the processing unit 702 is configured to combine the N random hierarchies with the M experiment attribute hierarchies to obtain N × M experiment comprehensive hierarchies, and when determining an experiment object belonging to each experiment comprehensive hierarchy of the N × M experiment comprehensive hierarchies, specifically configured to execute the following steps:

In one implementation, any one of the R test indexes is represented as the R-th test index, and R is a positive integer less than or equal to R; the processing unit 702 is configured to perform index aggregation calculation on each of the N × M experiment integrated tiers by using the R test indexes, and specifically, when determining index information of each of the N × M experiment integrated tiers under the R test indexes, perform the following steps:

In one implementation, the processing unit 702 is configured to, when randomly dividing P experimental subjects into N random hierarchies, specifically perform the following steps:

In one implementation, any one of the R test indexes is represented as the R-th test index, and R is a positive integer less than or equal to R; a processing unit 702, configured to compare a difference between the index information of the experimental group under the test index and the index information of the control group under the test index, specifically configured to perform the following steps:

In one implementation, any one of the N random hierarchies is represented as the ith random hierarchy, i being a positive integer less than or equal to N; a processing unit 702, configured to perform variance reduction calculation on the index information of each experimental integrated hierarchy in the N × M experimental integrated hierarchies under the r-th test index and the index information of each comparison integrated hierarchy in the N × M comparison integrated hierarchies under the r-th test index, and when determining an aggregate relative difference value of each random hierarchy in the N random hierarchies under the r-th test index, specifically configured to execute the following steps:

In an implementation manner, the processing unit 702 is configured to, when determining the test result of the target service according to the difference, specifically perform the following steps:

In one implementation, the experimental group includes P experimental subjects, P being a positive integer; one experimental attribute hierarchy in the M experimental attribute hierarchies corresponds to one attribute partition space; the processing unit 702 is configured to, when performing layering processing on each experimental object in the experimental group according to the historical attribute data of each experimental object in the experimental group to obtain M experimental attribute layers, specifically perform the following steps:

In one implementation, the processing unit 702 is further configured to perform the following steps:

In one implementation, the number of the test indexes is R, any one of the R test indexes is represented as the R-th test index, R and R are positive integers, and R is less than or equal to R; the experimental group corresponds to a first business strategy of the target business, and the comparison group corresponds to a second business strategy of the target business; the processing unit 702 is further configured to perform the following steps:

According to an embodiment of the present application, the method steps involved in the method shown in fig. 5 or fig. 6 may be performed by the units in the traffic testing apparatus shown in fig. 7. For example, step S501 shown in fig. 5 may be executed by the acquisition unit 701 shown in fig. 7, and steps S502 to S504 shown in fig. 5 may be executed by the processing unit 702 shown in fig. 7. As another example, step S601 shown in fig. 6 may be performed by the acquisition unit 701 shown in fig. 7, and steps S602 to S606 shown in fig. 6 may be performed by the processing unit 702 shown in fig. 7.

According to another embodiment of the present application, the units in the service testing apparatus shown in fig. 7 may be respectively or entirely combined into one or several other units to form the service testing apparatus, or some unit(s) therein may be further split into multiple units with smaller functions to form the service testing apparatus, which may implement the same operation without affecting implementation of technical effects of embodiments of the present application. The units are divided based on logic functions, and in practical application, the functions of one unit can be realized by a plurality of units, or the functions of a plurality of units can be realized by one unit. In other embodiments of the present application, the service testing apparatus may also include other units, and in practical applications, these functions may also be implemented by being assisted by other units, and may be implemented by cooperation of multiple units.

According to another embodiment of the present application, the business testing apparatus as shown in fig. 7 may be constructed by running a computer program (including program codes) capable of executing the steps involved in the corresponding method as shown in fig. 5 or fig. 6 on a general-purpose computing device such as a computer including a processing element such as a Central Processing Unit (CPU), a random access storage medium (RAM), a read-only storage medium (ROM), and a storage element, and the business testing method of the embodiment of the present application may be implemented. The computer program may be embodied on a computer-readable storage medium, for example, and loaded into and executed by the above-described computing apparatus via the computer-readable storage medium.

Based on the above method and apparatus embodiments, the present application provides a computer device, which may be the aforementioned test server. Referring to fig. 8, fig. 8 is a schematic structural diagram illustrating a computer device according to an exemplary embodiment of the present application. The computer device shown in fig. 8 comprises at least a processor 801, an input interface 802, an output interface 803, and a computer-readable storage medium 804. The processor 801, the input interface 802, the output interface 803, and the computer-readable storage medium 804 may be connected by a bus or other means.

The input interface 802 may be used to obtain an experimental group and a control group of a target service, and may also be used to obtain object index data of each experimental object in the experimental group and obtain object index data of each control object in the control group; the output interface 803 may be used to output the test result of the target service to the test client.

A computer-readable storage medium 804 may be stored in the memory of the computer device, the computer-readable storage medium 804 being for storing a computer program comprising computer instructions, the processor 801 being for executing the program instructions stored by the computer-readable storage medium 804. The processor 801 (or CPU) is a computing core and a control core of a computer device, and is adapted to implement one or more computer instructions, and specifically, adapted to load and execute the one or more computer instructions so as to implement a corresponding method flow or a corresponding function.

Embodiments of the present application also provide a computer-readable storage medium (Memory), which is a Memory device in a computer device and is used for storing programs and data. It is understood that the computer readable storage medium herein can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores an operating system of the computer device. Also, one or more computer instructions, which may be one or more computer programs (including program code), are stored in the memory space for loading and execution by the processor. It should be noted that the computer-readable storage medium may be a high-speed RAM Memory, or may be a Non-Volatile Memory (Non-Volatile Memory), such as at least one disk Memory; and optionally at least one computer readable storage medium located remotely from the aforementioned processor.

In one implementation, one or more computer instructions stored in the computer-readable storage medium 804 may be loaded and executed by the processor 801 to implement the corresponding steps described above with respect to the traffic testing method shown in FIG. 5 or FIG. 6. In particular implementations, the computer instructions in the computer-readable storage medium 804 are loaded and executed by the processor 801 to perform the steps of:

acquiring an experimental group and a control group of target services;

In one implementation, the experimental group includes P experimental subjects, the number of the test indexes is R, and P and R are positive integers; the computer instructions in the computer-readable storage medium 804 are loaded by the processor 801 and perform index aggregation calculation on the M experiment attribute hierarchies by using the test indexes, so as to obtain index information of the experiment group under the test indexes, and specifically, the following steps are performed:

the computer instructions in the computer-readable storage medium 804 are loaded and executed by the processor 801 to combine the N random hierarchies with the M experiment attribute hierarchies to obtain N × M experiment integrated hierarchies, and to determine an experiment object belonging to each experiment integrated hierarchy of the N × M experiment integrated hierarchies, and are specifically configured to perform the following steps:

In one implementation, any one of the R test indexes is represented as the R-th test index, and R is a positive integer less than or equal to R; the computer instructions in the computer-readable storage medium 804 are loaded by the processor 801 and executed to perform index aggregation calculation on each of the N × M experimental integrated hierarchies using the R test indexes, and when determining the index information of each of the N × M experimental integrated hierarchies under the R test indexes, the computer instructions are specifically configured to perform the following steps:

In one implementation, the computer instructions in the computer-readable storage medium 804 when loaded and executed by the processor 801 to randomly divide the P subjects into N random hierarchies are specifically configured to perform the following steps:

In one implementation, any one of the R test indexes is represented as the R-th test index, and R is a positive integer less than or equal to R; the computer instructions in the computer-readable storage medium 804 are loaded and executed by the processor 801 to compare the difference between the index information of the experimental group under the test index and the index information of the control group under the test index, and specifically to perform the following steps:

In one implementation, any one of the N random hierarchies is represented as the ith random hierarchy, i being a positive integer less than or equal to N; the computer instructions in the computer-readable storage medium 804 are loaded by the processor 801 and executed to perform a variance reduction calculation on the index information of each experimental integrated hierarchy in the N × M experimental integrated hierarchies under the r-th test index and the index information of each comparative integrated hierarchy in the N × M comparative integrated hierarchies under the r-th test index, and to determine an aggregate relative difference value of each random hierarchy in the N random hierarchies under the r-th test index, and specifically to perform the following steps:

In one implementation, when the computer instructions in the computer-readable storage medium 804 are loaded by the processor 801 and executed to determine the test result of the target service according to the difference, the following steps are specifically performed:

In one implementation, the experimental group includes P experimental subjects, P being a positive integer; one experimental attribute hierarchy in the M experimental attribute hierarchies corresponds to one attribute partition space; the computer instructions in the computer-readable storage medium 804 are loaded and executed by the processor 801 to perform layering processing on each experimental object in the experimental group according to the historical attribute data of each experimental object in the experimental group, so as to obtain M experimental attribute layers, and are specifically configured to perform the following steps:

In one implementation, the computer instructions in the computer-readable storage medium 804 are loaded by the processor 801 and are further configured to perform the steps of:

In one implementation, the number of the test indexes is R, any one of the R test indexes is represented as the R-th test index, R and R are positive integers, and R is less than or equal to R; the experimental group corresponds to a first business strategy of the target business, and the comparison group corresponds to a second business strategy of the target business; the computer instructions in the computer-readable storage medium 804 are loaded by the processor 801 and are further used to perform the steps of:

According to an aspect of the application, a computer program product or computer program is provided, comprising computer instructions, the computer instructions being stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the service test method provided in the above-mentioned various alternatives.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for service testing, the method comprising:

acquiring an experimental group and a control group of target services;

according to the historical attribute data of each experimental object in the experimental group, carrying out layering processing on each experimental object in the experimental group to obtain M experimental attribute layers; according to the historical attribute data of each comparison object in the comparison group, carrying out layering treatment on each comparison object in the comparison group to obtain M comparison attribute layers, wherein M is a positive integer;

performing index aggregation calculation on the M experimental attribute layers by adopting test indexes to obtain index information of the experimental group under the test indexes; performing index aggregation calculation on the M comparison attribute layers by adopting the test indexes to obtain index information of the comparison group under the test indexes;

and comparing the difference between the index information of the experimental group under the test index and the index information of the control group under the test index, and determining the test result of the target service according to the difference.

2. The method of claim 1, wherein the experimental group comprises P experimental subjects, the number of test markers is R, and P and R are both positive integers; the method for performing index aggregation calculation on the M experiment attribute layers by adopting the test indexes to obtain the index information of the experiment group under the test indexes comprises the following steps:

randomly dividing the P experimental objects into N random layers, wherein N is a positive integer;

combining the N random hierarchies with the M experiment attribute hierarchies to obtain N × M experiment comprehensive hierarchies, and determining an experiment object belonging to each experiment comprehensive hierarchy in the N × M experiment comprehensive hierarchies;

3. The method of claim 2, wherein any one of the nxm experimental synthesis hierarchies is represented as an experimental synthesis hierarchy X_i*Y_j，X_iRepresents the ith random hierarchy, Y, of the N random hierarchies_jRepresenting the jth experiment attribute hierarchy in the M experiment attribute hierarchies, wherein i is a positive integer less than or equal to N, and j is a positive integer less than or equal to M;

the combining the N random hierarchies with the M experiment attribute hierarchies to obtain N × M experiment integrated hierarchies, and determining an experiment object belonging to each experiment integrated hierarchy of the N × M experiment integrated hierarchies, includes:

combining the ith random hierarchy with the jth experiment attribute hierarchy to obtain the experiment comprehensive hierarchy X_i*Y_j；

4. The method of claim 3, wherein any one of the R test indicators is represented as the R-th test indicator, R being a positive integer less than or equal to R; the method for performing index aggregation calculation on each experiment comprehensive layer in the N × M experiment comprehensive layers by using R test indexes to determine the index information of each experiment comprehensive layer in the N × M experiment comprehensive layers under the R test indexes includes:

obtaining the experimental comprehensive stratification X_i*Y_jSubject index data for each subject in (a);

determining index information of a correlation index of the r-th test index from the object index data;

calculating the experimental comprehensive layering X according to the determined index information of the associated index_i*Y_jIndex information under the r-th test index.

5. The method of claim 2, wherein said randomly partitioning the P subjects into N random stratification comprises:

and randomly dividing the P experimental objects into the N random hierarchies according to the object hash of the P experimental objects.

6. The method of claim 2, wherein any one of the R test metrics is represented as an R-th test metric, R being a positive integer less than or equal to R; the comparing the difference between the indicator information of the experimental group under the test indicator and the indicator information of the control group under the test indicator comprises:

performing variance reduction calculation on the index information of each experimental comprehensive layer in the N multiplied by M experimental comprehensive layers under the r-th test index and the index information of each comparison comprehensive layer in the N multiplied by M comparison comprehensive layers under the r-th test index, and determining the polymerization relative difference value of each random layer in the N random layers under the r-th test index;

7. The method of claim 6, wherein any one of the N random hierarchies is represented as an ith random hierarchy, i being a positive integer less than or equal to N; the determining the aggregate relative difference value of each random hierarchy of the N random hierarchies under the r-th test index by performing variance reduction calculation on the index information of each experimental comprehensive hierarchy of the N × M experimental comprehensive hierarchies under the r-th test index and the index information of each control comprehensive hierarchy of the N × M control comprehensive hierarchies under the r-th test index includes:

calculating M initial relative differences between the index information of each experiment comprehensive layer in the M experiment comprehensive layers under the r-th test index and the index information of each comparison comprehensive layer in the M comparison comprehensive layers under the r-th test index; wherein, the jth initial relative difference value in the M initial relative difference values is obtained by calculation according to the index information of the jth experiment comprehensive layer in the M experiment comprehensive layers under the r-th test index and the index information of the jth comparison comprehensive layer in the M comparison comprehensive layers under the r-th test index, and j is a positive integer less than or equal to M;

and performing weighted aggregation on the M initial relative difference values to obtain an aggregated relative difference value of the ith random hierarchy under the r test index.

8. The method of claim 6, wherein said determining a test result of the target service based on the difference comprises:

performing significance calculation on the polymerization relative difference value of each random hierarchy in the N random hierarchies under the r test index to obtain a significance result of the target service under the r test index;

9. The method of claim 1, wherein said experimental group comprises P subjects, P being a positive integer; one experimental attribute hierarchy in the M experimental attribute hierarchies corresponds to one attribute partition space; the step of performing layering processing on each experimental object in the experimental group according to the historical attribute data of each experimental object in the experimental group to obtain M experimental attribute layers includes:

determining attribute partition areas to which the historical attribute data of any one of the P experimental subjects belongs;

and dividing any one experimental object into the experimental attribute layers corresponding to the determined attribute division areas.

10. The method of claim 9, wherein the method further comprises:

and screening the experimental objects in the M experimental attribute layers by adopting an attribute screening threshold value, and determining the experimental objects of which the historical attribute data in the M experimental attribute layers meet the attribute screening threshold value.

11. The method of claim 1, wherein the number of test indicators is R, any one of the R test indicators is represented as the R-th test indicator, R and R are both positive integers, and R is less than or equal to R; the experimental group corresponds to a first business strategy of the target business, and the comparison group corresponds to a second business strategy of the target business; the method further comprises the following steps:

12. A traffic testing apparatus, characterized in that the apparatus comprises:

the processing unit is used for carrying out layering processing on each experimental object in the experimental group according to the historical attribute data of each experimental object in the experimental group to obtain M experimental attribute layers; according to the historical attribute data of each comparison object in the comparison group, carrying out layering treatment on each comparison object in the comparison group to obtain M comparison attribute layers, wherein M is a positive integer;

the processing unit is further configured to perform index aggregation calculation on the M experiment attribute hierarchies by using test indexes to obtain index information of the experiment group under the test indexes; performing index aggregation calculation on the M comparison attribute layers by adopting the test indexes to obtain index information of the comparison group under the test indexes;

the processing unit is further configured to compare a difference between the index information of the experimental group under the test index and the index information of the control group under the test index, and determine a test result of the target service according to the difference.

13. A computer device, the device comprising:

a processor adapted to implement a computer program; and the number of the first and second groups,

a computer-readable storage medium, having stored thereon a computer program adapted to be loaded by the processor and to execute the traffic testing method according to any of claims 1 to 11.

14. A computer-readable storage medium, characterized in that the computer-readable storage medium comprises a computer program adapted to be loaded by a processor and to execute the traffic testing method according to any of claims 1 to 11.