CN110908901B - Automatic verification method and system for image recognition capability - Google Patents
Automatic verification method and system for image recognition capability Download PDFInfo
- Publication number
- CN110908901B CN110908901B CN201911095025.3A CN201911095025A CN110908901B CN 110908901 B CN110908901 B CN 110908901B CN 201911095025 A CN201911095025 A CN 201911095025A CN 110908901 B CN110908901 B CN 110908901B
- Authority
- CN
- China
- Prior art keywords
- test
- training
- offset
- data
- unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3684—Test management for test design, e.g. generating new test cases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3688—Test management for test execution, e.g. scheduling of test suites
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/172—Classification, e.g. identification
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- General Engineering & Computer Science (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Human Computer Interaction (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- General Health & Medical Sciences (AREA)
- Image Analysis (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides an automatic verification system of image recognition capability, which comprises a data management module, a configuration management module, a test execution module and a report display module; the method comprises the steps of storing various types of image resources as training sets and test sets through a data management module, and setting test types corresponding to test execution and required interface information through a configuration management module; the test execution module automatically executes training and testing according to the configuration information, records the test information in the execution process, and finally generates a corresponding test report through the report display module. The invention also provides an automatic verification method for the image recognition capability, which can complete automatic test, greatly reduce the input time of the test and relieve manual operation.
Description
Technical Field
The invention relates to the technical field of image recognition, in particular to an automatic verification method and system for image recognition capability.
Background
Image recognition is an important direction in the field of intelligence, and it is important for a recognition system to have a sufficiently high recognition rate. How to measure the image recognition capability becomes an important link. At present, the scenes such as face recognition, flower recognition and animal recognition are common in the field of image recognition, but at present, the most common test mode is to verify the recognition effect or capability of a system by manually shooting and uploading, and in addition, regression verification of automatic traversal is performed by preset data, so that whether return is expected or not is verified.
The existing image identification verification technology has the following defects:
the first disadvantage is that if the function test is accepted, the data size is very weak when the data size is high although the user is intuitive. The method is particularly prominent in regression testing of version iteration or BUG repair, program modification can only need a few lines of codes, the verified testing set range is hundreds, thousands of times, and the method is time-consuming and easy to wear out the endurance of testing personnel through manual testing.
The second disadvantage is that the accuracy verification of image recognition is performed in a terminal (client) based manner, which cannot effectively measure a response condition of the server in response time. Because the response time of the client includes other consumption + server consumption times such as picture processing of the client.
Third, conventional script execution can only verify an identification effect under the current system environment, including but not limited to the identification results such as similarity and response time, but cannot measure a dynamic performance condition of the whole system. Such as: the service with 100 units of training set and the service based on 100000 units of training set are different in performance when recognizing images.
The fourth disadvantage is that conventional image recognition mainly consists in verifying whether the image recognition result is correct, i.e. whether an object in the image is recognized. The recognition that a certain object is based on the similarity ratio of the picture to the most similar picture in the current training set may cause that the corresponding person cannot be recognized if the set threshold is too high, and may cause that the recognition is error and false recognition occurs if the set threshold is too low. However, conventional test methods do not give appropriate tuning reference suggestions because there is no data support.
Disclosure of Invention
In order to overcome the problems, the invention aims to provide an automatic verification system with image recognition capability, which can complete automatic testing, greatly reduce the test investment time and relieve manual operation.
The invention is realized by adopting the following scheme: an automated verification system of image recognition capabilities, the system comprising a data management module, a configuration management module, a test execution module, and a report display module;
the data management module is used for storing unit information under each type of image recognition, and the unit information comprises a test set picture and a training set picture;
the configuration management module is used for configuring a training interface and a testing interface required during testing, and a training set and a testing range required to be tested;
the test execution module is used for carrying out picture resource training of units under each range according to the information configured by the configuration management module, and executing the test of the corresponding unit test set after the training under the units of the range is completed;
the report display module is used for generating a corresponding test report;
the method comprises the steps of storing various types of image resources as training sets and test sets through a data management module, and setting test types corresponding to test execution and required interface information through a configuration management module; the test execution module automatically executes training and testing according to the configuration information, records the test information in the execution process, and finally generates a corresponding test report through the report display module.
Further, the data management module is further specifically: creating a data type of image recognition, and distributing a unique identification type_id for the data type; creating a unit under the data type, supplementing corresponding unit information, and distributing a unique identification unit_id for the unit; and uploading the training set pictures and the test set pictures in the unit under the data type.
Furthermore, the data management module can provide a group of api interfaces of restful for operators to create units through scripts and upload data of corresponding training sets and test sets without each image resource uploading being operated manually.
Further, the configuration management module is further specifically: creating relevant configuration executed by a test execution module, and firstly setting names and descriptions for the configuration items; then selecting the data type of the image resource; configuring relevant information of a training interface, including URL addresses of the requested training interface, for configuring training URLs of corresponding services of each identification type for automatic training of the system; configuring relevant information of a test interface, including a URL address of the test interface requested, wherein the URL address is used for configuring a test URL of each service corresponding to each identification type for automatic test of a system; finally, configuring the offset condition, wherein the offset condition is set by a user in a user-defined way, and the offset condition comprises but is not limited to: full-order recursion, delta recursion, and power recursion; the offset of the offset in the full-quantity recursion is 1, and an offset parameter P is required to be transmitted in the differential quantity recursion; the power of the power recursion is the requirement for an incoming offset parameter Q.
Further, the test execution module is further specifically: reading current configuration information of a configuration management module, acquiring a current offset range, and outputting a first offset as 1; since the offset 1 is smaller than the total unit number, the training range N is [ offset_old, offset ], offset_old defaults to 1, and the offset is a configured offset parameter; automatically traversing the training data in the unit in the range of [ offset_old, offset ], uploading through a training interface to complete training, and after the training is completed, testing and verifying, namely testing the range M as [1, offset ], namely performing testing set verification in the unit in the current range in full quantity after each system training is completed; automatically reading test data in units within the range of [1, offset ], uploading and verifying through a test interface, and recording request and returned response data and response time in each test process; and generating a report and pushing a message to finish the test.
Further, the report display module is further specifically: after the test is completed, the average response time under different training sets is counted, and a visual line graph is generated; when the test is completed and the test data is stored, the fields under the returned json data are automatically analyzed and stored in the database for the user to inquire or display a report.
In addition, the present invention provides an automatic verification method of image recognition capability, the method comprising the steps of:
step S1, storing unit information under each type of image recognition, wherein the unit information comprises a test set picture and a training set picture;
step S2, configuring a training interface and a testing interface required during testing, and configuring a training set and a testing range to be tested;
step S3, training the picture resources of units in each range according to the information configured by the configuration management module, and executing the test of the corresponding unit test set after the training of the units in the range is completed;
and S4, obtaining a test result to generate a corresponding test report.
Further, the step S1 is further specifically: creating a data type of image recognition, and distributing a unique identification type_id for the data type; creating a unit under the data type, supplementing corresponding unit information, and distributing a unique identification unit_id for the unit; and uploading the training set pictures and the test set pictures in the unit under the data type.
Furthermore, the method can provide a group of api interfaces of restful for operators to create units through scripts and upload data of corresponding training sets and test sets without each image resource uploading being operated manually.
Further, the step S2 is further specifically: creating a relevant configuration of the test, and firstly setting a name and a description for the configuration item; then selecting the data type of the image resource; configuring relevant information of a training interface, including URL addresses of the requested training interface, for configuring training URLs of corresponding services of each identification type for automatic training of the system; configuring relevant information of a test interface, including a URL address of the test interface requested, wherein the URL address is used for configuring a test URL of each service corresponding to each identification type for automatic test of a system; finally, configuring the offset condition, wherein the offset condition is set by a user in a user-defined way, and the offset condition comprises but is not limited to: full-order recursion, delta recursion, and power recursion; the offset of the offset in the full-quantity recursion is 1, and an offset parameter P is required to be transmitted in the differential quantity recursion; the power of the power recursion is the requirement for an incoming offset parameter Q.
Further, the step S3 is further specifically: reading current configuration information, acquiring a current offset range, and outputting a first offset to be 1; since the offset 1 is smaller than the total unit number, the training range N is [ offset_old, offset ], offset_old defaults to 1, and the offset is a configured offset parameter; automatically traversing the training data in the unit in the range of [ offset_old, offset ], uploading through a training interface to complete training, and after the training is completed, testing and verifying, namely testing the range M as [1, offset ], namely performing testing set verification in the unit in the current range in full quantity after each system training is completed; automatically reading test data in units within the range of [1, offset ], uploading and verifying through a test interface, and recording request and returned response data and response time in each test process; and generating a report and pushing a message to finish the test.
Further, the step S4 is further specifically: after the test is completed, the average response time under different training sets is counted, and a visual line graph is generated; when the test is completed and the test data is stored, the fields under the returned json data are automatically analyzed and stored in the database for the user to inquire or display a report.
Furthermore, the method supports multithreading execution test, the thread number thread_count needs to be filled in during test, and threads and data are distributed according to the thread_count input during user execution and are executed concurrently.
The invention has the beneficial effects that:
1. through the mode that this patent provided carries out the automated test of image recognition ability, the input time of reduction test that can be very big liberates manual operation.
2. By means of the method, the image recognition capability is automatically tested, one high concurrency execution of the script can be achieved through multithreading, and one test round trip time of the program is effectively shortened.
3. The image recognition capability is tested in the mode provided by the patent, the user-defined report presentation can be realized, the report presentation is realized in the configuration mode, and the development investment is reduced.
4. The data storage of the image recognition capability test result is carried out in the mode provided by the patent, so that a performance response report under different training sets can be intuitively displayed.
5. The image recognition type classification management is carried out in the mode provided by the patent, so that the image recognition fields in different aspects can be fully multiplexed, and the image recognition fields include but are not limited to the face recognition field.
6. Through the mode that this patent provided carries out the test of image recognition ability, can realize an automation of image training, need not artifical the participation.
7. The mode that provides through this patent carries out the test of image recognition ability, can realize an automation of image test, need not artifical the participation.
8. The image recognition capability is tested in the mode provided by the patent, so that the recognition similarity statistics of the training and non-training images can be counted, and the analysis and decision of threshold selection are facilitated.
Drawings
Fig. 1 is a system frame diagram of the present invention.
FIG. 2 is a schematic diagram of a specific workflow of the data management module of the present invention.
FIG. 3 is a schematic diagram of a specific workflow of the configuration management module of the present invention.
FIG. 4 is a schematic diagram of a specific workflow of the test execution module of the present invention.
Fig. 5 is a schematic flow chart of the method of the present invention.
Detailed Description
The invention is further described below with reference to the accompanying drawings.
Referring to fig. 1 to 4, an automated verification system for image recognition capability according to the present invention includes a data management module, a configuration management module, a test execution module, and a report display module;
the data management module is used for storing unit information under each type of image recognition, and the unit information comprises a test set picture and a training set picture;
the configuration management module is used for configuring a training interface and a testing interface required during testing, and a training set and a testing range required to be tested;
the test execution module is used for carrying out picture resource training of units under each range according to the information configured by the configuration management module, and executing the test of the corresponding unit test set after the training under the units of the range is completed;
the report display module is used for generating a corresponding test report;
the method comprises the steps of storing various types of image resources as training sets and test sets through a data management module, and setting test types corresponding to test execution and required interface information through a configuration management module; the test execution module automatically executes training and testing according to the configuration information, records the test information in the execution process, and finally generates a corresponding test report through the report display module.
As shown in fig. 2, the data management module is further specifically: creating a data type of image recognition, and distributing a unique identification type_id for the data type; creating a unit under the data type, supplementing corresponding unit information, and distributing a unique identification unit_id for the unit; and uploading the training set pictures and the test set pictures in the unit under the data type.
The data management module can provide a group of api interfaces of restful for operators to carry out unit creation through scripts and upload data of corresponding training sets and test sets without each image resource uploading being operated manually.
As shown in fig. 3, the configuration management module is further specifically: creating relevant configuration executed by a test execution module, and firstly setting names and descriptions for the configuration items; then selecting the data type of the image resource; configuring relevant information of a training interface, including URL addresses of the requested training interface, for configuring training URLs of corresponding services of each identification type for automatic training of the system; configuring relevant information of a test interface, including a URL address of the test interface requested, wherein the URL address is used for configuring a test URL of each service corresponding to each identification type for automatic test of a system; finally, configuring the offset condition, wherein the offset condition is set by a user in a user-defined way, and the offset condition comprises but is not limited to: full-order recursion, delta recursion, and power recursion; the offset of the offset in the full-quantity recursion is 1, and an offset parameter P is required to be transmitted in the differential quantity recursion; the power of the power recursion is the requirement for an incoming offset parameter Q.
As shown in fig. 4, the test execution module is further specifically: reading current configuration information of a configuration management module, acquiring a current offset range, and outputting a first offset as 1; since the offset 1 is smaller than the total unit number, the training range N is [ offset_old, offset ], offset_old defaults to 1, and the offset is a configured offset parameter; automatically traversing the training data in the unit in the range of [ offset_old, offset ], uploading through a training interface to complete training, and after the training is completed, testing and verifying, namely testing the range M as [1, offset ], namely performing testing set verification in the unit in the current range in full quantity after each system training is completed; automatically reading test data in units within the range of [1, offset ], uploading and verifying through a test interface, and recording request and returned response data, response time and unit_id of a test unit quoted by the test in each test process; and generating a report and pushing a message to finish the test.
The report display module is further specifically: after the test is completed, the average response time under different training sets is counted, and a visual line graph is generated; when the test is completed and the test data is stored, the fields under the returned json data are automatically analyzed and stored in the database together with other data (1, unit_id of the reference test unit; 2, response time timeline;3, request) for the user to inquire or display reports.
The invention is further described in connection with one embodiment as follows:
taking the face recognition type as an example, the invention stores various image resources (such as face recognition, flower recognition and animal recognition type images) as a training set and a test set through a data management module, and then sets the test type corresponding to test execution and the required interface information through a configuration management module; the test execution module automatically executes training and testing according to the configuration information, records the test information in the execution process, and finally generates a corresponding test report through the report display module.
Specific:
1. the data management module comprises the following contents:
1. for creation and management of data types. Taking the face recognition type as an example, a storage type named (face recognition) can be created, and the system can allocate a unique identification type_id for the storage type (the type_id is used as a database self-increment and is used as a data source identification when configuration management and test execution read resources);
2. creating a unit under the type (face recognition) and each person (each person is a unit, here only exemplified face recognition, flower recognition, etc. can be used, so that no matter what type of object is, the description will be expressed in units), supplementing corresponding unit information (description information such as person name can be supplemented but not limited to), and the system can allocate a unique identification unit_id (same type_id, unit_id is self-added to a database and is used as a data source recognition when configuration management and test are performed to read resources);
3. the uploading of training sets and test sets is supported in units under face recognition type. In order to avoid empty resource phenomenon of the unit during training or testing, the system still limits at least more than one picture resource of the test set and the training set in the unit when the unit is stored.
4. Preferably, in view of the fact that the project itself may already be provided with a data source (the data source includes but is not limited to the project itself, and may also be a selected third party), the data management module still provides a set of api interfaces for related operators to create units through scripts and upload data of corresponding training and testing sets, without requiring each resource upload to be manually operated.
2. The configuration management module comprises the following contents:
1. creating relevant configuration of test execution, firstly setting names and descriptions for the configuration items;
2. and then the resource type is selected, for example, the current data management background already has face recognition and flower recognition, and the type can be directly selected (face recognition);
3. configuring relevant information of a training interface, including URL addresses of the requested training interface, for configuring training URLs of corresponding services of each identification type for automatic training of the system;
4. the same relevant information for configuring the test interface comprises a URL address of the test interface requested, and the URL address is used for configuring a test URL of each identification type corresponding service for automatic testing of the system;
5. and finally, configuring an offset condition, wherein an offset object refers to a unit number range under the type. Default to 1, i.e., full recursion. The offset conditions may be user-defined and selected, including but not limited to 1) full-scale recursion, 2) delta recursion, 3) power recursion, and the like. Where the delta recursion requires an incoming offset parameter P (for delta recursion, i.e. if the input P is 10, the system will recursively execute with 10 offsets, generating a rule of 1, 11, 21, 31 … 1+n x P), the same power recursion also requires an incoming offset parameter Q (for power recursion, i.e. if the input Q is 10, the system will multiply with a base of 10, generating a rule of 1,10,100,1000 … x Q x n). Preferably, the system still supports a user-defined offset manner, first creating an offset condition name (custom offset condition one), then entering an offset list, e.g., 1,2,4,10,100,1000,10000, which the program will automatically partition into a list of [1,2,4,10,100,1000,10000 ].
3. The test execution module comprises the following contents:
1. after the configuration management module creates the configuration, two buttons (training test and non-training test) for executing the test are arranged on the configuration content page, and an automatic execution program is triggered after the button (training test) is clicked;
2. firstly, the preferential system reads the current configuration information, and presumes that a face recognition type test is configured currently, wherein 10000 unit information is stored in the face recognition, and meanwhile, the system sets the offset condition as power recursion and Q is 10. Then the system will calculate the current offset range preferentially after reading the information, and output the first offset as 1;
3. subsequently, offset 1 is less than the total unit number 10000, so that the training range N is calculated as [1,1] (N is calculated as [ offset_old, offset ], offset_old defaults to 1, when the second cycle starts with offset_old equal to the offset value of the previous round, i.e., the first round is [1,1], the second round is [1,10 ], the third round is [10, 100], the fourth round is [100, 1000], and the fifth round is [1000, 10000 ]);
4. after the range of N is calculated, the system automatically traverses training data in units in the range of [ offset_old, offset ] in (face recognition) and uploads the training data through a training interface, namely after the link is finished, the system has the face recognition capability of personnel in the current range;
5. after the training of the face recognition capability is completed, the face recognition capability of the person needs to be verified. All systems preferably perform the calculation of the identification data range M. The test range M is [1, offset ]. After each system training is finished, the test set verification under the unit in the current range is carried out in full quantity;
6. after calculating the range of M, the system automatically reads the test data in the unit of the [1, offset ] range in the face recognition, and performs uploading verification through the test interface, and records the request, the returned response data and the response time in each test process.
Remarks:
the training set of image recognition refers to: the data management module manages the picture resources of the training set under each type unit.
Examples: a face recognition type is created. Wherein, there are two units of the Ming and xiao Hua, the training set of the unit of the Ming will upload at least one picture (photo) A of the Ming, and the same test set refers to a picture B in the test set of the unit of the object.
The test scope list refers to: the unit range stored in the data management module to be used for each test.
Examples: the face recognition under the data management module stores 10000 units of data such as xiaoming, xiao Hua and the like, and the test range referred to herein is an area in the 10000 units. In the same example, 10000 units of data are in the face recognition type, Q is 10, then the list of the offsets is [1,10,100,1000,10000], and according to the training set range formula [ offset_old, offset ], the test set range formula [1, offset ], the training set of 1 unit is trained for the first round, and then the test set of one unit is tested; training a training set of 9 units in the second round, and then testing a testing set of 10 units; the third round trains the training set of 90 units, tests the test set of 100 units; the fourth wheel trains the training set of 900 units and tests the test set of 1000 units; the fifth round trained 9000 units of training set and tested 10000 units of testing set.
The objects in the training set and the testing set are picture resources. The test and training interfaces each include a field image (picture resource field), and the training interface further includes an ID (unit ID field, which is used as a unique identifier).
Examples: 1 unit named Xiaoming is created under face recognition, and when training, the id of the unit and an image picture under a training set are uploaded and trained through a training interface; then in the test stage, the image pictures under the test set under the unit of the Ming are carried to be uploaded and tested through the test interface, the id corresponding to the Ming is identified, and meanwhile, the Similarity is returned.
7. After the step 6 is completed, the program calculates and reads the next offset, calculates whether the offset is smaller than the total unit number, if so, continues to return to the step 4 until the offset is larger than the total unit number, generates a report and pushes a message, and completes the test;
8. more preferably, the system can verify the test of the system in a training unit and also support the test of the system in a non-training unit. After clicking on the (untrained test), the procedure repeats the operations of 3) 4) 5) 6) 7) 8), except that the range of M in 6) will become [ offset, total number of units ]. The unit without training is tested each time, so that a duty ratio condition of misrecognition can be checked in the report;
9. preferably, the system still supports multi-thread execution test, and the configuration content is provided with a switch for enabling acceleration execution besides two buttons for test execution (in training test and non-training test), and the switch is off by default, but when the switch is on, the thread number thread_count needs to be filled. The difference between the multithread execution is that in the test set verification link in 6), threads and data can be allocated according to the thread_count input by the user during execution, and concurrent execution can be performed.
4. The report display module comprises the following contents:
1. after the test is completed, the priority program automatically counts the average face recognition response time under different training sets, and generates a visual line graph for the program and test operators to check a performance trend of the service. Preferably, the system still supports viewing a ranking list of response times in each hierarchical test set;
2. preferably, when the system stores the test data, the system automatically analyzes the fields under the returned json data and stores the fields in the database. To this end, the system supports the use of custom database query commands to count or display reports. (e.g., the similarity is ordered in positive order, then it is convenient to see how much similarity needs to be thresholded to determine if the person is the same person, and similarly, it is also useful to query whether there are unidentified persons in the test set, and what the duty cycle is.
Referring to fig. 4, the present invention provides an automatic verification method of image recognition capability, the method comprising the steps of:
step S1, storing unit information under each type of image recognition, wherein the unit information comprises a test set picture and a training set picture;
step S2, configuring a training interface and a testing interface required during testing, and configuring a training set and a testing range to be tested;
step S3, training the picture resources of units in each range according to the information configured by the configuration management module, and executing the test of the corresponding unit test set after the training of the units in the range is completed;
and S4, obtaining a test result to generate a corresponding test report.
The step S1 is further specifically: creating a data type of image recognition, and distributing a unique identification type_id for the data type; creating a unit under the data type, supplementing corresponding unit information, and distributing a unique identification unit_id for the unit; and uploading the training set pictures and the test set pictures in the unit under the data type. The method can provide a group of api interfaces of restful for operators to create units through scripts and upload data of corresponding training sets and test sets without the need of manually operating each image resource upload.
The step S2 is further specifically: creating a relevant configuration of the test, and firstly setting a name and a description for the configuration item; then selecting the data type of the image resource; configuring relevant information of a training interface, including URL addresses of the requested training interface, for configuring training URLs of corresponding services of each identification type for automatic training of the system; configuring relevant information of a test interface, including a URL address of the test interface requested, wherein the URL address is used for configuring a test URL of each service corresponding to each identification type for automatic test of a system; finally, configuring the offset condition, wherein the offset condition is set by a user in a user-defined way, and the offset condition comprises but is not limited to: full-order recursion, delta recursion, and power recursion; the offset of the offset in the full-quantity recursion is 1, and an offset parameter P is required to be transmitted in the differential quantity recursion; the power of the power recursion is the requirement for an incoming offset parameter Q.
The step S3 is further specifically: reading current configuration information, acquiring a current offset range, and outputting a first offset to be 1; since the offset 1 is smaller than the total unit number, the training range N is [ offset_old, offset ], offset_old defaults to 1, and the offset is a configured offset parameter; automatically traversing the training data in the unit in the range of [ offset_old, offset ], uploading through a training interface to complete training, and after the training is completed, testing and verifying, namely testing the range M as [1, offset ], namely performing testing set verification in the unit in the current range in full quantity after each system training is completed; automatically reading test data in units within the range of [1, offset ], uploading and verifying through a test interface, and recording request and returned response data and response time in each test process; and generating a report and pushing a message to finish the test.
The step S4 is further specifically: after the test is completed, the average response time under different training sets is counted, and a visual line graph is generated; when the test is completed and the test data is stored, the fields under the returned json data are automatically analyzed and stored in the database for the user to inquire or display a report.
The method supports multithreading execution test, the thread number thread_count needs to be filled in during the test, threads and data are distributed according to the thread_count input during the execution of a user, and concurrent execution is performed.
The specific application scene of the invention is as follows:
specific application scene
Scene one:
the classmate A is responsible for testing a face recognition system, and is mainly used for completing testing of service recognition capability by uploading pictures on terminal software because of no script writing capability. In this process, the small A can only be identified by taking the photos collected by itself as test data. Then, when the service is updated every time, the small A uploads all the current photos one by one for testing, and the efficiency is very low, so that the small A is strange by project responsible persons.
Through the scheme of the patent, the classmate's small A can directly store the photos collected by the classmate's small A in the data management platform, configure training and testing interface information of the face recognition system through the platform, and then configure corresponding testing data sources, how to click on the platform (in-training test), the test can be automatically completed by the program, and statistical results are output, especially when the data is returned, the generation of the testing results can be completed only by clicking one key.
Scene II:
the classmate B is a development of face recognition service, and it is not clear at first how much the similarity probability is appropriate, so a threshold of 99% is set only by feel. Results at the time of the C-classmate test, the C-classmate found that he had imported his own photograph. However, the photo taken at a slightly changed angle cannot be recognized. Thus small B adjusts the threshold to 70%. At this time, small C finds that the photo of another colleague is also identified as itself. Thus, small B lets small C give a suitable similarity threshold and then he adjusts, but small C does not know how much similarity is appropriate to adjust specifically.
Through the scheme of the patent, after the related training set and test set are arranged by the classmate C and uploaded to the data management service, only clicking (non-training test) is needed, after the test is completed, the results are ordered according to the reverse similarity mode, and a corresponding threshold value can be given according to the expected recognition rate of the project, for example, the expected recognition probability of the project party is 99%, then the error recognition can only be 1%, and the similarity of the G test set (G=1%. Count) can be taken as the similarity value.
Scene III:
the classmate D is responsible for testing a face recognition system, but only if the function of the verified face recognition is satisfied, namely, if the face imported into the face recognition system is correctly recognized by other faces of the corresponding person, the corresponding information is correctly recognized by the classmate D. For the performance related content, small D is not very concerned, and because the test data possessed by small D is limited, no performance related problem is found in the test process. However, after the face recognition system delivers the product, the lead-in user is too much, the recognition efficiency of the customer service feedback system is very low, so that the user experience is poor, and the lead-in user is very unsatisfactory.
Through the scheme of this patent, the classmate little D only needs to arrange relevant training set and test set to upload it in the data management service, and select the power recursion, input Q is 10, and this system can be automatic statistics at 1 at this moment, 10,100,1000,10000 etc. a system response performance statistics diagram under different levels training set. Performance problems are found in advance when the training set unit exceeds 10000. And promotes the project side to improve the service hardware capability or optimize a program of the face recognition system.
Scene four:
the classmate F is responsible for testing a face recognition system, the data on the line is copied to a testing environment every time by the classmate F, so that the unit number is tens of thousands, and when the program is optimized in version, the script can be executed by the classmate F to carry out regression, and the execution efficiency is very low due to the large data volume.
Through the scheme of this patent, classmate little D only need arrange relevant training set and test set to in uploading it to data management service, when the test, select to open (accelerate the switch of carrying out), and fill in parallel line journey number, can realize concurrent execution, compare effectual reduction test time with original, realize quick test, quick return.
The foregoing description is only of the preferred embodiments of the invention, and all changes and modifications that come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.
Claims (6)
1. An automated verification system for image recognition capabilities, characterized by: the system comprises a data management module, a configuration management module, a test execution module and a report display module;
the data management module is used for storing unit information under each type of image recognition, and the unit information comprises a test set picture and a training set picture;
the configuration management module is used for configuring a training interface and a testing interface required during testing, and a training set and a testing range required to be tested;
the test execution module is used for carrying out picture resource training of units under each range according to the information configured by the configuration management module, and executing the test of the corresponding unit test set after the training under the units of the range is completed;
the report display module is used for generating a corresponding test report;
the method comprises the steps of storing various types of image resources as training sets and test sets through a data management module, and setting test types corresponding to test execution and required interface information through a configuration management module; the test execution module automatically executes training and testing according to the configuration information, records the test information in the execution process, and finally generates a corresponding test report through the report display module;
the data management module is further specifically: creating a data type of image recognition, and distributing a unique identification type_id for the data type; creating a unit under the data type, supplementing corresponding unit information, and distributing a unique identification unit_id for the unit; uploading the training set pictures and the test set pictures in units under the data type;
the configuration management module is further specifically: creating related configuration executed by a test execution module, and firstly setting names and descriptions for configuration items of the related configuration; then selecting the data type of the image resource; configuring relevant information of the training interface, including the URL address of the requested training interface, for configuring training URL of the service corresponding to each identification type for automatic training of the system; configuring relevant information of a test interface, including a requested test interface URL address, for configuring a test URL of each identification type corresponding service for automatic test of a system; finally, configuring the offset condition, wherein the offset condition is set by a user in a user-defined way, and the offset condition comprises but is not limited to: full-order recursion, delta recursion, and power recursion; the offset of the offset in the full-quantity recursion is 1, and an offset parameter P is required to be transmitted in the differential quantity recursion; the power recursion is the need for an incoming offset parameter Q; the test execution module is further specifically: reading current configuration information of a configuration management module, acquiring a current offset range, and outputting a first offset as 1; since the offset 1 is smaller than the total unit number, the training range N is [ offset_old, offset ], offset_old defaults to 1, and the offset is a configured offset parameter; automatically traversing the training data in the unit in the range of [ offset_old, offset ], uploading through a training interface to complete training, and after the training is completed, testing and verifying, namely testing the range M as [1, offset ], namely performing testing set verification in the unit in the current range in full quantity after each system training is completed; automatically reading test data in units within the range of [1, offset ], uploading and verifying through a test interface, and recording request and returned response data and response time in each test process; and generating a report and pushing a message to finish the test.
2. An automated verification system of image recognition capabilities of claim 1, wherein: the data management module can provide a group of api interfaces of restful for operators to carry out unit creation through scripts and upload data of corresponding training sets and test sets without each image resource uploading being operated manually.
3. An automated verification system of image recognition capabilities of claim 1, wherein: the report display module is further specifically: after the test is completed, the average response time under different training sets is counted, and a visual line graph is generated; when the test is completed and the test data is stored, the fields under the returned json data are automatically analyzed and stored in the database for the user to inquire or display a report.
4. An automatic verification method for image recognition capability is characterized in that: the method comprises the following steps:
step S1, storing unit information under each type of image recognition, wherein the unit information comprises a test set picture and a training set picture;
step S2, configuring a training interface and a testing interface required during testing, and configuring a training set and a testing range to be tested;
step S3, training the picture resources of units in each range according to the information configured by the configuration management module, and executing the test of the corresponding unit test set after the training of the units in the range is completed;
s4, obtaining a test result to generate a corresponding test report;
the step S1 is further specifically: creating a data type of image recognition, and distributing a unique identification type_id for the data type; creating a unit under the data type, supplementing corresponding unit information, and distributing a unique identification unit_id for the unit; uploading the training set pictures and the test set pictures in units under the data type; the step S2 is further specifically: creating a relevant configuration of a test, and firstly setting names and descriptions for configuration items of the relevant configuration; then selecting the data type of the image resource; configuring relevant information of the training interface, including the URL address of the requested training interface, for configuring training URL of the service corresponding to each identification type for automatic training of the system; configuring relevant information of a test interface, including a requested test interface URL address, for configuring a test URL of each identification type corresponding service for automatic test of a system; finally, configuring the offset condition, wherein the offset condition is set by a user in a user-defined way, and the offset condition comprises but is not limited to: full-order recursion, delta recursion, and power recursion; the offset of the offset in the full-quantity recursion is 1, and an offset parameter P is required to be transmitted in the differential quantity recursion; the power recursion is the need for an incoming offset parameter Q;
the step S3 is further specifically: reading current configuration information, acquiring a current offset range, and outputting a first offset to be 1; since the offset 1 is smaller than the total unit number, the training range N is [ offset_old, offset ], offset_old defaults to 1, and the offset is a configured offset parameter; automatically traversing the training data in the unit in the range of [ offset_old, offset ], uploading through a training interface to complete training, and after the training is completed, testing and verifying, namely testing the range M as [1, offset ], namely performing testing set verification in the unit in the current range in full quantity after each system training is completed; automatically reading test data in units within the range of [1, offset ], uploading and verifying through a test interface, and recording request and returned response data and response time in each test process; and generating a report and pushing a message to finish the test.
5. The method for automatically verifying image recognition capabilities of claim 4, wherein: the method can provide a group of api interfaces of restful for operators to create units through scripts and upload data of corresponding training sets and test sets without the need of manually operating each image resource upload.
6. The method for automatically verifying image recognition capabilities of claim 4, wherein: the step S4 is further specifically: after the test is completed, the average response time under different training sets is counted, and a visual line graph is generated; when the test is completed and the test data is stored, the fields under the returned json data are automatically analyzed and stored in the database for the user to inquire or display a report.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911095025.3A CN110908901B (en) | 2019-11-11 | 2019-11-11 | Automatic verification method and system for image recognition capability |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911095025.3A CN110908901B (en) | 2019-11-11 | 2019-11-11 | Automatic verification method and system for image recognition capability |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110908901A CN110908901A (en) | 2020-03-24 |
| CN110908901B true CN110908901B (en) | 2023-05-02 |
Family
ID=69816627
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911095025.3A Active CN110908901B (en) | 2019-11-11 | 2019-11-11 | Automatic verification method and system for image recognition capability |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110908901B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112153378B (en) * | 2020-09-27 | 2023-05-26 | 福建天晴数码有限公司 | Method and system for testing video auditing capability |
| CN114154562B (en) * | 2021-11-15 | 2023-04-21 | 国网河北省电力有限公司电力科学研究院 | Device and method for detecting image recognition capability of intelligent monitoring terminal |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106529427A (en) * | 2016-10-24 | 2017-03-22 | 合肥华凌股份有限公司 | Method and system of identifying target object in image, and method and system of identifying food in refrigerator |
| CN108460649A (en) * | 2017-02-22 | 2018-08-28 | 阿里巴巴集团控股有限公司 | A kind of image-recognizing method and device |
| CN109614488A (en) * | 2018-12-04 | 2019-04-12 | 广西大学 | Distribution network live line work condition distinguishing method based on text classification and image recognition |
| CN109766915A (en) * | 2018-12-14 | 2019-05-17 | 深圳壹账通智能科技有限公司 | Test method, device, computer equipment and storage medium based on image recognition |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9760990B2 (en) * | 2014-12-14 | 2017-09-12 | International Business Machines Corporation | Cloud-based infrastructure for feedback-driven training and image recognition |
-
2019
- 2019-11-11 CN CN201911095025.3A patent/CN110908901B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106529427A (en) * | 2016-10-24 | 2017-03-22 | 合肥华凌股份有限公司 | Method and system of identifying target object in image, and method and system of identifying food in refrigerator |
| CN108460649A (en) * | 2017-02-22 | 2018-08-28 | 阿里巴巴集团控股有限公司 | A kind of image-recognizing method and device |
| CN109614488A (en) * | 2018-12-04 | 2019-04-12 | 广西大学 | Distribution network live line work condition distinguishing method based on text classification and image recognition |
| CN109766915A (en) * | 2018-12-14 | 2019-05-17 | 深圳壹账通智能科技有限公司 | Test method, device, computer equipment and storage medium based on image recognition |
Non-Patent Citations (1)
| Title |
|---|
| 基于卷积神经网络与迁移学习的油茶病害图像识别;龙满生 等;《农业工程学报》;第194-201页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110908901A (en) | 2020-03-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10733079B2 (en) | Systems and methods for end-to-end testing of applications using dynamically simulated data | |
| US10453161B2 (en) | Watermarking and scalability techniques for a virtual desktop planning tool | |
| US9424481B2 (en) | Screenshot database for application verification | |
| US8954397B2 (en) | Creation and replay of a simulation workload using captured workloads | |
| EP2778929B1 (en) | Test script generation system | |
| AU2015315522B2 (en) | Data-driven testing framework | |
| CN109409533B (en) | Method, device, equipment and storage medium for generating machine learning model | |
| CN110908901B (en) | Automatic verification method and system for image recognition capability | |
| CN111401722B (en) | Intelligent decision method and intelligent decision system | |
| US10067859B2 (en) | Constructing test-centric model of application | |
| CN108197155A (en) | Information data synchronous method, device and computer readable storage medium | |
| US10713070B2 (en) | Systems and methods for capturing and visualizing user interactions across devices | |
| US10901875B2 (en) | Evaluating and presenting software testing project status indicators | |
| CN112799782B (en) | Model generation system, method, electronic device and storage medium | |
| CN112463588A (en) | Automatic test system and method, storage medium and computing equipment | |
| US11169910B2 (en) | Probabilistic software testing via dynamic graphs | |
| CN111190817B (en) | Method and device for processing software defects | |
| US11995587B2 (en) | Method and device for managing project by using data merging | |
| CN106202162A (en) | A kind of for testing the test system and method recommending room data list | |
| CN113923443A (en) | Network video recorder testing method and device and computer readable storage medium | |
| WO2024119471A1 (en) | Algorithm testing method, algorithm testing platform, and computer readable storage medium | |
| CN111553375A (en) | Using transformations to verify computer vision quality | |
| CN115484560B (en) | Intelligent short message processing method and device, electronic equipment and storage medium | |
| US20220405676A1 (en) | Method and device for managing project by using data filtering | |
| US20220405678A1 (en) | Method and device for managing project by using data pointer |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |