CN113011473A - Model optimization method, model optimization device and electronic equipment - Google Patents

Model optimization method, model optimization device and electronic equipment Download PDF

Info

Publication number
CN113011473A
CN113011473A CN202110229068.7A CN202110229068A CN113011473A CN 113011473 A CN113011473 A CN 113011473A CN 202110229068 A CN202110229068 A CN 202110229068A CN 113011473 A CN113011473 A CN 113011473A
Authority
CN
China
Prior art keywords
test
image
model optimization
model
static
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110229068.7A
Other languages
Chinese (zh)
Inventor
许政伟
姜璐
李铁岭
杨宇喆
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Industrial and Commercial Bank of China Ltd ICBC
Original Assignee
Industrial and Commercial Bank of China Ltd ICBC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Industrial and Commercial Bank of China Ltd ICBC filed Critical Industrial and Commercial Bank of China Ltd ICBC
Priority to CN202110229068.7A priority Critical patent/CN113011473A/en
Publication of CN113011473A publication Critical patent/CN113011473A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computational Linguistics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Evolutionary Biology (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Image Analysis (AREA)

Abstract

The present disclosure provides a model optimization method, a model optimization device, and an electronic apparatus, which can be used in the field of artificial intelligence, finance, or other fields, the method including: sending a test request including a set identifier to a server so that the server can obtain a value of a static index of a test image in a test image set corresponding to the set identifier; responding to threshold input operation, and sending index thresholds of at least part of static indexes to the server side so that the server side determines the test accuracy rate aiming at the test image set based on the values of the static indexes of the test images and the index thresholds; and responding to the test accuracy rate from the server side, and displaying the test accuracy rate so as to determine model optimization information.

Description

Model optimization method, model optimization device and electronic equipment
Technical Field
The present disclosure relates to the field of artificial intelligence technology and the field of finance, and more particularly, to a model optimization method, a model optimization apparatus, and an electronic device.
Background
Artificial intelligence techniques are finding increasingly widespread use in areas such as image recognition. Before the model is formally released to be on-line, whether the model meets the on-line standard needs to be tested, and meanwhile, the model is iterated and optimized according to a test result.
In the process of realizing the concept disclosed by the invention, the applicant finds that at least the following problems exist in the related technology, the test condition of the model is complex, the related technical test indexes are numerous, and the technical test indexes are discrete and have weak relevance. This does not facilitate modelers to identify the problems with the model from a single test index.
Disclosure of Invention
In view of this, the present disclosure provides a model optimization method, a model optimization apparatus, and an electronic device, which dynamically combine static indicators, so that a user can determine the influence of a single static indicator on a model identification result by adjusting the value of the static indicator, so as to quickly determine the problem existing in the model.
One aspect of the present disclosure provides a model optimization method performed by a terminal, including: sending a test request including a set identifier to a server so that the server can obtain a value of a static index of a test image in a test image set corresponding to the set identifier; responding to threshold input operation, and sending index thresholds of at least part of static indexes to the server side so that the server side determines the test accuracy rate aiming at the test image set based on the values of the static indexes of the test images and the index thresholds; and responding to the test accuracy rate from the server side, and displaying the test accuracy rate so as to determine model optimization information.
One aspect of the present disclosure provides a method for model optimization performed by a server, including: receiving a test request from a terminal, wherein the test request comprises a set identifier of a test image set; processing the test image set corresponding to the set identification by using the trained model to obtain a value of a static index of the test image in the test image set; responding to an index threshold value of at least part of static indexes from the terminal, and determining the test accuracy rate aiming at the test image set based on the value of the static indexes and the index threshold value; and sending the test accuracy to the terminal so as to determine model optimization information.
One aspect of the present disclosure provides a model optimization apparatus, which is disposed in a terminal, and includes: the device comprises a request sending module, an index threshold sending module and an accuracy rate display module. The request sending module is used for sending a test request comprising a set identifier to the server so that the server can obtain a value of a static index of a test image in a test image set corresponding to the set identifier; the index threshold value sending module is used for responding to threshold value input operation and sending at least part of index threshold values of the static indexes to the server side so that the server side can determine the test accuracy rate aiming at the test image set based on the values of the static indexes of the test images and the index threshold values; and the accuracy rate display module is used for responding to the test accuracy rate from the server side and displaying the test accuracy rate so as to determine the model optimization information.
One aspect of the present disclosure provides a model optimization apparatus, disposed at a server, including: the device comprises a request receiving module, an image processing module, an accuracy rate determining module and an accuracy rate sending module. The request receiving module is used for receiving a test request from a terminal, wherein the test request comprises a set identifier of a test image set; the image processing module is used for processing the test image set corresponding to the set identification by using the trained model to obtain a value of a static index of the test image in the test image set; the accuracy rate determining module is used for responding to an index threshold value of at least part of static indexes from the terminal, and determining the test accuracy rate aiming at the test image set based on the value of the static indexes and the index threshold value; and the accuracy sending module is used for sending the test accuracy to the terminal so as to determine the model optimization information.
Another aspect of the present disclosure provides an electronic device comprising one or more processors and a storage device, wherein the storage device is configured to store executable instructions, which when executed by the processors, implement the method as above.
Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions for implementing the method of training a model optimization model and/or the model optimization method as above when executed.
Another aspect of the present disclosure provides a computer program comprising computer executable instructions for implementing a method of training a model optimization model and/or a model optimization method as above when executed.
According to the model optimization method, the model optimization device and the electronic equipment provided by the embodiment of the disclosure, when model testing is performed, a user can adjust the index threshold of the static index on line, so that a plurality of static indexes can be dynamically linked, and a modeling worker or a testing worker can be helped to quickly determine the problems of the model based on the testing accuracy rate linked with the index threshold.
According to the model optimization method, the model optimization device and the electronic equipment provided by the embodiment of the disclosure, part of static indexes which are closely matched with the service indexes are set to be dynamically adjustable, so that service-related workers can adjust the static indexes conveniently, the technical threshold of the personnel participating in the test is effectively reduced, the service-related workers and the like can give model optimization suggestions based on self service experiences conveniently, and the problem that a model which only depends on the technical test cannot meet the requirements of a specific scene is solved.
The model optimization method, the model optimization device and the electronic equipment provided by the embodiment of the disclosure are convenient for a modeling worker or a testing worker to determine and identify the category of the error image by observing the error test image, are beneficial to assisting in optimizing the training image set, and effectively accelerate the model iteration speed. In addition, the test results of multiple iteration versions are conveniently longitudinally compared, and a model suitable for the current scene is conveniently selected.
According to the model optimization method, the model optimization device and the electronic equipment, by combining the experience of business related workers, invalid optimization on some rare categories can be avoided, the optimization direction of the model can be better met in the selection of training images, the model is close to the indexes of business, the iteration of the model is accelerated, the iteration process is optimized, and the development time is saved.
Drawings
The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure with reference to the accompanying drawings, in which:
fig. 1 schematically illustrates an application scenario of a model optimization method, a model optimization apparatus and an electronic device according to an embodiment of the present disclosure;
FIG. 2 schematically illustrates an exemplary system architecture to which a model optimization method, a model optimization apparatus, may be applied, according to an embodiment of the disclosure;
FIG. 3 schematically illustrates a flow chart of a model optimization method according to an embodiment of the present disclosure;
FIG. 4 schematically shows a schematic view of a test image according to an embodiment of the present disclosure;
FIG. 5 schematically illustrates a linkage analysis interface diagram of confidence and intersection ratios, in accordance with an embodiment of the present disclosure;
FIG. 6 schematically shows a diagram of test accuracy for a set of test images, in accordance with an embodiment of the present disclosure;
FIG. 7 schematically illustrates a schematic diagram of identifying an erroneous image according to an embodiment of the present disclosure;
FIG. 8 schematically illustrates a flow diagram of a model optimization method according to another embodiment of the present disclosure;
FIG. 9 schematically illustrates a logic diagram of a model optimization method according to an embodiment of the present disclosure;
FIG. 10 schematically illustrates a block diagram of a model optimization apparatus according to an embodiment of the present disclosure;
FIG. 11 schematically illustrates a block diagram of a model optimization apparatus according to another embodiment of the present disclosure; and
FIG. 12 schematically shows a block diagram of an electronic device according to an embodiment of the disclosure.
Detailed Description
Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.
All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.
Where a convention analogous to "A, B or at least one of C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B or C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). The terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more features.
In enterprise applications, pre-online testing is required before a model is formally released online. The test condition of the model is complex, the test indexes relate to multi-class recognition effects, the positions of the target positioning frames and the like, more than ten common test indexes exist, and the applicant finds that the following problems exist.
On one hand, the technical test indexes are discrete and have weak relevance. Such as loss (loss) values, confusion matrices, cross-over ratios, etc., it is inconvenient for modelers to locate problems with the model directly from a single index.
On the one hand, modelers may lack experience of relevant scenes and cannot provide model improvement schemes in time. In enterprise application, a method completely based on technical test cannot be intuitively close to business indexes, cannot fully integrate the characteristics of personnel in each professional field, and is not beneficial to analysis and optimization of a model.
On the one hand, a purely static model test result does not facilitate obtaining a comprehensive test result for the model. And a method for dynamic real-time linkage adjustment after parameter adjustment is lacked, so that iterative testing of technical indexes and service indexes and three-dimensional comprehensive model effect display are not convenient to realize.
The embodiment of the disclosure provides a model optimization method, a model optimization device and electronic equipment. The model optimization method executed by the terminal comprises an instruction sending process and an information display process, and is characterized in that a test request comprising a set identifier is sent to a server side firstly, so that the server side can obtain values of static indexes of a test image in a test image set corresponding to the set identifier, and then, in response to threshold value input operation, at least part of index threshold values of the static indexes are sent to the server side, so that the server side can determine the test accuracy rate of the test image set based on the values of the static indexes of the test image and the index threshold values. And entering an information display process after the instruction sending process is completed, and displaying the test accuracy in response to the test accuracy from the server so as to determine the model optimization information.
Fig. 1 schematically illustrates an application scenario of a model optimization method, a model optimization apparatus and an electronic device according to an embodiment of the present disclosure.
As shown in fig. 1, a user may set at least part of the static indicators on the terminal device, for example, modify confidence, intersection ratio, and/or value of the confusion matrix of the recognition result, so that the obtained output result of the model changes, for example, the test accuracy and the recognition error image subset change. Therefore, the user and/or the terminal can determine the model optimization information based on the change of the model output result, and the model developer can perform iterative optimization on the model.
Fig. 2 schematically illustrates an exemplary system architecture to which the model optimization method, model optimization apparatus, according to an embodiment of the present disclosure, may be applied. It should be noted that fig. 2 is only an example of a system architecture to which the embodiments of the present disclosure may be applied to help those skilled in the art understand the technical content of the present disclosure, and does not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments or scenarios. It should be noted that the model optimization method, the model optimization device, and the electronic device provided in the embodiments of the present disclosure may be used in the field of artificial intelligence in terms of model optimization, and may also be used in various fields other than the field of artificial intelligence, such as the financial field, and the like.
As shown in fig. 2, the system architecture 200 according to this embodiment may include terminal devices 201, 202, 203, a network 204 and a server 205. The network 204 may include a plurality of gateways, routers, hubs, network wires, etc. to provide a medium for communication links between the end devices 201, 202, 203 and the server 205. Network 204 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
The user may use the terminal device 201, 202, 203 to interact with other terminal devices and the server 205 via the network 204 to receive or send information, etc., such as upload set identifiers, send requests, upload test images, send model optimization instructions, receive recognition results, etc. The terminal devices 201, 202, 203 may be installed with various communication client applications, such as web browser applications, banking-like applications, e-commerce-like applications, search-like applications, office-like applications, instant messaging tools, mailbox clients, social platform software, etc. (just examples).
The terminal devices 201, 202, 203 include, but are not limited to, electronic devices that can support functions such as web browsing, such as smart phones, desktop computers, augmented reality devices, tablet computers, laptop computers, and the like. The terminal device can download a trained model to identify the image.
The server 205 may receive and process model training requests, model testing requests, model optimization requests, image processing requests, model download requests, and the like. For example, the server 205 may be a back office management server, a cluster of servers, or the like. The background management server can analyze and process the received model training request, model optimization request, image processing request and the like, and feed back processing results (such as recognition results, test results, model training results and the like) to the terminal equipment.
It should be noted that the model optimization method provided by the embodiments of the present disclosure may be executed by the terminal devices 201, 202, 203 or the server 205. Accordingly, the model optimization device provided by the embodiment of the present disclosure may be disposed in the terminal device 201, 202, 203 or the server 205. It should be understood that the number of terminal devices, networks, and servers are merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
FIG. 3 schematically shows a flow chart of a model optimization method according to an embodiment of the disclosure. As shown in fig. 3, the model optimization method may include operations S302 to S306.
In operation S302, a test request including a set identifier is sent to the server, so that the server obtains a value of a static indicator of a test image in a test image set corresponding to the set identifier.
In this embodiment, the test request may be generated based on text type information input by the user, or may be generated based on voice type information input by the user.
Specifically, the user may log in the server side by using a browser of the terminal, so as to select a test image set to be used in a user interface displayed by the browser. The set of test images may be determined based on a user-selected set identification.
For example, after the user sends a test request to the server through the terminal, the server may determine a corresponding test image set based on the set identifier included in the test request, and then input the test images in the test image set into a trained model (e.g., an image recognition model trained by a model developer) to obtain an image recognition result. The image recognition result may include intermediate results and final results, such as intermediate results output by each layer of the neural network, results output by each sub-network, and results output by the entire neural network. Wherein, the value of the static index can be the intermediate result of the model. The test accuracy may be the final result of the model output or a statistical result of the final result for the overall model output.
In one embodiment, the test image has annotation positioning frame information and object annotation information for the image within the annotation positioning frame.
FIG. 4 schematically shows a schematic diagram of a test image according to an embodiment of the disclosure.
As shown in fig. 4, the test image is an image of a contract, and the image includes a plurality of terms, contract signing dates, and identity information and certification information of both parties of the contract, such as company seal, representative signature, and the like. The test image has annotation information, and the annotation information can include an image identifier, an annotation positioning frame, object annotation information for the image in the annotation positioning frame, and the like. The labeling information can be manually labeled and can be used as supervision information of the training model. Fig. 4 includes 4 positioning frames, and the object labeling information of the image in each positioning frame is the first party stamp, the second party signature, the legal representative and the order date.
Accordingly, in the process of identifying the test image by the trained image, the object labeling information of the image in the target area needs to be determined. The target area can be represented by a positioning box which is automatically determined by a model needing to be trained, and the text information or the information type included in the image in the positioning box is determined. The positioning frame may be represented by coordinates, such as point coordinates of two points at opposite angles.
In a specific embodiment, a user selects parameters such as a test data set and a container according to a model to be tested, and the parameters such as the container can be dynamically bound through a control at the front end. The test data set may be a test image set, and the test images in the test image set have annotation information, such as image identification (id), location frame information, and identification content, which are stored in a database and input into the model together with the transmission of the test image set.
In operation S304, in response to the threshold input operation, at least part of the index threshold of the static index is sent to the server side, so that the server side determines the test accuracy for the test image set based on the value of the static index of the test image and the index threshold.
In this embodiment, a user may set an index threshold of a static index on an interactive interface of a browser of a terminal, and transmit the index threshold to a server, so that the server may determine a test result for a test image based on the index threshold. For example, the test results are affected by modifying a confidence threshold of the recognition results. If the recognition result of the test image 1 includes the first party stamp, the confidence of the recognition result is 90%. If the confidence threshold is modified from 89% to 91%, the recognition result output by the model may change from a first-party seal to a non-first-party seal, which may affect test accuracy.
In one embodiment, for a scenario in which a test image has annotation positioning box information and object annotation information for an image within the annotation positioning box, the static metrics may include: and aiming at the confidence coefficient of the recognition result of the test image and the intersection ratio of the labeling positioning frame, the intersection ratio of the labeling positioning frame represents the coincidence proportion of the recognition positioning frame relative to the labeling positioning frame.
FIG. 5 schematically illustrates a linkage analysis interface diagram of confidence and intersection ratios according to an embodiment of the disclosure.
As shown in fig. 5, the user may set a threshold of the confidence level and/or the cross-over ratio in the interactive interface, so as to send the threshold of the confidence level and/or the cross-over ratio to the server side.
In operation S306, the test accuracy is presented in response to the test accuracy from the server side in order to determine model optimization information.
In this embodiment, the test accuracy may be calculated by the server side or may be calculated by the terminal side. The model optimization information may be determined by the user based on set confidence and/or threshold of intersection ratio, and test accuracy from the server side, such as based on expert experience. The model optimization information may be determined based on a preset rule or a preset mapping relationship, the set confidence and/or threshold of the cross-over ratio, and the test accuracy from the server side.
For example, the test accuracy is the ratio between the number of test images in the subset of identified faulty images and the total number of test images in the set of test images. The interactive interface shown in fig. 5 may further display an identification error image, so that a user can conveniently view a reason of the identification error, and may further display an identification positioning frame, a labeling positioning frame, object labeling information, and object identification information corresponding to the identification error image.
FIG. 6 schematically shows a diagram of test accuracy for a set of test images, according to an embodiment of the disclosure.
As shown in FIG. 6, the user has tested the trained model using different sets of test images (test1, test2, and test4), respectively, with 75%, 82%, and 97% accuracy, respectively.
In one embodiment, the model optimization information includes: at least one of a type of the training image set, a number of training images of the training image set, a number of layers of the network, a model parameter, and a size of the training images. By feeding back the model optimization information to the server side, the model optimization information can be fed back to model construction personnel or model training personnel quickly, so that the model can be optimized.
In one embodiment, after demonstrating the test accuracy, the method may further include the following operations: and responding to the optimization information input operation, and sending the model optimization information to the server side. The user can send model optimization suggestions to the server side through the terminal based on personal experience or business experience and the like. For example, in the process of viewing the recognition error image subset, the service personnel finds that the recognition error image belongs to an unusual service scene, and can propose the following suggestions: the error case belongs to a rare service scene, a large amount of time cost and computing resources are not needed to be consumed to retrain the model, and manual input is prompted according to the rare service scene. For another example, if the test images of a plurality of error cases belong to the same category, it indicates that the number of training images for the category is insufficient, and the number of training images needs to be increased. For example, identifying at least a portion of a subset of erroneous images is shown to include: for each of the at least partially identified false images of the presentation, presenting the identified false image and at least one of: and the identification positioning frame, the marking positioning frame, the object marking information and the object identification information correspond to the identification error image.
In an embodiment, after sending the metric threshold of at least part of the static metrics to the server, the method may further include the following operations: and receiving a subset of the identified error images for the test image set from the server side.
Accordingly, in response to the test accuracy from the server side, the demonstrating the test accuracy so as to determine the model optimization information may include the following operations: and responding to the identification error image subset and the test accuracy from the server, and displaying at least part of the identification error image subset and the index threshold and the test accuracy of the static indexes, so that the user determines model optimization information based on the identification error image subset, the index threshold and the test accuracy of the static indexes.
The embodiment of the disclosure facilitates the combination of the experience of professional service personnel, can avoid the invalid optimization of some rare categories, can better accord with the optimization direction of the model in the selection of the training image, is close to the index of the service, accelerates the iteration of the model, optimizes the iterative flow and saves the development time.
In order to improve the quality of the model optimization information given by the user, the client can also display prompt information. For example, a preset mapping relationship may be presented to guide the user.
In one embodiment, the method may further include the following operations.
For example, a preset mapping relationship is displayed, and the preset mapping relationship includes a corresponding relationship between an index threshold and/or a test accuracy of the static index and the model optimization information.
For another example, model optimization information corresponding to the identified faulty image subset is determined based on at least the preset mapping relationship, the index threshold of the static index, and the test accuracy.
In a specific embodiment, if the recognition result confidence is less than a first threshold and the intersection ratio is less than a second preset threshold, the model optimization information includes at least one of the following: and the labeling information of the training image set is abnormal, the number of the training image sets is smaller than a preset image number threshold value, the model is under-fitted, and the size of the input image is increased.
If the confidence of the recognition result is smaller than a first threshold and the intersection ratio is larger than a second preset threshold, the model optimization information comprises at least one of the following: the number of the training image sets is smaller than a preset image number threshold value, the number of rounds of model training is increased, the number of images input in each round of training is reduced, the training images are sliced, and at least one of the images of which the types are overlapped with other types exists.
If the confidence of the recognition result is greater than a first threshold and the intersection ratio is less than a second preset threshold, the model optimization information comprises at least one of the following: and the labeling information of the training image set is abnormal, the quality of the training image set is improved, and the size of the input image is increased.
If the type of at least part of the images in the error image subset is identified to belong to a rare type in the business application, the model optimization information comprises at least one of reducing or canceling the training image set belonging to the rare type and prompting manual processing. For example, when a service person performs a test, it is found that the recognition accuracy of some types of test images is low, and according to the past service entry experience, the category has an extremely low proportion in the whole scene, so that the data enhancement and iterative optimization of the category can be ignored, and a manual entry mode is adopted every time the category is touched, so that the time for model optimization is greatly reduced, and the modeling efficiency is improved. In the selection of the training set, the quantity of each category of data can be selected according to the input condition of business personnel in the past. For some frequently-occurring classes, more training image sets are provided, so that the recognition accuracy is improved, and for some frequently-occurring classes, less training image sets can be provided.
In one embodiment, the static indicator further comprises a confusion matrix. Accordingly, if the first fraction of false negative classes relative to negative classes is greater than a third threshold and/or the second fraction of false positive classes relative to positive classes is greater than a fourth threshold, the model optimization information includes at least one of: the number of the training image sets is smaller than a preset image number threshold value, the number of rounds of model training is increased, the number of pictures input in each round of training is reduced, the pictures are sliced, and at least one of the images of which the types of the training images are overlapped with other types exists.
Due to the fact that the model training effect is poor, the training graph set needs to be modified. For example, when the intersection ratio of a certain category is significantly lower or is not detected at all, the training images of the category need to be added into the training image set to improve the feature extraction of the model on the category, so as to improve the recognition accuracy.
Referring to the schematic diagram of the confidence and cross-over linkage analysis interface shown in fig. 5, if the test result for a certain test image set is successful (the status is successful as shown in fig. 6), the user can click the operation button to view the error case information. Images meeting the conditions can be screened from the database at the server side by adjusting two static indexes given in the interface of fig. 5 and are transmitted to the terminal. And drawing the labeling positioning frame, the identifying content and the labeling content by using a plug-in of the terminal, and simultaneously giving out a corresponding abnormal analysis and model optimization suggestion according to the current intersection ratio (IOU), the confidence coefficient parameter, the category and the training image ratio.
For example, if both the IOU and confidence are below the desired threshold, then it is indicated that: 1) the labeling has problems, 2) the quality and the quantity of the data do not reach the standard, and 3) the model is not fit enough, and model parameters need to be adjusted, such as increasing the number of training rounds epoch, adjusting the number of training images step of each round of training, increasing the image size of the input model and the like. If the IOU meets expectations but the confidence is too low, it indicates that: 1) the number of the training images is insufficient, the training images are not fully trained by the model, 2) because the model is not fit to the training images, the number of training rounds epoch can be properly increased, the number of pictures input in each training round is reduced, and the pictures are sliced (ThreePartscut, FourPatrcut and the like), and 3) the training images with the identification categories overlapped with other categories exist. If the confidence exceeds the threshold but the IOU is low, then it indicates: 1) obvious deviation exists during data annotation, re-annotation work needs to be carried out on the data, and 2) the quality of a training image is improved, and the image size of an input model is increased. If the detection algorithm is classified into two categories, the information of the confusion matrix can also be added. If the proportion of the false negative class and the false positive class is too high, the provided optimization suggestion has high intersection-to-parallel ratio and consistent confidence degree.
Fig. 7 schematically shows a schematic diagram of identifying an erroneous image according to an embodiment of the present disclosure.
As shown in fig. 7, the solid line frame is a labeled positioning frame, the dashed line frame is an identified positioning frame, and it can be seen that the coincidence ratio of the two positioning frames is higher, but the identification result is wrong, and if the proportion of such wrong cases relative to all wrong cases is higher, an optimization suggestion can be given according to the above mapping relationship: for example, the number of training images is insufficient, the number of training rounds is increased, there are training images with identity cards and seals overlapped, more similar training images need to be added or more similar training images need to be manufactured by using an image technology.
According to the method and the device, the problem existing in the model can be known by modeling personnel quickly by combining a plurality of indexes from the past static indexes to the dynamic indexes, and the categories can be determined quickly to be subjected to incremental optimization by observing wrong cases, so that the iteration speed of the model is greatly increased, and meanwhile, the model which is most suitable for the current scene can be selected better by comparing results of a plurality of iteration versions longitudinally.
Another aspect of the present disclosure provides a method of model optimization performed by a server-side.
FIG. 8 schematically shows a flow chart of a method of model optimization according to another embodiment of the present disclosure.
As shown in fig. 8, the model optimization method may include operations S802 to S808.
In operation S802, a test request is received from a terminal, the test request including a set identification of a set of test images.
In operation S804, the trained model is used to process the test image set corresponding to the set identifier, so as to obtain a value of a static indicator for the test image in the test image set.
In addition, after the model processes the input test image set, the information such as the coordinates of the positioning frames, the number of the positioning frames, the identification content, the confidence coefficient and the like can be returned, and the information returned by the model is stored in the database corresponding to the test image.
In one embodiment, the recognition and positioning accuracy and the cross-comparison information of the whole test image set are calculated, and the information is respectively filled into a database to prepare data for error case display and model linkage analysis.
And calculating the intersection ratio according to the positioning information returned by the model and the coordinates of the upper left vertex and the lower right vertex of the two positioning frames, wherein the calculation formula is shown as the formula (1).
Figure BDA0002958184990000131
Harea is the combined area of two positioning frames, e.g., (x)1,y1),(x2,y2) Coordinates of the top left vertex of the label orientation box and the identification orientation box, respectively, (x)3,y3),(x4,y4) Respectively, the coordinates of the lower right vertex of the marking positioning frame and the identifying positioning frame. Wherein, the calculation formula of harea is shown as formula (2).
harea＝(min(x2,x4)-Max(x1,x3))*(min(y2,y4)-Max(y1,y3) Formula (2)
The cross-over ratio can be calculated through the above method. It should be noted that, in this embodiment, the process of determining the intersection ratio is performed at the server side for illustration. However, the process may also be implemented in the terminal, for example, the server side sends the coordinates of the vertex of the identification positioning frame and the coordinates of the vertex of the labeling positioning frame to the terminal, so that the terminal calculates the intersection ratio.
The confidence of the recognition result is output by the model, and the higher the confidence, the higher the probability that the recognition result is the correct result.
In operation S806, a test accuracy rate for the test image set is determined based on a value of the static indicator and the indicator threshold in response to the indicator threshold of at least part of the static indicator from the terminal.
In this embodiment, the calculation formula of the test accuracy is as follows, and for the picture with inconsistent number of the positioning frames and identification positioning frames or inconsistent identification content, an error picture is identified and an error label (tag) is marked. The calculation formula of the test accuracy is shown in formula (3).
Figure BDA0002958184990000141
It should be noted that the operation of testing accuracy may also be performed by the terminal.
In operation S808, the test accuracy is transmitted to the terminal in order to determine model optimization information.
In one embodiment, after determining the test accuracy for the set of test images, the method may further include the following operations.
First, a subset of recognition error images corresponding to the test accuracy is determined.
Then, the identification error image subset is sent to the terminal, so that the terminal can display the identification error image subset.
For example, the test image has annotation positioning frame information and object annotation information for the image within the annotation positioning frame.
Accordingly, the static metrics include: and aiming at the confidence coefficient of the recognition result of the test image and the intersection ratio of the labeling positioning frame, the intersection ratio of the labeling positioning frame represents the coincidence proportion of the recognition positioning frame relative to the labeling positioning frame.
FIG. 9 schematically illustrates a logic diagram of a model optimization method according to an embodiment of the present disclosure.
As shown in fig. 9, after the user selects the test image set at the terminal, the server side processes the selected test image set, and the server side feeds back a processing result to the terminal. In addition, the server side can calculate the intersection ratio and the test accuracy according to the result returned by the model and the identification information of the test image set, and stores the intersection ratio and the test accuracy in the database. And the server feeds the determined result back to the terminal so that the terminal foreground can display the test accuracy. At least part of the model optimization information may be given by the above operations or by the experience of the above operations in cooperation with the business person.
In addition, the terminal can also inquire an error case according to a confidence threshold and a cross ratio threshold set by a user and display the error case to the user, so that the user can determine model optimization information based on a preset mapping relation according to the cross ratio, the confidence and the test accuracy information.
In addition, the business personnel can also give model optimization information aiming at the training image set and the like by combining with self business experience so as to optimize the training image set used for training the model.
In this embodiment, taking a scene in which the test image has the annotation positioning frame information and the object annotation information for the image in the annotation positioning frame as an example, the static indexes include: and aiming at the confidence coefficient of the recognition result of the test image and the intersection ratio of the labeling positioning frame, the intersection ratio of the labeling positioning frame represents the coincidence proportion of the recognition positioning frame relative to the labeling positioning frame.
The embodiment of the disclosure dynamically fuses multiple static indexes such as IOU, confidence, confusion matrix and other static indexes, adds a part of auditing the model by service personnel, helps the modeling personnel to locate the problems existing in the model more quickly, accelerates the iterative optimization of the model, and simultaneously enables the optimization direction of the model to be closer to the service indexes, thereby fully playing the roles of the modeling personnel and the service personnel.
Another aspect of the present disclosure provides a model optimization apparatus, which is disposed at a terminal.
FIG. 10 schematically shows a block diagram of a model optimization apparatus according to an embodiment of the present disclosure.
As shown in fig. 10, the model optimization apparatus 1000 may include: a request sending module 1010, an index threshold sending module 1020 and an accuracy rate showing module 1030.
The request sending module 1010 is configured to send a test request including a set identifier to a server, so that the server obtains a value of a static indicator of a test image in a test image set corresponding to the set identifier.
The indicator threshold sending module 1020 is configured to send, in response to a threshold input operation, an indicator threshold of at least part of the static indicators to the server, so that the server determines a test accuracy for the test image set based on a value of the static indicators of the test image and the indicator threshold.
The accuracy demonstration module 1030 is configured to respond to the test accuracy from the server side to demonstrate the test accuracy so as to determine model optimization information.
The embodiment of the disclosure provides a testing system for a deep learning target detection algorithm based on a webpage (web) front-end technology, and efficient testing of a model is realized through cooperation of modeling personnel and business personnel, linkage adjustment of technical indexes and table display of a plurality of testing result accuracy rates.
Another aspect of the present disclosure provides a model optimization apparatus, which is disposed on a server side.
FIG. 11 schematically illustrates a block diagram of a model optimization apparatus according to another embodiment of the present disclosure.
As shown in fig. 11, the apparatus 1100 includes: a request receiving module 1110, an image processing module 1120, an accuracy determining module 1130, and an accuracy sending module 1140.
The request receiving module 1110 is configured to receive a test request from a terminal, where the test request includes a set identifier of a test image set.
The image processing module 1120 is configured to process the test image set corresponding to the set identifier by using the trained model, and obtain a value of a static indicator for the test image in the test image set.
The accuracy determination module 1130 is configured to determine, in response to an index threshold of at least part of the static indexes from the terminal, a test accuracy for the test image set based on a value of the static index and the index threshold.
The accuracy sending module 1140 is used to send the test accuracy to the terminal in order to determine the model optimization information.
It should be noted that the implementation, solved technical problems, implemented functions, and achieved technical effects of each module/unit and the like in the apparatus part embodiment are respectively the same as or similar to the implementation, solved technical problems, implemented functions, and achieved technical effects of each corresponding step in the method part embodiment, and are not described in detail herein.
Any of the modules, units, or at least part of the functionality of any of them according to embodiments of the present disclosure may be implemented in one module. Any one or more of the modules and units according to the embodiments of the present disclosure may be implemented by being split into a plurality of modules. Any one or more of the modules, units according to the embodiments of the present disclosure may be implemented at least partially as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented by any other reasonable means of hardware or firmware by integrating or packaging the circuits, or in any one of three implementations of software, hardware and firmware, or in any suitable combination of any of them. Alternatively, one or more of the modules, units according to embodiments of the present disclosure may be implemented at least partly as computer program modules, which, when executed, may perform the respective functions.
For example, any of the request sending module 1010, the index threshold sending module 1020, and the accuracy rate exposing module 1030 may be combined and implemented in one module, or any one of the modules may be split into a plurality of modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of the other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the request sending module 1010, the indicator threshold sending module 1020, and the accuracy rate presenting module 1030 may be implemented at least partially as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented by hardware or firmware in any other reasonable manner of integrating or packaging a circuit, or may be implemented by any one of three implementations of software, hardware, and firmware, or any suitable combination of any of the three. Alternatively, at least one of the request sending module 1010, the metric threshold sending module 1020, and the accuracy rate exposing module 1030 may be at least partially implemented as a computer program module, which when executed, may perform a corresponding function.
FIG. 12 schematically shows a block diagram of an electronic device according to an embodiment of the disclosure. The electronic device shown in fig. 12 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 12, an electronic apparatus 1200 according to an embodiment of the present disclosure includes a processor 1201, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)1202 or a program loaded from a storage section 1208 into a Random Access Memory (RAM) 1203. The processor 1201 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or associated chipset, and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), among others. The processor 1201 may also include on-board memory for caching purposes. The processor 1201 may include a single processing unit or multiple processing units for performing the different actions of the method flows according to embodiments of the present disclosure.
In the RAM 1203, various programs and data necessary for the operation of the electronic apparatus 1200 are stored. The processor 1201, the ROM 1202, and the RAM 1203 are communicatively connected to each other by a bus 1204. The processor 1201 performs various operations of the method flow according to the embodiments of the present disclosure by executing programs in the ROM 1202 and/or the RAM 1203. Note that the programs may also be stored in one or more memories other than the ROM 1202 and the RAM 1203. The processor 1201 may also perform various operations of method flows according to embodiments of the present disclosure by executing programs stored in the one or more memories.
Electronic device 1200 may also include input/output (I/O) interface 1205, according to an embodiment of the disclosure, input/output (I/O) interface 1205 also connected to bus 1204. The electronic device 1200 may also include one or more of the following components connected to the I/O interface 1205: an input section 1206 including a keyboard, a mouse, and the like; an output portion 1207 including a display device such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 1208 including a hard disk and the like; and a communication section 1209 including a network interface card such as a LAN card, a modem, or the like. The communication section 1209 performs communication processing via a network such as the internet. A driver 1210 is also connected to the I/O interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like, is mounted on the drive 1210 as necessary, so that a computer program read out therefrom is mounted into the storage section 1208 as necessary.
According to embodiments of the present disclosure, method flows according to embodiments of the present disclosure may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable storage medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 1209, and/or installed from the removable medium 1211. The computer program, when executed by the processor 1201, performs the above-described functions defined in the system of the embodiments of the present disclosure. The systems, devices, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.
The present disclosure also provides a computer-readable storage medium, which may be contained in the apparatus/device/system described in the above embodiments; or may exist separately and not be assembled into the device/apparatus/system. The computer-readable storage medium carries one or more programs which, when executed, implement the method according to an embodiment of the disclosure.
According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, a computer-readable storage medium may include the ROM 1202 and/or the RAM 1203 and/or one or more memories other than the ROM 1202 and the RAM 1203 described above.
Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for performing the method provided by the embodiments of the present disclosure, when the computer program product is run on an electronic device, the program code being configured to cause the electronic device to implement the image model training method or the model optimization method provided by the embodiments of the present disclosure.
The computer program, when executed by the processor 1201, performs the above-described functions defined in the system/apparatus of the embodiments of the present disclosure. The systems, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.
In one embodiment, the computer program may be hosted on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted, distributed in the form of a signal on a network medium, downloaded and installed through the communication section 1209, and/or installed from the removable medium 1211. The computer program containing program code may be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.
In accordance with embodiments of the present disclosure, program code for executing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, these computer programs may be implemented using high level procedural and/or object oriented programming languages, and/or assembly/machine languages. The programming language includes, but is not limited to, programming languages such as Java, C + +, python, the "C" language, or the like. The program code may execute entirely on the user computing device, partly on the user device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
Those skilled in the art will appreciate that various combinations and/or combinations of features recited in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not expressly recited in the present disclosure. These examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described separately above, this does not mean that the measures in the embodiments cannot be used in advantageous combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be devised by those skilled in the art without departing from the scope of the present disclosure, and such alternatives and modifications are intended to be within the scope of the present disclosure.

Claims (15)

1. A method of model optimization performed by a terminal, comprising:
sending a test request including a set identifier to a server so that the server can obtain a value of a static index of a test image in a test image set corresponding to the set identifier;
responding to a threshold input operation, and sending an index threshold of at least part of static indexes to the server side so that the server side determines the test accuracy rate aiming at the test image set based on the value of the static indexes of the test image and the index threshold; and
responding to the test accuracy from the server side, and displaying the test accuracy so as to determine model optimization information.
2. The method of claim 1, further comprising: after the index threshold value of at least part of static indexes is sent to the server side,
receiving a subset of error-identified images from the server for the set of test images; and
the displaying the test accuracy in response to the test accuracy from the server side so as to determine model optimization information comprises:
and responding to the identification error image subset and the test accuracy from the server side, and displaying at least part of the identification error image subset, the index threshold value of the static index and the test accuracy so that a user can determine the model optimization information based on the identification error image subset, the index threshold value of the static index and the test accuracy.
3. The method of claim 2, further comprising:
displaying a preset mapping relation, wherein the preset mapping relation comprises an index threshold value of a static index and/or a corresponding relation between the test accuracy and model optimization information; and/or
And determining model optimization information corresponding to the error image identification subset at least based on a preset mapping relation, an index threshold value of the static index and the test accuracy.
4. The method of claim 2, wherein the test image has annotation alignment box information and object annotation information for images within the annotation alignment box; and
the static indicators include: aiming at the confidence coefficient of the recognition result of the test image and the intersection ratio of the labeling positioning frame, wherein the intersection ratio of the labeling positioning frame represents the coincidence proportion of the recognition positioning frame relative to the labeling positioning frame;
said presenting at least part of said subset of identified erroneous images comprises: for each of the presented at least partially identified false images, presenting the identified false image and at least one of: and the identification positioning frame, the marking positioning frame, the object marking information and the object identification information correspond to the identification error image.
5. The method of claim 4, wherein:
if the recognition result confidence is smaller than a first threshold and the intersection ratio is smaller than a second preset threshold, the model optimization information comprises at least one of the following: at least one of the abnormal labeling information of the training image set, the number of the training image set smaller than a preset image number threshold value, model under-fitting and the increase of the size of the input image;
if the recognition result confidence is smaller than a first threshold and the intersection ratio is larger than a second preset threshold, the model optimization information comprises at least one of the following: the number of the training image sets is smaller than a preset image number threshold, the number of rounds of model training is increased, the number of images input in each round of training is reduced, the training images are sliced, and at least one of the images of which the types are overlapped with other types exists;
if the recognition result confidence is greater than a first threshold and the intersection ratio is less than a second preset threshold, the model optimization information includes at least one of: the labeling information of the training image set is abnormal, the quality of the training image set is improved, and the size of the input image is increased; and
and if the type of at least part of the images in the error image subset is identified to belong to a rare type in the business application, the model optimization information comprises at least one of reduction or cancellation of a training image set belonging to the rare type and prompt manual processing.
6. The method of claim 5, wherein:
the static index further comprises a confusion matrix; and
if the first fraction of the false negative class relative to the negative class is greater than a third threshold and/or the second fraction of the false positive class relative to the positive class is greater than a fourth threshold, the model optimization information includes at least one of: the number of the training image sets is smaller than a preset image number threshold value, the number of rounds of model training is increased, the number of pictures input in each round of training is reduced, the pictures are sliced, and at least one of the images of which the types of the training images are overlapped with other types exists.
7. The method of any of claims 1 to 6, further comprising: after said demonstrating of said test accuracy rate,
and responding to an optimization information input operation, and sending the model optimization information to the server side.
8. The method of any of claims 1 to 6, wherein the model optimization information comprises: at least one of a type of the training image set, a number of training images of the training image set, a number of layers of the network, a model parameter, and a size of the training images.
9. A method of model optimization performed by a server, comprising:
receiving a test request from a terminal, wherein the test request comprises a set identifier of a test image set;
processing a test image set corresponding to the set identification by using the trained model to obtain a value of a static index of the test image in the test image set;
responding to an index threshold value of at least part of static indexes from the terminal, and determining the test accuracy rate aiming at the test image set based on the value of the static indexes and the index threshold value; and
and sending the test accuracy to the terminal so as to determine model optimization information.
10. The method of claim 9, further comprising: after said determining a test accuracy rate for said set of test images,
determining a subset of recognition error images corresponding to the test accuracy; and
and sending the identification error image subset to the terminal so that the terminal can display the identification error image subset.
11. The method of claim 9, wherein the test image has annotation positioning box information and object annotation information for images within the annotation positioning box; and
the static indicators include: and aiming at the confidence coefficient of the recognition result of the test image and the intersection ratio of the labeling positioning frame, wherein the intersection ratio of the labeling positioning frame represents the coincidence proportion of the recognition positioning frame relative to the labeling positioning frame.
12. A model optimization device arranged at a terminal comprises:
the request sending module is used for sending a test request comprising a set identifier to a server so that the server can obtain a value of a static index of a test image in a test image set corresponding to the set identifier;
the index threshold value sending module is used for responding to threshold value input operation and sending the index threshold value of at least part of static indexes to the server end so that the server end can determine the test accuracy rate aiming at the test image set based on the value of the static indexes of the test image and the index threshold value; and
and the accuracy display module is used for responding to the test accuracy from the server side and displaying the test accuracy so as to determine model optimization information.
13. A model optimization apparatus, disposed on a server side, the apparatus comprising:
the device comprises a request receiving module, a test processing module and a test processing module, wherein the request receiving module is used for receiving a test request from a terminal, and the test request comprises a set identifier of a test image set;
the image processing module is used for processing the test image set corresponding to the set identification by using the trained model to obtain a value of a static index of the test image in the test image set;
an accuracy determining module, configured to determine, in response to an index threshold of at least part of static indexes from the terminal, a test accuracy for the test image set based on a value of the static index and the index threshold; and
and the accuracy sending module is used for sending the test accuracy to the terminal so as to determine model optimization information.
14. An electronic device, comprising:
one or more processors;
storage means for storing executable instructions which, when executed by the processor, implement a model optimization method performed by a terminal according to any of claims 1 to 8, or implement a model optimization method performed by a server according to any of claims 9 to 11.
15. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, implement a model optimization method performed by a terminal according to any of claims 1 to 8, or implement a model optimization method performed by a server according to any of claims 9 to 11.
CN202110229068.7A 2021-03-02 2021-03-02 Model optimization method, model optimization device and electronic equipment Pending CN113011473A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110229068.7A CN113011473A (en) 2021-03-02 2021-03-02 Model optimization method, model optimization device and electronic equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110229068.7A CN113011473A (en) 2021-03-02 2021-03-02 Model optimization method, model optimization device and electronic equipment

Publications (1)

Publication Number Publication Date
CN113011473A true CN113011473A (en) 2021-06-22

Family

ID=76402176

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110229068.7A Pending CN113011473A (en) 2021-03-02 2021-03-02 Model optimization method, model optimization device and electronic equipment

Country Status (1)

Country Link
CN (1) CN113011473A (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113435305A (en) * 2021-06-23 2021-09-24 平安国际智慧城市科技股份有限公司 Precision detection method, device and equipment of target object identification algorithm and storage medium
CN114513378A (en) * 2021-12-31 2022-05-17 浙江慧居智能物联有限公司 Slicing-based local scene linkage gateway implementation method and device
CN116204670A (en) * 2023-04-27 2023-06-02 菲特（天津）检测技术有限公司 Management method and system of vehicle target detection data and electronic equipment

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113435305A (en) * 2021-06-23 2021-09-24 平安国际智慧城市科技股份有限公司 Precision detection method, device and equipment of target object identification algorithm and storage medium
CN114513378A (en) * 2021-12-31 2022-05-17 浙江慧居智能物联有限公司 Slicing-based local scene linkage gateway implementation method and device
CN114513378B (en) * 2021-12-31 2023-12-01 绿碳智能建筑(杭州)有限公司 Local scene linkage gateway realization method and device based on slicing
CN116204670A (en) * 2023-04-27 2023-06-02 菲特（天津）检测技术有限公司 Management method and system of vehicle target detection data and electronic equipment

Similar Documents

Publication Publication Date Title
EP3335131B1 (en) Systems and methods for automatic content verification
CN107766940B (en) Method and apparatus for generating a model
CN113011473A (en) Model optimization method, model optimization device and electronic equipment
US10936672B2 (en) Automatic document negotiation
CN109783227A (en) Method for allocating tasks, device, system and computer readable storage medium
CN113076104A (en) Page generation method, device, equipment and storage medium
US20160275003A1 (en) Test script evaluation system and method
CN108933695B (en) Method and apparatus for processing information
AU2018282441A1 (en) Systems and methods for determining an optimal strategy
CN113761514B (en) Cloud desktop multi-factor security authentication method and system
CN110647523A (en) Data quality analysis method and device, storage medium and electronic equipment
CN112988727A (en) Data annotation method, device, equipment, storage medium and computer program product
CN107153694B (en) Method, device, equipment and storage medium for automatically modifying question errors
CN110688827A (en) Data processing method and device, electronic equipment and storage medium
CN111311393A (en) Credit risk assessment method, device, server and storage medium
CN111859985B (en) AI customer service model test method and device, electronic equipment and storage medium
CN115203178A (en) Data quality inspection method and device, electronic equipment and storage medium
US11810022B2 (en) Contact center call volume prediction
CN114115628A (en) U shield display information acquisition method, device, equipment, medium and program product applied to U shield test
CN113052509A (en) Model evaluation method, model evaluation apparatus, electronic device, and storage medium
CN111897883A (en) Entity model construction method and device, electronic equipment and medium
CN111125501A (en) Method and apparatus for processing information
CN115484560B (en) Intelligent short message processing method and device, electronic equipment and storage medium
US20210089992A1 (en) Method for automated code reviewer recommendation
CN117874211B (en) Intelligent question-answering method, system, medium and electronic equipment based on SAAS software

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination