WO2020007287A1 - 执行机器学习过程的方法、装置、设备以及存储介质 - Google Patents
执行机器学习过程的方法、装置、设备以及存储介质 Download PDFInfo
- Publication number
- WO2020007287A1 WO2020007287A1 PCT/CN2019/094363 CN2019094363W WO2020007287A1 WO 2020007287 A1 WO2020007287 A1 WO 2020007287A1 CN 2019094363 W CN2019094363 W CN 2019094363W WO 2020007287 A1 WO2020007287 A1 WO 2020007287A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- model
- user
- data
- labeling
- training
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
- G06F40/216—Parsing using statistical methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/02—Knowledge representation; Symbolic representation
- G06N5/027—Frames
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/04—Inference or reasoning models
Definitions
- the present disclosure relates generally to the field of machine learning, and more particularly, to a method, an apparatus, a device, and a storage medium for performing a machine learning process.
- Machine learning (including deep learning) is an inevitable product of the development of artificial intelligence research to a certain stage. It is committed to improving the performance of the system itself through the use of calculations and experience.
- experience usually exists in the form of “data”.
- models can be generated from data. That is, by providing empirical data to machine learning algorithms, it can be generated based on these empirical data The model, when faced with new examples, will provide corresponding judgments, that is, prediction results.
- Exemplary embodiments of the present disclosure are to provide a method, an apparatus, a device, and a storage medium for performing a machine learning process to solve at least one of the problems described above.
- a method for performing a machine learning process including: acquiring data; acquiring annotated results of the data; and selecting a model framework that matches a user's needs and / or a user's prediction target
- the matched model is based on the selected model framework and / or model and uses the data and its labeling results to perform model training.
- the model framework is a framework for training models based on machine learning algorithms.
- a device for performing a machine learning process including: a data acquisition module for acquiring data; a labeling result acquisition module for acquiring the labeling results of the data; a selection module for Select a model framework that matches the user's needs and / or a model that matches the user's prediction goals, where the model framework is a framework for training a model based on a machine learning algorithm; and a training module is used based on the selected model framework And / or model, using the data and its labeled results for model training.
- a computing device including: a processor; and a memory, which stores executable code thereon, and when the executable code is executed by the processor, causes the processor to execute as described in the present disclosure. The method described in the first aspect.
- a non-transitory machine-readable storage medium on which executable code is stored, and when the executable code is executed by the processor of the electronic device, the processor is caused to execute as described Disclose the method mentioned in the first aspect.
- a model framework and / or model can be automatically selected for model training, so that the machine learning process can be reduced to a level that does not require the user to know algorithm knowledge.
- FIG. 1 illustrates a flowchart of a method of performing a machine learning process according to an exemplary embodiment of the present disclosure.
- FIG. 2 illustrates a flowchart of a data acquisition method according to an exemplary embodiment of the present disclosure.
- FIG. 3 illustrates a flowchart of an auxiliary labeling method according to an exemplary embodiment of the present disclosure.
- FIG. 4 is a schematic diagram illustrating a labeling interface according to an embodiment of the present disclosure.
- Figure 5 shows a schematic diagram of the interface after the model is generated.
- FIG. 6 illustrates a flowchart of a model interpretation method according to an exemplary embodiment of the present disclosure.
- FIG. 7 illustrates a model training process according to an exemplary embodiment of the present disclosure.
- FIG. 8 shows a platform architecture diagram when the present disclosure is implemented as a full-flow automatic machine learning platform.
- FIG. 9 illustrates a block diagram of an apparatus for performing a machine learning process according to an exemplary embodiment of the present disclosure.
- FIG. 10 shows a structural schematic diagram of functional modules that a data acquisition module may have.
- FIG. 11 is a schematic structural diagram of a functional module that the annotation result acquiring module may have.
- FIG. 12 shows a structural schematic diagram of functional modules that an interpretation module may have.
- FIG. 13 is a schematic structural diagram of a computing device that can be used to implement data processing of the foregoing method according to an embodiment of the present disclosure.
- FIG. 1 illustrates a flowchart of a method of performing a machine learning process according to an exemplary embodiment of the present disclosure.
- the machine learning described in this disclosure may include neural network-based deep learning in addition to algorithms such as logistic regression, support vector machines, GBDT (Gradient Boosted Decision Tree), and Naive Bayes algorithm.
- the method is performed by at least one computing device.
- step S110 data is acquired.
- the acquired data may be data uploaded by the user, or data collected through other methods, such as data collected through web crawling, database retrieval, and sending data collection tasks to the collector.
- the users mentioned in the present disclosure refer to users who desire to perform model training, and the collectors refer to persons who can perform data collection tasks to collect corresponding data.
- step S120 a labeling result of the data is acquired.
- the data acquired based on step S110 may be data with annotated results, or data without annotated results.
- This disclosure does not place any restrictions on the way to obtain the labeling results, that is, the data can have labeling results in any way, and the labeling results can be either objective and true labeling conclusions, or subjective results of artificial labeling.
- the labeling result of the data can be directly obtained; in the case that the data obtained based on step S110 has no labeling result, or some of the acquired data has no labeling result, the Annotate the data to get the annotation result of the data.
- a labeling task corresponding to the prediction target can be issued, and the labeler can perform manual labeling to obtain the labeling result of the data.
- the prediction target refers to a prediction function that the user expects to be trained by the model. For example, when the user expects to train a model for identifying a cat in an image, the prediction target is "recognize a cat in an image".
- the labeler refers to a person who can perform manual labeling.
- step S130 a model frame matching the user's needs and / or a model matching the user's prediction target is selected, and based on the selected model frame and / or model, the data and its labeling results are used for model training.
- the model framework described in this disclosure is a framework for training models based on machine learning algorithms.
- one or more task types may be set in advance, and one or more model frames may be set for each task type.
- one or more corresponding model frameworks may be set in advance for the characteristics of each task type, so that the machine learning algorithm corresponding to the set model framework is helpful to solve the corresponding task type. Therefore, selecting a model frame that matches the needs of the user may refer to selecting a model frame from the model frames corresponding to the task types that match the needs of the user.
- the description of the implementation process and task types of the selected model framework will be described in detail below.
- the model mentioned here may be a previously trained model, and the model may be a model trained by using the present disclosure, or a model trained by other methods.
- the model may be a model obtained by performing model training using training samples based on a corresponding model framework.
- selecting a model that matches the user's prediction target may refer to selecting a model that matches the user's prediction target from the previously trained models.
- the prediction target refers to a prediction function implemented by a model that a user desires to train. For example, when it is desired to train a model for identifying a cat in an image, the prediction target is "recognize a cat in an image".
- a model that matches the user's prediction target refers to a model that can achieve the same or similar functions as the prediction target. For example, when the user's prediction target is "recognize a cat in an image", the previously trained can be used for The model that recognizes cats in the image is used as a model that matches the user's prediction target.
- the model that was previously trained to recognize other types of animals in the image can be used as Models that match users' prediction goals.
- the model mentioned in step S130 may be obtained by performing model training based on a corresponding model framework.
- a matching from a previously trained model is selected to match a user's prediction target.
- Model or you can first select a model trained based on the model frame corresponding to the task type that matches the user's needs from the previously trained model, and then further search for the user-friendly prediction from the selected model
- the model of the target Therefore, the model obtained by the search can be more adapted to the needs of users.
- the selected model framework When using data and its labeling results for training, you can use the data and its labeling results for model training based on the selected model framework; you can also use the selected model for model training. For example, you can use the data and its labeling results to update The selected model; you can also use the data and labeling results for model training according to the model frame and the overall selection of the model. If you can preferably use the selected model for model training, if you cannot select a suitable model, then Model training based on the selected model framework. Specifically, in the case where a model matching the user's prediction target is selected, the selected model may be updated by using the data and its labeling result, and when the model matching the user's prediction target is not selected.
- the data and its labeling results are used for model training.
- the selected model when performing model training based on the selected model, the selected model can also be adjusted, for example, the network structure of the model can be fine-tuned, and model training can be performed based on the fine-tuned model.
- the acquired data and its annotation results can be stored in a user database corresponding to the user, and the trained model can be saved.
- the trained model that is, the user model described below
- the external use right of the user database may be related to the settings of the user.
- a user-oriented application programming interface may be generated in response to a user's model application request, so that the user can obtain predictions provided by the model through the application programming interface. service.
- API application programming interface
- the resources used by the model to provide prediction services can also be dynamically adjusted according to the amount of prediction requests initiated by the user through the application programming interface. For example, when the amount of predicted requests is large, users can be allocated more resources such as CPU and memory, and when the amount of predicted requests is small, users can be allocated fewer resources such as CPU and memory. As a result, platform resources can be saved while meeting user needs.
- the present disclosure can automatically fit the relationship between the data x and the learning object y (that is, the labeling result) by automatically selecting a suitable model framework and / or model based on the user's needs and / or prediction goals, so as to A model that meets user needs is obtained, and an online prediction service can also be provided by using the obtained model.
- the user can upload the input image x through various methods such as http request and grpc request, and the service returns the prediction y for x.
- the machine learning process can be reduced to a level that does not require the user to know the knowledge of the algorithm, and the online service capability of the model can be provided to the user quickly based on the present disclosure.
- the method shown in FIG. 1 can be implemented as a machine learning platform, which can help users implement a machine learning model (such as a deep learning model) process more automatically.
- the platform can also collect one or more of many functions such as data acquisition, data annotation, automatic model launch (that is, using the model to provide online prediction services), model update, and model interpretation to better serve users.
- the data uploaded by the user may be obtained.
- the data uploaded by the user may be data with annotations or data without annotations.
- data can also be collected according to user needs. For example, data can be collected when the user has no data or the uploaded data is insufficient.
- FIG. 2 illustrates a flowchart of a data acquisition method according to an exemplary embodiment of the present disclosure.
- step S111 a data collection requirement of a user is obtained.
- the user's data collection requirements may be obtained when the user does not upload data for model training, or if the amount of data uploaded by the user for model training is insufficient (eg, less than a first predetermined threshold).
- Data collection requirements are the description of the data that users expect to collect.
- Data collection requirements can be text or voice.
- the user's data collection requirements can be the text "Need to collect pictures containing various fruits", or it can correspond to the text's voice.
- the data collection requirements here can also be obtained by analyzing the prediction targets of the model. For example, when the user specifies the prediction target as a cat and dog classification model, the data collection needs can be determined to include cats or dogs by analyzing this prediction target. image.
- step S112 the data collection requirements are analyzed to determine keywords that are suitable for the collected data.
- the keywords of related data can be further obtained by analyzing the meaning or components of the requirements.
- the data collection requirement may be directly parsed by means of semantic analysis (for example, NLP technology) to determine keywords that are suitable for the collected data.
- semantic analysis for example, NLP technology
- Keywords can be considered as a general description of the data in one or more characteristic dimensions. Taking data as pictures as an example, keywords can be tags of pictures. Tags such as definition, picture description, picture source, etc. can be used as keywords.
- a semantic analysis may be performed on the data collection requirements to determine the data objects that the user desires to obtain.
- the data objects may refer to the objects that the user desires to obtain, such as the objects (or items) included in the data.
- the data collection requirement of “need to collect pictures containing various fruits” it can be determined that the data object that the user desires to obtain is “fruit”.
- the data object may also be applied to a knowledge map to obtain a derived object.
- the derived object may be a horizontally derived object of the same or similar type as the data object, or a downwardly derived object belonging to a subclass of the data object.
- you can get multiple downward-derived objects such as “apple”, “banana”, “orange”, and “cherry”; for example, the data object includes “apple” , "Banana”, “orange”, you can also use the knowledge map to get “pear", “peach”, “pineapple” and other horizontally derived objects. Therefore, keywords mentioned in this disclosure may indicate data objects and / or derived objects.
- step S113 data with keywords is collected.
- Data with keywords can be retrieved from a database.
- the data in the database is data with known keywords.
- the database mentioned here may include a public database and / or a user database, where the public database refers to A database that is open to the public.
- the user database is a private database maintained by other users. The open permissions of the user database are related to the settings of the corresponding user.
- data with keywords can be searched from the network, for example, data with keywords can be obtained by web crawling.
- a collection task for collecting data with keywords can be generated, and the collection task is distributed to one or more collectors, so that the collector can collect data with keywords.
- a user database corresponding to the user can also be maintained, and the collected data can be stored in the user database.
- a labeling task corresponding to the predicted target may be issued according to the prediction target of the model training, and the labeling result of the collected data may be obtained, and the labeling result of the data and the data may be stored in the user database in association with the data.
- the external use right of the user database may be determined according to the user's right setting.
- the data in the user database can be used by other users. For example, when retrieving data with keywords from a database, you can search from user databases of other users that are open to the public.
- the annotation task corresponding to the prediction target may be issued according to the prediction target of model training To get the annotation results of the data.
- the labeling task may be issued to one or more labelers who can perform manual labeling. The labeler will perform manual labeling on the data to be labeled, and then the manual labeling results can be arranged. For example, the labeling results can be stored in association with the data.
- the present disclosure also proposes an auxiliary labeling scheme.
- FIG. 3 illustrates a flowchart of an auxiliary labeling method according to an exemplary embodiment of the present disclosure.
- step S121 an object to be annotated is displayed to an annotator.
- the object to be annotated is mainly displayed to the annotator in a visual form.
- the object to be annotated may include original data to be annotated.
- the object to be annotated may be an image including an object (or item) to be annotated, or a piece of text including a word to be annotated.
- the labeling form, and the labeling content are related to the specific labeling requirements of the labeling task.
- step S122 auxiliary prompting information for prompting the labeling conclusion of the object to be labeled is acquired.
- step S122 may be performed before step S121, or may be performed at the same time as or after step S121.
- the order of execution of step S122 and step S121 is not limited in this disclosure.
- the labeling conclusion can be regarded as the true label of the object to be labeled. It should be noted that what is described here is that the obtained auxiliary prompt information is used to prompt the label conclusion of the object to be labeled, that is, the auxiliary prompt information itself is not the same as the label conclusion, but is only a preliminary label conclusion. The function is to be able to prompt the true labeling conclusion of the object to be labeled to a certain extent. Therefore, in practical applications, the obtained auxiliary prompt information is likely to deviate from the true labeling conclusion of the object to be labeled, or even the opposite.
- step S122 is to obtain auxiliary prompt information that can approach the true labeling conclusion of the object to be annotated, due to the different means (to be described below) of acquiring the auxiliary prompt information and the accuracy is different, the final acquired auxiliary information
- the prompt information may also indicate the wrong conclusion.
- step S123 the annotator is provided with auxiliary prompting information, so that the annotator performs artificial labeling in the object to be annotated based on the auxiliary prompting information.
- the auxiliary prompting information is mainly provided to the labeler in a way that can be understood by the person, for example, the auxiliary prompting information can be displayed to the labeler visually.
- the content and display form of the auxiliary prompt information are also different.
- labeling problems can be divided into classification problems and identification problems. Specifically, it can be subdivided into various labeling problems such as image classification, object frame, semantic segmentation, image annotation, face highlighting, and video tracking.
- image classification may refer to selecting an image or a label to which an object belongs in the image according to the image content, such as a scene label, an object type label, an object attribute label, a character gender label, a person age label, and the like;
- an object picture frame may refer to According to the labeling requirements, frame the target object in the image, such as vehicles, license plates, pedestrians, roads, buildings, ships, text, human body parts, etc. in the image and label them accordingly.
- the image annotation can be a pointer to the specific situation and labeling needs in the image, and generate a Chinese label sentence for each image; Face and labeling requirements in the image, locate and mark key positions of the face, such as face contour, eyebrows, eyes, lips; video tracking can refer to extracting key frames from the target video at a specified frequency, and then for each frame Label the frame, keep the same label and serial number of the same target in each frame
- the auxiliary prompting information may be a preliminary labeling conclusion (such as a label) of the object to be labeled (that is, the image to be labeled); taking the labeling problem as an object picture frame as an example, the auxiliary prompting information may include a frame selection result And labeling information; taking the labeling problem as a semantic segmentation as an example, the auxiliary prompting information can be the result of the outline selection of the target object in the object to be labeled; taking the labeling problem as a face dot as an example, the auxiliary prompting information can be the object to be labeled The results of the dots in multiple key positions on the human face; taking the video tracking as an example, the auxiliary prompt information can be the result of the frame of the target object in each frame of the picture.
- the specific content and display form of the auxiliary prompt information are also different, which will not be described in detail in this disclosure.
- the auxiliary prompt information is used to prompt the labeling conclusion of the object to be labeled. Therefore, the annotator can perform artificial annotation in the object to be annotated based on the auxiliary prompt information.
- the auxiliary prompt information provided to the annotator can be regarded as the default conclusion of the system. The annotator can determine whether the auxiliary prompt information is consistent with the result that he wants to annotate based on his own knowledge. The auxiliary prompt information completes the labeling of the object to be labeled, thereby greatly improving the labeling efficiency.
- the annotator can also adjust the auxiliary prompt information, such as adjusting the frame selection range, adjusting the content description, and so on. Furthermore, if the annotator thinks that the auxiliary prompt information is very different from the result that it wants to annotate, the annotator can also discard the auxiliary prompt information and perform an artificial annotation completely different from the auxiliary prompt information in the object to be annotated.
- the annotator can be provided with auxiliary prompting information while showing the object to be annotated. That is, the to-be-annotated object displayed to the annotator may include auxiliary prompt information.
- the labeling task may be "pig face labeling", and the specific labeling requirement may be frame selection of the pig face in the image to be labeled. Therefore, the auxiliary prompt information may be the preliminary result of frame selection of the pig face in the image.
- the object to be labeled is an image
- a wire frame in the image is auxiliary prompt information, that is, a preliminary result of selecting a pig face frame.
- the annotator can accept the frame selection result, and can also readjust the frame selection result to re-determine the frame selection range. For example, the annotator can reduce the scope of the wire frame and add a new line frame to frame the two pigs. Face, and seldom frame the part outside the pig's face.
- the auxiliary prompt information is only used to prompt the labeling conclusion of the object to be labeled, and it is not necessarily accurate. Therefore, the annotator can accept the auxiliary prompt information based on his / her own cognition, can also adjust the auxiliary prompt information, and can also discard the auxiliary prompt information, and make a label operation completely different from the auxiliary prompt information. In other words, the auxiliary prompt information only serves as a prompt, and the final labeling result is still in the hands of the labeler.
- the difference between the artificial annotation result and the auxiliary prompting information may be presented to the annotator.
- an artificial annotation result of the object to be annotated may be obtained in response to an artificial annotation performed by the annotator, and a difference between the artificial annotation result and the auxiliary prompt information may be provided to the annotator.
- the difference may be promptly notified in real time in response to the artificial annotation performed by the annotator, or may only be used if the difference is greater than a certain threshold (for the convenience of distinguishing, it may be referred to herein as the "third predetermined threshold").
- This kind of difference is prompted to the annotator, so as to prompt the annotator to pay attention to such an excessively different annotation result, so that the mislabeling operation caused by the annotator's intention can be reduced to a certain extent.
- the auxiliary prompt information can be obtained in the following two ways.
- the auxiliary prompt information may be obtained based on a labeling conclusion of an object that is the same as or similar to the object to be labeled.
- the labeling conclusion of an object that is the same as or similar to the object to be labeled may be directly used as auxiliary prompting information of the object to be labeled.
- the labeling conclusion of the same or similar object as the object to be labeled may be the result of artificial labeling, the prediction result of the model, or the true conclusion.
- an object having the same conclusion as the object to be annotated may be obtained in various ways.
- an object having the same conclusion as the object to be annotated may be selected from a database storing various objects.
- the database may be a database maintained by the platform, and the object stored in the database may preferably be an object whose conclusion is known.
- the source of the objects in the database is not limited in this disclosure.
- the objects may be manually annotated objects, or they may be objects with real annotated conclusions (for example, public data, etc.).
- an object that is the same as or similar to the object to be annotated with the labeling conclusion may also be obtained through the network.
- an object that is the same or similar to the object to be annotated with a true labeling conclusion may be obtained by a web crawler.
- labeled objects that belong to the same labeling task as the object to be labeled can also be regarded as the same or similar objects as the object to be labeled.
- the labeled object may be an object that has been labeled and has passed the labeling result verification.
- the prediction result of the machine learning model for the object to be labeled can be obtained as auxiliary prompt information.
- the machine learning model is trained to predict the labeling conclusion of the object to be labeled. For example, it can be trained for the same labeling task. Forecasting model. For example, when a specific customer (for example, the user who wants to train the model mentioned above) publishes an image labeling task on the platform (for example, the customer uploads the image data of his own pig farm and expects the labeler to label ), A unified machine learning model can be trained for the customer (ie, the customer's labeling task) without having to consider the labeling differences of different labelers. The machine learning model can be trained to predict the labeling conclusion of the object to be labeled, and the predicted labeling conclusion can be used as auxiliary prompt information.
- a specific customer for example, the user who wants to train the model mentioned above
- a unified machine learning model can be trained for the customer (ie, the customer's labeling task) without having to consider the labeling differences
- the machine learning model may be trained based on at least a part of the labeled objects that belong to the same labeling task as the object to be labeled and their artificial labeling results. That is to say, at least a part of the labeled objects and their artificial labeling results in the same labeling task can be used as training samples for model training.
- the training samples herein may preferably be generated from labeled objects that have been artificially verified by the test results, that is, the labeled objects and their artificial labeling results may be used as training samples for the model. Therefore, the training process of the machine learning model can be performed after the labeling task is released for a period of time so that an appropriate number of training samples are accumulated.
- the machine learning model may also be trained based on non-labeled objects that are the same or similar to the object to be labeled and their true labeling conclusions.
- non-labeled objects that are the same as or similar to the object to be labeled and their true labeling conclusions can be used as training samples for model training.
- the non-labeled object may be an object whose true labeling conclusion is known, for example, it may be known data collected in a database, previously stored data, or data from a network source, and so on. Therefore, the cold start problem can also be solved, so that the training process of the machine learning model can be performed in advance.
- the machine learning model can be trained in advance for the tagging task before issuing the tagging task to the tagger and showing the objects to be labeled.
- the machine learning model is trained to predict the labeling conclusion of the object to be labeled, and the predicted labeling conclusion can be used as auxiliary prompt information. Therefore, the higher the accuracy of the machine learning model's prediction, the closer the auxiliary prompt information provided to the annotator will be closer to the true labeling conclusion, the less labor the tagger will need to perform the artificial annotation based on the auxiliary prompt information, and the cost of manual annotation will be lower. The lower.
- the present disclosure proposes that the machine learning model can be updated in accordance with the artificial annotation results of the annotated object for the object to be annotated to improve the accuracy of the prediction result of the machine learning model.
- the artificial annotation result of the object to be annotated may be obtained in response to the artificial annotation performed by the annotator, and the machine learning model may be updated based on the to-be-annotated object and the artificial annotation result.
- the artificial labeling results are obtained based on the labeler's own cognition, they are not necessarily accurate. Therefore, preferably, the artificial labeling result of the object to be labeled can be reviewed, and the machine-learning model can be retrained or incrementally trained using the to-be-labeled object and the artificial labeling result that have passed the review.
- the features of the objects to be labeled that have passed the review can be used as the features of the training samples
- the results of the artificial labeling can be used as the marks of the training samples, constructing the training samples, retraining, or incrementally training the machine learning model.
- the process of retraining or incremental training of the model is a mature technology in the art, and is not repeated here.
- the machine learning model may be updated based on the to-be-annotated object and the artificially-annotated result whose difference between the artificially-annotated result and the auxiliary prompt information is greater than a third predetermined threshold.
- the artificial labeling result here may be a result of auditing, that is, the artificial labeling result and the object to be labeled that have passed the review and have a difference greater than a third predetermined threshold may be used as training samples to update the machine learning model.
- a feedback labeling result of the object to be labeled may be additionally obtained to form a training sample for updating the machine learning model.
- an additional feedback mechanism can be established on the labeling platform, which can collect feedback labeling results about the object to be labeled (for example, feedback labeling results obtained by other subjects by correcting the labeler's artificial labeling results), and then based on An object to be annotated with feedback annotation results to update a machine learning model.
- the machine learning model can be continuously updated based on the annotation data generated or collected by the platform, thereby improving the accuracy of the auxiliary prompt information.
- the annotator's annotation quality can be evaluated according to the difference between the annotator and one or more other annotators on the artificial annotation result of the same object to be annotated. For example, after sending the same object to be labeled under the same labeling task to the labeler A, it can also be sent to multiple other labelers, such as labeler B, labeler C, and labeler D. When delivering to multiple other taggers, it is preferable to select a tagger with a higher tagging quality evaluation for delivery. Then, the annotation quality of the annotator A can be evaluated based on the difference between the annotator A and the multiple annotators on the artificial annotation results of the same object to be annotated. When the differences in the labeling results are large, the labeling quality of the labeler A is considered to be poor.
- the annotator's annotation quality can also be evaluated according to the difference between the annotator's artificial annotation result and the true annotation conclusion for the object to be annotated with the authentic annotation conclusion. For example, an object with a known true labeling conclusion can be randomly sent as an object to be labeled, and the labeler can be manually labeled, and then the artificial labeling result can be compared with the true labeling conclusion. If the difference is large, then It is considered that the labeler's labeling quality is poor. If the artificial labeling result is consistent with or substantially the same as the true labeling conclusion, the labeler's labeling quality is considered high.
- the labeling quality of the labeler can also be evaluated based on the difference between the artificial labeling result and the auxiliary prompting information.
- the labeler's label quality can be evaluated based on the difference between the artificial labeling result and the auxiliary prompt information.
- the difference between the annotator's artificial annotation results and the auxiliary prompt information is large, it can be considered that the annotator's annotation quality is poor. Among them, it is possible to focus on monitoring and evaluating the labeling quality of labelers who continuously receive auxiliary prompting information.
- one of the above evaluation methods may be selected to evaluate the labeling quality of the annotator, or the above multiple evaluation methods may be used to evaluate the labeling quality of the labeler, which is not limited in this disclosure.
- the labeling level of the labeler can be adjusted, for example, a corresponding credit score can be given to the labeler according to the labeler's labeling quality. It is also possible to adjust the tagging reward or punishment of the tagger to motivate the tagger to improve the quality of the tagging.
- different labeling tasks can be sent to the labeler. For example, you can send high-pay tagging tasks to taggers with higher tag quality, and increase the number of tasks for taggers with higher tag quality; accordingly, you can send to taggers with lower tag quality. Low-pay tagging tasks, and for taggers with low tagging quality, the number of tasks can be reduced.
- one or more model frameworks can be set in advance for the characteristics of each task type, so that the machine learning algorithm corresponding to the set model framework can help solve the corresponding task type.
- a task type may be set according to a type of a problem to be solved, and different task types correspond to different problem classifications.
- the task types can be divided into image classification tasks, object recognition tasks, text recognition tasks, image segmentation tasks, and feature point detection tasks.
- Image classification refers to different image categories based on the semantic information of the image.
- Image classification is an important basic problem in computer vision. Different image categories are distinguished according to the semantic information of the images, and the images are labeled with different categories.
- Image classification is the basis for other high-level visual tasks such as image detection, entity segmentation, and object tracking. It is widely used in many fields, including face recognition in the security field, intelligent video analysis, and traffic scene recognition in the transportation field.
- Object recognition is to target the image content first and then classify the object.
- Object recognition refers to the process of classifying and marking different objects that may exist in the frame based on the semantic information of the image after detection and positioning. Considering that real-life picture data usually describes a scene where multiple objects coexist, it is often difficult to deal with separate image classification effectively. At this time, the object recognition uses the idea of divide and conquer, locate first and then classify, which can greatly improve the accuracy of the recognition results. It has a wide range of applications in aerospace, medicine, communications, industrial automation, robotics, and military fields.
- Character recognition is to perform text positioning on the text of a picture before extracting text.
- Text recognition intelligently recognizes text content on pictures as computer-editable text. According to the actual use of the scene can be divided into print text recognition and handwritten text recognition.
- the former has relatively high recognition accuracy due to its uniform text standards and fixed styles; the latter has relatively high recognition costs due to its own openness and freedom.
- the text-recognition technology using deep learning can effectively replace manual information entry with its end-to-end modeling capabilities, and has been significantly promoted in the financial and insurance industries where bills and documents are identified with frequent needs.
- Image segmentation is the division of subregions of picture content based on visual characteristics.
- Image segmentation refers to the process of subdividing a digital image into multiple image sub-domains (sets of pixels). The purpose of image segmentation is to simplify or change the representation of the image, making the image easier to understand and analyze.
- Image segmentation is commonly used to locate objects and boundaries (lines, curves, etc.) in an image. More precisely, image segmentation is the process of labeling each pixel in an image. This process allows pixels with the same label to have certain common visual characteristics, such as color, brightness, texture, and so on. Object tracking and positioning in satellite images, tumor positioning in medical images, and volume measurement all require the use of image segmentation techniques.
- Feature point detection is the extraction of key feature points with significant visual characteristics (such as grayscale, etc.) in a picture.
- the image feature point refers to a point where the gray value of the image changes drastically or a point with a large curvature on the edge of the image (that is, the intersection of the two edges).
- the image feature points can reflect the essential characteristics of the image, can identify the target object in the image, and can complete the image matching by matching the feature points. Among them, color and texture as the global representation of the image can assist the understanding of the image, but are easily affected by the environment. In contrast, local feature points such as spots and corners often correspond to light and dark structures such as lines and edges in the image, and are less affected by the environment. They can be effectively applied to application scenarios such as image matching and retrieval.
- a model framework When selecting a model framework, first determine the task type that matches the user's needs, and then select the model framework from the model framework that corresponds to the task type that matches the user's needs.
- a user can characterize his or her needs by defining a task form, and can select a task type that matches a user-defined task from a variety of preset task types as a task type that matches the user's needs.
- the user can also be provided with introduction information of multiple task types, so that the user can select a suitable task type according to his or her own needs.
- a model frame in response to the user's task type selection operation, can be selected from the model frame corresponding to the task type selected by the user, or a task type matching the user-defined task can be selected from one or more task types. , And select a model frame from the model frame corresponding to the selected task type.
- the model frame when selecting a model frame from a model frame corresponding to a task type that matches a user's needs, the model frame may be randomly selected or may be selected by the user.
- the optimal hyperparameter combination of each model frame can also be obtained by means of hyperparameter optimization, and the performance is selected The best model framework and its optimal hyperparameter combinations.
- the set of hyperparameters with the best test results (such as can be evaluated according to test indicators such as accuracy and loss) can be used as the optimal hyperparameters in the model framework. combination.
- the model framework is a framework for training models based on machine learning algorithms. Based on the selected model framework, training samples can be used for model training. For example, model training can be performed using training samples based on the selected model framework and its optimal hyperparameter combination. Among them, when the optimal hyperparameter combination of the selected model framework is not determined, grid search (random search), random search (Bayesian search), Bayesian optimization and other algorithms can be used to determine the selected model framework.
- grid search random search
- random search random search
- Bayesian search Bayesian optimization and other algorithms
- model frameworks have different amounts of parameters, computational time and accuracy that can be achieved in theory
- users can also define usage scenarios of the model, such as terminal usage scenarios (fast computation speed, accuracy rate) Performance can be relatively discounted), cloud usage scenarios (computing speed can be slower, accuracy performance is good) and other scenarios.
- usage scenarios of the model such as terminal usage scenarios (fast computation speed, accuracy rate) Performance can be relatively discounted), cloud usage scenarios (computing speed can be slower, accuracy performance is good) and other scenarios.
- a model framework matching the model usage scenario can be selected from the model framework corresponding to the task type matching the user's needs.
- Figure 5 shows a schematic diagram of the interface after the model is generated.
- the user can be shown basic information about the model, such as data source, task type, task status, and output model.
- the user can also show the parameter configuration information used in the training process.
- the parameter configuration information may include, but is not limited to, data pre-processing parameters, algorithm parameters, and resource parameters.
- the data preprocessing parameters mainly include parameters of one or more preprocessing operations performed on the data. Taking image data as an example, the data preprocessing parameters may include random cropping, scaling, flipping left and right, flipping up and down, rotation, and super pixels.
- Algorithm parameters may be a combination of hyperparameters of a model framework, which may be an optimal combination of hyperparameters determined by means of hyperparameter optimization.
- the resource parameters may include physical resource parameters during model training such as CPU and memory or when the model is used.
- the disclosure can also explain the impact of different input parts on the output, so that after users use the machine learning model to obtain the output for the input, the user can also know the impact of different input parts on the output, which can to some extent Understanding the model is mainly based on which parts of the input make predictions, that is, the output, which can further enhance the credibility of the output of the machine learning model at the user level to a certain extent.
- FIG. 6 illustrates a flowchart of a model interpretation method according to an exemplary embodiment of the present disclosure.
- the method shown in FIG. 6 is mainly used to explain the output (also referred to as “original output”) of the model for the input (for ease of differentiation, also called “original input”). Specifically, It can be used to explain the effect of different input parts on the output.
- the model mentioned here may be a machine learning model, for example, it may include a deep learning model based on a neural network.
- the following mainly uses the model as the image model, the input as the image, and the output as the prediction result of the image as an example.
- the model interpretation method of the present disclosure can also be applied to a model that predicts input in other formats (such as text input). .
- step S410 the input is divided into a plurality of input sections.
- the image can be divided into multiple regions of the same shape or size to obtain multiple input parts.
- an image can be divided into a N ⁇ M grid.
- the image may be divided into a plurality of input parts according to the similarity degree of the image features.
- the image features may refer to features such as color, texture, and brightness, and pixels in the same input part have the same or similar image features.
- the pixels in the same input section may have the same or similar image features and be in adjacent positions. That is to say, when dividing the input part according to the image features, the position factor can also be considered, and pixels with the same or similar image features and adjacent positions are divided into the same input part.
- step S420 for each input part, while keeping other input parts unchanged, only the input part is subjected to transformation processing to obtain a new input.
- each input part is replaced by an input part (which may be referred to as a "perturbation input") after the transformation process.
- the input part can be transformed in various ways to obtain the disturbance input (also referred to as "noise disturbance") of the input part.
- the input part may be randomly transformed within a predetermined transformation range to obtain a disturbing part for replacing the input part.
- the value of each pixel in the input part can be randomly transformed within the value range of the pixels.
- the pixel value range refers to the range of pixel values.
- the value range is related to bit per pixel (BPP). For example, for 8bpp, the number of pixel values is 2 8 , and the value range can be 0-255. For 16bpp, the number of pixels is 2 16 , and the value can be in the range of 0-65535.
- step S430 each new input is re-input to the model to obtain a new output obtained by the model for the new input.
- step S440 according to the difference between the new output and the output (that is, the original output), the influence of different input parts on the output is determined.
- Each new input can be regarded as an input in which only one input part of the original input is subjected to transformation processing, that is, an input in which only one input part is subjected to perturbation processing. If the new output is different from the original output, it can be considered that the input part that has been transformed in the new input has a certain effect on the output. On the contrary, if the new output is the same as the original output, the input part that has been transformed in the new input can be considered The output is less important and has no effect.
- the new output is the same as the output, it can be determined that the input part that has undergone transformation processing in the new input corresponding to the new output has no effect on the output; and / or when the new output is different from the output, it can be determined that The input part that has undergone transformation processing in the new input corresponding to the new output has an effect on the output.
- step S450 the influence of different input parts on the output is notified to the user in an understandable form.
- the user can be informed of the impact of different input parts on the output in a variety of ways that can be understood by the user. For example, you can inform the user of the impact of different input parts on the output in the form of text (such as a list), or you can mark the impact of different input parts on the output in the input. The effect of different input sections in the. Taking the input as an image as an example, a heat map of the image can be generated according to the influence of the input portion on the output, where the degree of prominence of the input portion in the heat map is directly proportional to its impact on the output.
- the user can also know the impact of different input parts on the output, so that they can understand to some extent what predictions the model mainly makes based on the input, that is, the output, which can further enhance The credibility of the output of a machine learning model at the user level.
- a predetermined number of transformation processes may be performed for each input part, and then a predetermined number of new inputs obtained after each input part is subjected to a predetermined number of transformation processes may be counted and respectively obtained after being input into a machine learning model.
- the new output is the same number of times as the output.
- the influence of the input part on the output is inversely proportional to the number of times obtained by the statistics, that is, the more times the new output is the same as the output, the smaller the influence of the input part on the output (also can be understood as the importance).
- a predetermined number of transformation processes may be performed for each input part to obtain a plurality of new inputs, and a confidence level of each new output may be obtained.
- the confidence level refers to the confidence level of the machine learning model for the input, that is, the probability value and output weight of the machine learning model for the input.
- the difference between a predetermined number of new outputs and the original output corresponding to each input part can be calculated under the condition of considering the confidence, and the influence of different input parts on the output can be determined according to the difference.
- the outputs representing the two categories can be taken as +1 and -1, respectively, and the product of the output and the confidence (ie, the probability value) can be used as the output result of the comparison.
- the product corresponding to each output can be accumulated, so as to obtain the output result representing the entirety of the new output based on the accumulated sum or its average value, and compare the output result with the original output as the two difference. Therefore, for each input portion, the influence of different input portions on the output can be determined according to the difference between the corresponding predetermined number of new outputs and the original output as a whole.
- the impact of the input part on the output is proportional to the size of the difference, that is, the larger the difference, the greater the impact.
- the model After the model goes online to provide users with prediction services, they can also receive user feedback on the prediction service and update the model based on the feedback information.
- the feedback information may include updated annotations that correct the predicted annotations provided by the prediction service, may form a new training sample based on the data corresponding to the predicted annotations and their updated annotations, and use the new training samples.
- the update annotation may be made by the user.
- the feedback information may further include only rejection information that rejects the prediction annotation provided by the prediction service.
- the annotation result of the data corresponding to the rejected prediction annotation may be obtained again, based on The data and its re-acquired labeling results form a new training sample, and use the new training sample to update the model.
- the user when using the model's prediction service, the user can accept the model's prediction or not accept the model's prediction.
- the user can use the rejection information as feedback information for feedback, and can also perform prediction annotation.
- the corrected update annotation is used as feedback information, and the model can be updated according to the user's feedback information, so that the model can be closer and closer to the user's expectations.
- FIG. 7 illustrates a model training process according to an exemplary embodiment of the present disclosure.
- step S510 may be first performed to determine whether the data uploaded by the user is sufficient.
- a data upload interface may be provided for the user, and the user may upload data through the interface.
- the user's data may be considered insufficient.
- step S530 may be executed to initiate data collection.
- the process of data collection can be seen in Figure 2 above, which is not repeated here.
- step S540 may be executed to initiate data annotation.
- the data labeling process can be performed manually. For example, according to the prediction target of the model training, a labeling task corresponding to the prediction target can be issued, manually labeled, and then the labeling result of the collected data can be obtained, and the labeling result of the data can be associated with the data. Store to user database.
- step S520 may be performed to further determine whether the labeled data in the data uploaded by the user is sufficient, and if it is sufficient, the data may be directly stored in the user database. If it is not enough, step S540 may be executed to initiate data annotation.
- the labeling process can be referred to the description above, and will not be repeated here.
- the data collected for model training can be automatically realized, and optionally, it can be further automatically Finish labeling the data.
- the data platform can maintain public databases and user databases.
- the data in the public database can be completely open data.
- the labeled data and its labeled results can be stored in the user database in association.
- the external use right of the user database is related to the user's settings.
- the user database is set to be open to the public, the data in the user database can be used by other users. Therefore, when step S530 is performed, when data collection is initiated, data can be collected not only from a public database, but also from user databases of other users that are open to the public.
- step S550 may be performed to perform model training.
- a model framework that matches the user's needs can be selected from the model framework library, and model training can be performed based on the selected model framework.
- the model framework library can include multiple tasks that are specific to a specific task.
- the model framework corresponding to the type.
- a previously trained model that matches the user's prediction target can also be selected from the model library, and the selected model can be updated by using the data and its labeling results to implement model training.
- the models in the model library may include models based on public data and user models.
- the model based on public data may be a model trained based on public data
- the user model may be a model trained based on user data, which may be a model trained using the present disclosure.
- the use rights of the model based on public data can be open to all users, and the external use rights of the user model can be related to the user's settings. When selecting a model from the model library, you can only choose from the models whose permissions are open to the outside world.
- the generated model can be saved as a user model, wherein the external use rights of the model are related to the user's settings.
- a user-oriented application programming interface can be generated so that the user can obtain the prediction service provided by the model through the application programming interface.
- a user when using a prediction service, a user may return a prediction that accepts a model or a prediction that does not accept a model.
- the prediction annotations and data of the predictions accepted by the user can be saved to the user database, and for the data corresponding to the predictions that the user does not accept, the data annotation can be re-initiated and re-annotated to the model to focus on learning, so the model will be Close to user expectations.
- FIG. 8 shows a platform architecture diagram when the present disclosure is implemented as a full-flow automatic learning platform.
- the present disclosure can be implemented as a full-process automatic learning platform.
- the service platform can be composed of a data acquisition platform, a data annotation platform, an algorithm platform, and a use feedback platform.
- the data acquisition platform can provide users with a data upload interface to receive data uploaded by users for model training.
- the data collection platform can also provide users with data collection services.
- the user's data for model training is insufficient, the user's data collection requirements can be obtained for data collection operations.
- the user can define tasks such as "need to collect pictures containing various fruits", and the data acquisition platform can collect raw data that meets the user's needs based on the tasks entered by the user. Among them, the collected raw data may be data without annotated results.
- the data collection process can be described with reference to FIG. 2 above, and is not repeated here.
- the data annotation platform can provide users with data annotation services.
- the general workflow of the data labeling platform is that the data labeling platform can receive data labeling requests from users or data acquisition platforms, and package the desired labeling data into labeling tasks, and issue them to one or more labelers who can perform manual labeling. , The annotator will perform artificial annotation on the data to be annotated, and then the data annotation platform can organize the artificial annotation results, and save or send the collated annotation results.
- the algorithm platform can receive the data and its labeling results sent by the data labeling platform, and use the data and its labeling results to automatically perform model training.
- model training process reference may be made to the description of step S130 in FIG. 1 above, and details are not described herein again.
- the data labeling platform when the data labeling platform displays the object to be labeled to the labeler, it can also display the auxiliary prompt information to the labeler, so that the labeler can manually label the object to be labeled based on the auxiliary prompting information.
- the auxiliary prompt information may be generated by the labeling platform, or the algorithm platform may generate the auxiliary prompt information and send it to the labeling platform.
- the user can return the prediction of the accepted model or the prediction of the unacceptable model to the feedback platform.
- the data corresponding to the predictions that the user does not accept can be fed back to the data annotation platform, and then re-annotated to the model to focus on learning. As a result, the model will be closer to the user's expectations.
- the platform of the present disclosure can be an automated and intelligent full-process computer vision platform. It can integrate data collection, data annotation, automatic model generation, automatic model launch, model update, and model interpretation. One. Based on this disclosure, users can quickly obtain the online service capabilities of the model by selecting the problem mode to be solved, and reduce the threshold for obtaining model prediction services to a level that does not require the user to know algorithm knowledge.
- the user can upload labeled image data, or upload unlabeled image data for labeling through the platform, and can also publish collection tasks and labeling tasks to obtain three forms of labeled data to obtain data x and learning objects y ( (Ie labeling results).
- the platform can automatically fit the relationship between x and y, and output the fitted model to the online.
- the user can upload the input image x through various methods such as http request and grpc request, and the service returns the Forecast y.
- the present disclosure can automatically select the optimal model framework and parameters by defining the task type. Regardless of whether the user has annotations or even data, users can obtain annotations and data through crowdsourcing after defining tasks on the platform.
- the platform can interpret and evaluate the training result model, and automatically go online as a web service that users can use.
- FIG. 9 illustrates a block diagram of an apparatus for performing a machine learning process according to an exemplary embodiment of the present disclosure.
- the functional modules of the device for implementing the marking may be implemented by hardware, software, or a combination of hardware and software that implements the principles of the present disclosure.
- the functional modules described in FIG. 9 can be combined or divided into sub-modules, thereby realizing the principle of the above invention. Therefore, the description herein may support any possible combination, or division, or further definition of the functional modules described herein.
- the following only briefly describes the functional modules that the device performing the machine learning process can have and the operations that can be performed by each functional module. For the details involved, please refer to the relevant descriptions above, and will not be repeated here.
- the apparatus 900 for performing a machine learning process may include a data acquisition module 910, a labeling result acquisition module 920, a selection module 930, and a training module 940.
- the data acquisition module 910 is configured to acquire data. As shown in FIG. 10, the data acquisition module 910 may optionally include a requirement acquisition module 911, an analysis module 913, and an acquisition module 915.
- the demand obtaining module 911 may be used to obtain a data collection demand of a user.
- the analysis module 913 may be used to analyze the data collection requirements to determine the keywords possessed by the data suitable for collection.
- the collection module 915 may be used to collect data with keywords.
- the labeling result acquisition module 920 is configured to acquire the labeling results of the data.
- the labeling result acquiring module 920 may issue a labeling task corresponding to the prediction target according to the prediction target of the model training to obtain the labeling result of the data.
- the annotation result acquisition module 920 may optionally further include a display module 921, an auxiliary prompt information acquisition module 923, and a providing module 925.
- the display module 921 is used to display the object to be labeled to the labeler.
- the auxiliary prompt information acquisition module 923 is configured to acquire auxiliary prompt information for prompting a labeling conclusion of an object to be labeled.
- the providing module 925 is configured to provide ancillary prompting information to the annotator, so that the annotator performs artificial annotation in the object to be annotated based on the ancillary prompting information.
- the selection module 930 is used to select a model frame that matches the user's needs and / or a model that matches the user's prediction goal.
- the model framework is a framework for training a model based on a machine learning algorithm.
- the selection module 930 may select a model framework from a model framework corresponding to a task type matching a user's needs, and / or, the selection module 930 may select a model framework from a previous one.
- a model matching the user's prediction target is selected.
- the device 900 may optionally further include a setting module 950 shown by a dashed box in the figure.
- the setting module 950 is configured to preset one or more task types, and set one or more model frameworks for each of the task types.
- the previously trained model may be obtained by performing model training based on a corresponding model framework.
- the selection module 930 may first search a model suitable for the user's prediction target from the previously trained models, and then further select from the searched models a model frame training obtained based on a task type that matches the user's needs. model.
- the selection module 930 may first select a model trained based on a model framework corresponding to a task type matching a user's needs from a previously trained model, and then further search for a user-friendly model from the selected model.
- a model for predicting targets may be obtained by performing model training based on a corresponding model framework.
- the training module 940 is configured to perform model training based on the selected model framework and / or model by using the data and the labeled results.
- the training module 940 may use the data and its labeled results to perform model training based on the selected model framework, or the training module 940 may also use the data and its labeled results to update the selected model, or the training module 940 may also In the case where a model matching the user's prediction target is selected, the selected model is updated by using the data and its labeling result, and if the model matching the user's prediction target is not selected, based on the selected Model framework, using the data and its labeled results for model training.
- the device 900 may optionally further include a storage module 970 and / or a storage module 975 shown by dashed boxes in the figure.
- the storage module 970 is configured to store the data and the labeling result in a user database corresponding to the user.
- the saving module 975 is configured to save the trained model.
- the external usage rights of the trained model and / or user database are related to user settings. As an example, in a case where the user database is set to be open to the outside, the data in the user database may be used by other users, and / or in a case where the model is set to be open to the outside, the model may be used by other users.
- the device 900 may optionally further include an interface generation module 980 shown by a dashed box in the figure.
- the interface generating module 980 is configured to generate a user-oriented application programming interface in response to a user's model application request after the model training is completed, so that the user can obtain the prediction service provided by the model through the application programming interface.
- the device 900 may optionally further include a feedback information receiving module 985 and a model update module 990 indicated by dashed boxes in the figure.
- the feedback information receiving module 985 is configured to receive feedback information of the user on the prediction service
- the model update module 990 is configured to update the model according to the feedback information.
- the feedback information may include updated annotations that correct the predicted annotations provided by the prediction service, wherein the model update module 990 may form a new training sample based on the data corresponding to the predicted annotations and their updated annotations, And use the new training samples to update the model.
- the feedback information may also include rejection information for rejecting the predictive annotations provided by the prediction service, where the model update module 990 may re-obtain the annotation results of the data corresponding to the rejected predictive annotations, based on The data and its re-acquired labeling results form a new training sample, and use the new training sample to update the model.
- the device 900 may optionally further include an interpretation module 995 shown by a dashed box in the figure.
- the interpretation module 995 is configured to, after the user uses the model to predict an input to obtain an output, explain the influence of different input parts on the output on the output.
- the interpretation module 995 may optionally include a division module 9951, a transformation processing module 9953, an operation module 9955, an influence determination module 9957, and a notification module 9959.
- the dividing module 9951 is configured to divide the input into a plurality of input parts.
- the transformation processing module 9953 is used for each input part, and keeps the other input parts unchanged, and only performs transformation processing on the input part to obtain a new input.
- the operation module 9955 is configured to re-input each new input into the model to budget, so as to obtain a new output obtained by the model for the new input.
- the influence determination module 9957 is used to determine the influence of different input parts on the output according to the difference between the new output and the output.
- the notification module 9959 is used to notify the user of the influence of different input parts on the output in an understandable form.
- the device 900 may optionally further include an adjustment module 997 indicated by a dashed box in the figure.
- the adjustment module 997 is configured to dynamically adjust the resources used by the model to provide prediction services according to the amount of prediction requests initiated by the user through the application programming interface.
- FIG. 13 is a schematic structural diagram of a computing device that can be used to implement data processing of the foregoing method according to an embodiment of the present disclosure.
- the computing device 1200 includes a memory 1210 and a processor 1220.
- the processor 1220 may be a multi-core processor, or may include multiple processors.
- the processor 1220 may include a general-purpose main processor and one or more special co-processors, such as a graphics processor (GPU), a digital signal processor (DSP), and the like.
- the processor 1220 may be implemented using a custom circuit, such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA).
- ASIC Application Specific Integrated Circuit
- FPGA Field Programmable Gate Array
- the memory 1210 may include various types of storage units, such as a system memory, a read-only memory (ROM), and a permanent storage device.
- the ROM may store static data or instructions required by the processor 1220 or other modules of the computer.
- the persistent storage device may be a readable and writable storage device.
- the permanent storage device may be a non-volatile storage device that does not lose stored instructions and data even after the computer is powered off.
- the permanent storage device uses a mass storage device (eg, magnetic or optical disk, flash memory) as the permanent storage device.
- the permanent storage device may be a removable storage device (for example, a floppy disk, an optical drive).
- the system memory can be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory.
- System memory can store some or all of the instructions and data required by the processor while it is running.
- the memory 1210 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and magnetic disks and / or optical disks may also be used.
- the memory 1210 may include a readable and / or writeable removable storage device, such as a compact disc (CD), a read-only digital versatile disc (eg, DVD-ROM, dual-layer DVD-ROM), Read-only Blu-ray discs, ultra-density discs, flash cards (such as SD cards, SD cards, Micro-SD cards, etc.), magnetic floppy disks, etc.
- a readable and / or writeable removable storage device such as a compact disc (CD), a read-only digital versatile disc (eg, DVD-ROM, dual-layer DVD-ROM), Read-only Blu-ray discs, ultra-density discs, flash cards (such as SD cards, SD cards, Micro-SD cards, etc.), magnetic floppy disks, etc.
- Computer-readable storage media does not contain carrier waves and transient electronic signals transmitted by wireless or wire.
- the memory 1210 stores processable code.
- the processor 1220 can be caused to execute the method mentioned above.
- the processor may be implemented as a computing device
- the memory may be implemented as at least one storage device storing instructions
- the computing device may be implemented as a storage device including at least one computing device and at least one storage instruction. The system, wherein the instructions, when executed by the at least one computing device, cause the at least one computing device to perform related steps for performing a machine learning process.
- the method according to the present disclosure may also be implemented as a computer program or computer program product including computer program code instructions for performing the above steps defined in the above method of the present disclosure.
- it may be implemented as a computer-readable storage medium storing instructions, wherein when the instructions are executed by at least one computing device, the at least one computing device is caused to perform the above steps defined in the above method of the present disclosure.
- the present disclosure may also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium or machine-readable storage medium) on which executable code (or computer program or computer instruction code) is stored. )
- executable code or computer program, or computer instruction code
- the processor is caused to perform each step of the above-mentioned method according to the present disclosure.
- each block in the flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable functions for implementing a specified logical function. instruction.
- the functions labeled in the blocks may occur in a different order than those labeled in the figures. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved.
- each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts can be implemented by a dedicated hardware-based system that performs the specified function or operation , Or it can be implemented with a combination of dedicated hardware and computer instructions.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Medical Informatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Probability & Statistics with Applications (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- General Health & Medical Sciences (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Electrically Operated Instructional Devices (AREA)
- Image Analysis (AREA)
Abstract
一种执行机器学***线上。
Description
本公开总体来说涉及机器学习领域,更具体地讲,涉及一种执行机器学习过程的方法、装置、设备以及存储介质。
机器学习(包括深度学习)是人工智能研究发展到一定阶段的必然产物,其致力于通过计算的手段,利用经验来改善***自身的性能。在计算机***中,“经验”通常以“数据”形式存在,通过机器学习算法,可从数据中产生“模型”,也就是说,将经验数据提供给机器学习算法,就能基于这些经验数据产生模型,在面对新的样例时,模型会提供相应的判断,即,预测结果。
目前,在训练机器学习模型时,需要相关人员熟悉机器学习技术,为此,通常会专门建立算法团队,人力成本投入较大。
发明内容
本公开的示例性实施例在于提供一种执行机器学习过程的方法、装置、设备以及存储介质,以解决上述至少一种问题。
根据本公开的第一个方面,提供了一种执行机器学习过程的方法,包括:获取数据;获取数据的标注结果;以及选取与用户的需求相匹配的模型框架和/或与用户的预测目标相匹配的模型,基于选取到的模型框架和/或模型,利用数据及其标注结果来进行模型训练,其中,模型框架为用于基于机器学习算法训练模型的框架。
根据本公开的第二个方面,还提供了一种执行机器学习过程的装置,包括:数据获取模块,用于获取数据;标注结果获取模块,用于获取数据的标注结果;选取模块,用于选取与用户的需求相匹配的模型框架和/或与用户的预测目标相匹配的模型其中,模型框架为用于基于机器学习算法训练模型的框架;以及训练模块,用于基于选取到的模型框架和/或模型,利用数据及其标注结果来进行模型训练。
根据本公开的第三个方面,还提供了一种计算设备,包括:处理器;以及存储器,其上存储有可执行代码,当可执行代码被处理器执行时,使处理器执行如本公开第一个方面述及的方法。
根据本公开的第四个方面,还提供了一种非暂时性机器可读存储介质,其上存储有可执行代码,当可执行代码被电子设备的处理器执行时,使处理器执行如本公开第一个方面述及的方法。
基于本公开可以自动地选择模型框架和/或模型进行模型训练,从而可以把机器学***线上。
通过下面结合示例性地示出实施例的附图进行的描述,本公开示例性实施例的上述和其他目的和特点将会变得更加清楚,其中:
图1示出了根据本公开示例性实施例的执行机器学习过程的方法的流程图。
图2示出了根据本公开示例性实施例的数据获取方法的流程图。
图3示出了根据本公开示例性实施例的辅助标注方法的流程图。
图4是示出了根据本公开一实施例的标注界面示意图。
图5示出了模型生成后的界面示意图。
图6示出了根据本公开示例性实施例的模型解释方法的流程图。
图7示出了根据本公开示例性实施例的模型训练过程示意图。
图8示出了本公开实现为一种全流程自动机器学***台架构图。
图9示出了根据本公开示例性实施例的执行机器学习过程的装置的框图。
图10示出了数据获取模块可以具有的功能模块的结构示意图。
图11示出了标注结果获取模块可以具有的功能模块的结构示意图。
图12示出了解释模块可以具有的功能模块的结构示意图。
图13示出了根据本公开一实施例可用于实现上述方法的数据处理的计算设备的结构示意图。
现将详细参照本公开的实施例,所述实施例的示例在附图中示出,其中,相同的标号始终指的是相同的部件。以下将通过参照附图来说明所述实施例,以便解释本公开。在此需要说明的是,在本公开中出现的“并且/或者”、“和/或”均表示包含三种并列的情况。例如“包括A和/或B”表示包括A和B中的至少一下,即包括如下三种并列的情况:(1)包括A;(2)包括B;(3)包括A和B。又例如“执行步骤一并且/或者步骤二”表示执行步骤一和步骤二中的至少一个,即表示如下三种并列的情况:(1)执行步骤一;(2)执行步骤二;(3)执行步骤一和步骤二。
图1示出了根据本公开示例性实施例的执行机器学习过程的方法的流程图。其中,本公开述及的机器学习除了包括诸如逻辑回归、支持向量机、GBDT(梯度提升决策树)、朴素贝叶斯算法等算法之外,还可以包括基于神经网络的深度学习。该方法由至少一个计算装置执行。
参见图1,在步骤S110,获取数据。
所获取的数据可以是用户上传的数据,也可以是通过其它方式采集的数据,如可以是通过网络爬取、数据库检索、下发数据采集任务给采集者等方式采集得到的数据。其中,本公开述及的用户是指期望进行模型训练的用户,采集者是指能执行数据采集任务以采集相应数据的人员。
在步骤S120,获取数据的标注结果。
基于步骤S110获取的数据可以是具有标注结果的数据,也可以是不具有标注结果的数据。本公开对标注结果的获取方式不进行任何限制,也就是说,数据可通过任何方式而具有标注结果,该标注结果既可以是客观真实的标注结论,也可以是人为标注的主观结果。在基于步骤S110获取的数据具有标注结果的情况下,可以直接获取数据的标注结果;在基于步骤S110获取的数据没有标注结果,或者获取的数据中有部分数据没有标注结果的情况下,可以对数据进行标注来获取数据的标注结果。作为示例,可以根据模型训练的预测目标,发布与预测目标相应的标注任务,由标注者进行人工标注,以此来获取数据的标注结果。其中,预测目标是指用户期望训练出的模型所能实现的预测功能,例如在用户期望训练出用于识别图像中的猫的模型时,预测目标是“识别图像中的猫”。标注者是指能够进行人工标注的人员。
在步骤S130,选取与用户的需求相匹配的模型框架和/或与用户的预测目标相匹配的模型,基于选取到的模型框架和/或模型,利用数据及其标注结果来进行模型训练。
本公开述及的模型框架为用于基于机器学习算法训练模型的框架。在本公开中,可以预先设定一种或多种任务类型,并且可以针对每种任务类型,设置一个或多个模型框架。根据本公开的示例性实施例,可针对每种任务类型的特性,预先设置相应的一个或多个模型框架,使得与设置的模型框架所对应的机器学习算法有助于解决相应的任务类型。由此,选取与用户的需求相匹配的模型框架,可以是指从与用户的需求相匹配的任务类型所对应的模型框架中选取模型框架。其中,关于选取模型框架的实现过程及任务类型的描述将在下文详细说明。
此处述及的模型可以是之前训练好的模型,该模型可以是利用本公开训练得到的模型,也可以是利用其它方式训练得到的模型。例如,该模型可以是基于相应模型框架使用训练样本进行模型训练得到的模型。
因此,选取与用户的预测目标相匹配的模型,可以是指从之前训练好的模型中选取与用户的预测目标相匹配的模型。其中,预测目标是指用户期望训练出的模型所实现的预测功能,例如,在期望训练出用于识别图像中的猫的模型时,预测目标是“识别图像中的猫”。 与用户的预测目标相匹配的模型是指所能实现的功能与预测目标相同或相似的模型,例如,在用户的预测目标为“识别图像中的猫”时,可以将之前训练好的用于对图像中的猫进行识别的模型作为与用户的预测目标相匹配的模型,也可以将之前训练好的用于对图像中其它类型的动物(例如狗、猪等动物)进行识别的模型作为与用户的预测目标相匹配的模型。
根据本公开的示例性实施例,步骤S130中述及的模型可以是基于相应模型框架进行模型训练得到的,在这种情况下,在从之前训练好的模型中选取与用户的预测目标相匹配的模型时,可以首先从之前训练好的模型中搜索适用于用户的预测目标的模型,然后再从搜索出的模型中进一步选取基于与用户的需求相匹配的任务类型所对应的模型框架训练得到的模型;或者也可以首先从之前训练好的模型中选取基于与用户的需求相匹配的任务类型所对应的模型框架训练得到的模型,然后再从选取出的模型中进一步搜索适用于用户的预测目标的模型。由此,可以使得搜索得到的模型更加契合用户的需求。
在利用数据及其标注结果进行训练时,可以基于选取到的模型框架,利用数据及其标注结果进行模型训练;也可以基于选取到的模型进行模型训练,如可以利用数据及其标注结果来更新选取到的模型;还可以根据模型框架和模型的总选取情况,利用数据及标注结果进行模型训练,如可以优选利用选取到的模型进行模型训练,在选取不到合适的模型的情况下,再基于选取的模型框架进行模型训练。具体地,可以在选取到与用户的预测目标相匹配的模型的情况下,利用数据及其标注结果来更新选取到的模型,并且在没有选取到与用户的预测目标相匹配的模型的情况下,基于选取的模型框架,利用数据及其标注结果进行模型训练。其中,在基于选取到的模型进行模型训练时,还可以对选取到的模型进行调整,如可以对模型的网络结构进行微调,并基于微调后的模型进行模型训练。
根据本公开的示例性实施例,可以将获取的数据及其标注结果存储在与用户对应的用户数据库中,并且可以保存训练得到的模型。其中,训练得到的模型(也即下文述及的用户模型)和/或用户数据库的对外使用权限可以与用户的设置有关。在用户数据库设置为对外开放的情况下,用户数据库中的数据可被其他用户使用,并且/或者,在模型设置为对外开放的情况下,模型可被其他用户使用。
根据本公开的另一示例性实施例,在模型训练完成后,可以响应于用户的模型应用请求,生成面向用户的应用程序编程接口(API),以便用户通过应用程序编程接口获取模型提供的预测服务。
进一步地,还可以根据用户通过应用程序编程接口发起的预测请求量,来动态地调整利用模型提供预测服务所使用的资源。例如,在预测请求量较大时,可以为用户分配较多的CPU、内存等资源,在预测请求量较小时,可以为用户分配较少的CPU、内存等资源。由此,在满足用户需求的同时,可以节省平台资源。
至此,结合图1就本公开的执行机器学***线上,并且基于本公开可以为用户快速提供模型的线上服务能力。
图1所示的方法可以实现为一种机器学***台可以帮助用户较为自动地实现机器学***台还可以集数据获取、数据标注、模型自动上线(即,利用模型提供在线预测服务)、模型更新、模型解释等诸多功能中的一种或多种,以更好地服务用户。
下面就本公开涉及的各方面及相关细节做进一步说明。
【数据获取】
在获取数据时,可以获取用户上传的数据,其中,用户上传的数据可以是带标注的数据,也可以是不带标注的数据。另外,也可以根据用户需求,采集数据,如可以在用户没有数据或者上传的数据不足的情况下,采集数据。
图2示出了根据本公开示例性实施例的数据获取方法的流程图。
参见图2,在步骤S111,获取用户的数据采集需求。
作为示例,可以在用户没有上传用于模型训练的数据,或者,用户上传的用于模型训练的数据的数据量不足(例如小于第一预定阈值)的情况下,获取用户的数据采集需求。
数据采集需求是用户对其期望采集的数据的描述,数据采集需求可以是文本,也可以是语音。例如,用户的数据采集需求可以是“需要采集包含各种水果的图片”这一文本,也可以对应该文本的语音。此外,这里的数据采集需求还可通过分析模型的预测目标而获得,例如,当用户指定预测目标为猫狗分类模型,则可通过分析这一预测目标而确定数据采集需求为包含猫或狗的图片。
在步骤S112,对数据采集需求进行解析,以确定适于采集的数据所具有的关键字。
在确定了数据采集需求之后,可通过解析该需求的含义或组成部分而进一步得到相关数据的关键字。
作为示例,在数据采集需求是文本的情况下,可以直接利用语义分析(例如NLP技术)的方式对数据采集需求进行解析,以确定适于采集的数据所具有的关键字。在数据采集需求是语音的情况下,可以首先利用语音识别技术将语音识别为文本,然后利用语义分析(例如NLP技术)的方式对数据采集需求进行解析,以确定适于采集的数据所具有的关键字。关键字可以视为数据在一个或多个特征维度下的概括性描述。以数据是图片为例,关键字可以是图片的标签,如清晰度、图片内容描述、图片来源等标签都可以作为关键字。
作为示例,可以对数据采集需求进行语义分析,以确定用户期望获取的数据对象,例如,数据对象可以是指用户期望获取的数据所包含的对象,如可以是数据所包含的物体(或项目)。例如,对于“需要采集包含各种水果的图片”这一数据采集需求来说,可以确定用户期望获取的数据对象是“水果”。
可选地,在得到数据对象后,还可以将数据对象应用于知识图谱,以得到衍生对象。其中,衍生对象可以是与数据对象的类型相同或相似的横向衍生对象,也可以是属于数据对象的子类的向下衍生对象。例如,对于“水果”这一数据对象,通过知识图谱,可以得到“苹果”、“香蕉”、“橘子”、“樱桃”等多个向下衍生对象;再例如,在数据对象包括“苹果”、“香蕉”、“橘子”时,也可以通过知识图谱,得到“梨”、“桃子”、“菠萝”等横向衍生对象。因此,本公开述及的关键字可以指示数据对象和/或衍生对象。
在步骤S113,采集具有关键字的数据。
作为示例,可以通过但不限于如下三种方式中的任意一种或多种来采集数据。方式一,可以从数据库中检索具有关键字的数据,其中,数据库中的数据为关键字已知的数据,此处述及的数据库可以包括公开数据库和/或用户数据库,其中,公开数据库是指对外开放的数据库,用户数据库是针其他用户维护的私有数据库,用户数据库的对外开放权限与对应用户的设置有关。方式二,可以从网络搜索具有关键字的数据,例如可以通过网络爬取的方式获取具有关键字的数据。方式三,可以生成用于采集具有关键字的数据的采集任务,将采集任务下发给一个或多个采集者,以由采集者采集具有关键字的数据。
如上文所述,还可以维护与用户对应的用户数据库,采集到的数据可以存储到用户数据库中。并且,在采集得到数据后,还可以根据模型训练的预测目标,发布与预测目标相应的标注任务,获取采集到的数据的标注结果,并将数据的标注结果与数据关联地存储到用户数据库。
在本公开中,可以根据用户的权限设置,确定用户数据库的对外使用权限。在用户数据库被设置为对外开放的情况下,用户数据库中的数据可被其他用户使用。例如,在从数据库中检索具有关键字的数据时,可以从对外开放的其他用户的用户数据库中进行检索。
由此,对于没有数据或数据量不足的用户,在期望通过机器学习技术训练用于解决特定问题的模型时,可自动地有效实现应用于模型训练的数据的采集。
【数据标注】
在获取的数据不带有标注结果,或者数据中带标注结果的数据的数据量不足(例如小于第二预定阈值)的情况下,可以根据模型训练的预测目标,发布与预测目标相应的标注 任务,以获取数据的标注结果。其中,标注任务可以下发给一个或多个能够执行人工标注的标注者,由标注者对待标注的数据执行人为标注,然后可以整理人为标注结果,如可以将标注结果与数据关联地保存。
在标注者进行人工标注的过程中,为了减少标注者执行标注所付出的劳力,提高标注效率,节约人力成本,本公开还提出了一种辅助标注方案。
图3示出了根据本公开示例性实施例的辅助标注方法的流程图。
参见图3,在步骤S121,向标注者展示待标注对象。
此处主要是以可视化地形式向标注者展示待标注对象。其中,待标注对象可包括待标注的原始数据,例如,待标注对象可以是包括待标注物体(或项目)的图像,也可以是一段包括待标注词性的词语的文本。至于需要在待标注对象中进行标注的物体(或项目)、标注形式以及标注内容,与标注任务的具体标注需求有关。
在步骤S122,获取用于提示待标注对象的标注结论的辅助提示信息。其中,步骤S122可以在步骤S121之前执行,也可以与步骤S121同时或在其之后执行,关于步骤S122和步骤S121的先后执行顺序本公开不做限定。
标注结论可以视为待标注对象的真实标记(label)。应注意,这里描述的是,所获取的辅助提示信息用于提示待标注对象的标注结论,也就是说,辅助提示信息本身并不等同于标注结论,而仅仅是一种初步的标注结论,其作用在于能够在一定程度上提示出待标注对象的真实标注结论。所以,在实际应用中,所获取的辅助提示信息很有可能会偏离待标注对象的真实标注结论,甚至相反。也就是说,虽然步骤S122的本意在获取能够接近待标注对象的真实标注结论的辅助提示信息,但是由于辅助提示信息的获取手段(将在下文述及)不同、准确度不同,最终获取的辅助提示信息也有可能提示错误的标注结论。
在步骤S123,向标注者提供辅助提示信息,以便标注者基于辅助提示信息在待标注对象中执行人为标注。
此处主要是以可被人理解的方式向标注者提供辅助提示信息,例如可以可视化地向标注者展示辅助提示信息。其中,根据标注任务所涉及的标注问题不同,辅助提示信息的内容及展示形式也不尽相同。
一般来说,标注问题主要可以分为分类问题和识别问题。具体又可以细分为图像分类、物体画框、语义分割、图像注释、人脸打点、视频追踪等多种标注问题。其中,图像分类可以是指根据图像内容,选择图像或图像中物体所属的标签,比如所属场景标签、物体种类标签、物体属性标签、人物性别标签、人物年龄标签等等;物体画框可以是指根据标注需求,对图像中的目标物体进行画框,比如图像中的车辆、车牌、行人、道路、建筑、船只、文字、人体部位等画框并打上对应标签;语义分割可以是指根据标注需求,使用用多边形标注图像中目标物体的轮廓,返回所有轮廓组成点的坐标;图像注释可以是指针对图像中具体情景及标注需求,为每张图像生成中文标注语句;人脸打点可以是指针对图像中的人脸及标注需求,对人脸的关键位置,如脸廓、眉毛、眼睛、嘴唇进行定位并打点;视频追踪可以是指从目标视频中按指定频率抽取关键帧,然后对每帧进行画框标注,各帧中同一目标的标签和序号保持一致。
以标注问题是图像分类为例,辅助提示信息可以是待标注对象(即待标注图像)的初步标注结论(例如标签);以标注问题是物体画框为例,辅助提示信息可以包括框选结果和标注信息两部分;以标注问题是语义分割为例,辅助提示信息可以是待标注对象中目标物体的轮廓框选结果;以标注问题是人脸打点为例,辅助提示信息可以是待标注对象中人脸上多个关键位置的打点结果;以标注问题是视频追踪为例,辅助提示信息可以是抽取的每帧画面中目标物体的画框结果。另外,对于其它类型的标注问题,辅助提示信息的具体内容及展示形式也不尽相同,对此本公开不再赘述。
如上文所述,辅助提示信息用于提示待标注对象的标注结论。因此,标注者可以基于辅助提示信息在待标注对象中执行人为标注。其中,向标注者提供的辅助提示信息可以视为***默认的标注结论,标注者可以根据自身认知,确定辅助提示信息是否与其想要标注的结果一致,如果认为一致,标注者可以通过接受该辅助提示信息,完成待标注对象的标 注,由此可以大大提高标注效率。另外,如果标注者认为辅助提示信息与其想要标注的结果稍有差别,标注者还可以对辅助提示信息进行调整,例如调整框选范围、调整内容描述等等。更进一步地,如果标注者认为该辅助提示信息与其想要标注的结果差异很大,标注者还可以丢弃辅助提示信息,并在待标注对象中执行与辅助提示信息完全不同的人为标注。
作为本公开的一个示例,可以在向标注者展示待标注对象的同时,向标注者提供辅助提示信息。也就是说,向标注者展示的待标注对象中可以包括辅助提示信息。举例来说,标注任务可以是“猪脸标注”,具体标注需求可以是对待标注图像中的猪脸进行框选,因此,辅助提示信息可以是图像中的猪脸框选初步结果。参见图4,待标注对象为图像,图像中的线框即是辅助提示信息,也即猪脸框选初步结果。标注者可以接受该框选结果,也可以对框选结果进行重新调整,以重新确定框选范围,例如,标注者可以缩小线框范围,并新增一个线框,以分别框选两个猪脸,并较少地框选猪脸外的部分。
可见,辅助提示信息仅用于提示待标注对象的标注结论,其并非一定是准确无误的。因此,标注者可以基于自身认知,接受辅助提示信息,也可以对辅助提示信息进行调整,还可以舍弃辅助提示信息,而做出与辅助提示信息完全不同的标注操作。也就是说,辅助提示信息仅起提示作用,最终的标注结果仍掌握在标注者手中。
作为本公开的一个示例,可以将人为标注结果与辅助提示信息之间的差异提示给标注者。具体地,可以响应于标注者执行的人为标注而获取所述待标注对象的人为标注结果,并且将人为标注结果与辅助提示信息之间的差异提供给标注者。其中,可以响应于标注者执行的人为标注,实时地提示这种差异,也可以仅在差异大于一定阈值(为了便于区分,此处可以称为“第三预定阈值”)的情况下,将这种差异提示给标注者,以提示标注者关注这种差异过大的标注结果,从而可以在一定程度上降低因标注者大意而造成的误标注操作。
至此,结合图3、图4就本公开的辅助标注方案的基本原理及实现过程做说明。下面就辅助标注方案中涉及的各方面做进一步说明。
1、辅助提示信息
作为示例,可以通过如下两种方式获取辅助提示信息。
第一种
可以依靠标注结论已知的对象获取辅助提示信息。具体地,可以基于与待标注对象相同或相似的对象的标注结论,得到辅助提示信息。例如,可以直接将与待标注对象相同或相似的对象的标注结论,作为待标注对象的辅助提示信息。其中,与待标注对象相同或相似的对象的标注结论可以是人为标注结果,也可以是模型预测结果,还可以是真实结论等等。
作为示例,可以通过多种方式获取具有标注结论的与待标注对象相同或相似的对象。
例如,可以从存储了各种对象的数据库中选取具有标注结论的与待标注对象相同或相似的对象。其中,数据库可以是平台维护的数据库,数据库中存储的对象优选地可以是标注结论已知的对象。其中,关于数据库中对象的来源,本公开不做限定,如可以是经过人工标注后的对象,也可以是具有真实标注结论的对象(例如,公共数据等)。
再例如,也可以通过网络获取具有标注结论的与待标注对象相同或相似的对象,如可以通过网络爬虫的方式获取真实标注结论已知的与待标注对象相同或相似的对象。
还例如,还可以将与待标注对象属于同一标注任务中的其它已标注对象,作为与待标注对象相同或相似的对象。其中,已标注对象可以是经过标注且标注结果检验通过的对象。
另外,还可以采取其它方式获取与待标注对象相同或相似的具有标注结论的对象,并且在获取与待标注对象相同或相似的对象时,可以从上述多种获取方式中任意选取一种或多种,对此本公开不做限定。
第二种
可以依靠机器学***台上发布图像标注任务的情况下(例如,客户把自己猪场的图像数据上传,期待标注者来标注),则可以针对该客户(也即该客户的标注任务)训练统一的机器学习模型,而不必考虑不同标注者的标注差异。机器学习模型可以被训练为用于预测待标注对象的标注结论,所预测的标注结论即可以作为辅助提示信息。
作为本公开的一个示例,机器学习模型可以是基于至少一部分与待标注对象属于同一标注任务中的已标注对象及其人为标注结果而训练出的。也就是说,可以将同一标注任务中的至少一部分已标注对象及其人为标注结果,作为训练样本,进行模型训练。其中,这里的训练样本优选地可产生自人为标注结果得到检验认可的已标注对象,也就是说,该已标注对象及其人为标注结果可作为模型的训练样本。由此,机器学习模型的训练过程可以在标注任务发布了一段时间从而积攒了适当数量的训练样本之后进行。
作为本公开的另一个示例,机器学习模型也可以是基于与待标注对象相同或相似的非标注对象及其真实标注结论而训练出的。也就是说,可以将与待标注对象相同或相似的非标注对象及其真实标注结论作为训练样本,进行模型训练。其中,非标注对象可以是真实标注结论已知的对象,例如,可以是数据库中收集的已知数据、先前存有数据或者网络来源的数据等等。由此,还可以解决冷启动问题,使得机器学习模型的训练过程可以事先进行,如可以在向标注者下发标注任务,展示待标注对象前,针对该标注任务提前训练机器学习模型。
2、机器学习模型
如上文所述,机器学习模型被训练为用于预测待标注对象的标注结论,所预测的标注结论即可以作为辅助提示信息。因此,机器学习模型预测的准确度越高,向标注者提供的辅助提示信息越加接近真实标注结论,标注者基于辅助提示信息执行人为标注所付出的劳动也就越小,人工标注成本也就越低。
基于这种考虑,本公开提出,可以根据标注者针对待标注对象的人为标注结果,反过来对机器学习模型进行更新,以提高机器学习模型的预测结果的准确性。具体地,可以响应于标注者执行的人为标注而获取待标注对象的人为标注结果,并基于待标注对象及其人为标注结果,更新机器学习模型。
由于人为标注结果是基于标注者自身认知得到的,并不一定是准确的。因此,优选地,可以对待标注对象的人为标注结果进行审核,并利用审核通过的待标注对象及其人为标注结果来重新训练或增量训练机器学习模型。例如,在分类任务下,可以将审核通过的待标注对象的特征作为训练样本的特征,人为标注结果作为训练样本的标记,构造训练样本,重新训练或增量训练机器学习模型。其中,模型的重新训练或增量训练过程为本领域成熟技术,此处不再赘述。
作为本公开的一个示例,可以重点学习与模型的预测结果(即辅助提示信息)差异较大的人为标注结果(前提是该差异不是标注者胡乱标注引起的)。也就是说,可以基于人为标注结果与辅助提示信息之间的差异大于第三预定阈值的待标注对象及其人为标注结果,更新机器学习模型。其中,这里的人为标注结果可以是审核通过的结果,也就是说,可以将审核通过的且差异大于第三预定阈值的人为标注结果及待标注对象作为训练样本,来更新机器学习模型。
作为可选方式,还可额外获取待标注对象的反馈标注结果,以构成用于更新机器学***台上可额外建立反馈机制,能够收集关于待标注对象的反馈标注结果(例如,其他主体通过对标注者的人为标注结果进行纠正而得到的反馈标注结果),并进而基于具有反馈标注结果的待标注对象来更新机器学习模型。
如上文所述,作为示例,机器学***台产生或收集的标注数据而不断得到更新,从而提高辅助提示信息的准确性。
3、标注质量评价
为了防止标注者不经思考而盲目接受辅助提示信息、更好地了解标注者的标注结果的可信度、向标注者派发报酬、调整标注者的标注等级、调整标注者的报酬奖励机制等等, 可以对标注者的标注质量进行评价。例如,可以设置人工审核员来随机抽查标注者的标注质量。
作为本公开的一个示例,可以根据标注者和一个或多个其他标注者对同一待标注对象的人为标注结果的差异,对标注者的标注质量进行评价。例如,对于同一标注任务下的同一待标注对象,在将其下发给标注者A后,还可以将其下发给多个其他标注者,例如标注者B、标注者C以及标注者D,其中下发给多个其他标注者时,优选地可以选择标注质量评价较高的标注者进行下发。然后可以根据标注者A和这多个标注者对同一待标注对象的人为标注结果的差异,对标注者A的标注质量进行评价,例如可以在标注者A的标注结果与其他多个标注者的标注结果的差异都较大时,认为标注者A的标注质量较差。
作为本公开的另一个示例,还可以根据标注者对具有真实标注结论的待标注对象的人为标注结果与真实标注结论的差异,对标注者的标注质量进行评价。例如,可以随机地将真实标注结论已知的对象作为待标注对象,发送给标注者,由标注者对其进行人为标注,然后将人为标注结果与真实标注结论进行比较,如果差异较大,则认为标注者标注质量较差,如果人为标注结果与真实标注结论一致或基本一致,则认为标注者标注质量较高。
作为本公开的再一个示例,还可以根据人为标注结果与辅助提示信息之间的差异,对标注者的标注质量进行评价。例如,在利用机器学习模型获取辅助提示信息的情况下,可以在机器学习模型得到充分的更新训练后,根据人为标注结果与辅助提示信息之间的差异,对标注者的标注质量进行评价,如果标注者的人为标注结果与辅助提示信息之间的差异较大,则可以认为标注者的标注质量较差。其中,可以重点对连续接受辅助提示信息的标注者的标注质量进行监督评价。
需要说明的是,可以择一选取上述评价方式对标注者的标注质量进行评价,也可以结合上述多种评价方式对标注者的标注质量进行评价,对此本公开不做限定。
根据标注者的标注质量评价结果,可以对标注者的标注等级进行调整,例如可以根据标注者的标注质量为标注者赋予相应的信用分数。也可以调整标注者的标注报酬或惩罚,以激励标注者提升标注质量。并且,还可以根据标注者的标注质量,向标注者下发不同的标注任务。例如,可以向标注质量较高的标注者下发报酬高的标注任务,并且对于标注质量较高的标注者,可以增加任务下发数量;相应地,可以向标注质量较低的标注者下发低报酬的标注任务,并且对于标注质量较低的标注者,可以减少任务下发数量。
【模型框架】
如上文所述,可以针对每种任务类型的特性,预先设置相应的一个或多个模型框架,使得与设置的模型框架所对应的机器学习算法有助于解决相应的任务类型。
作为示例,可以根据期望解决的问题的类别,设定任务类型,不同的任务类型对应不同的问题分类。以针对图像数据的问题为例,任务类型可以分为图像分类任务、物体识别任务、文字识别任务、图像分割任务以及特征点检测任务。
图像分类,是指根据图像的语义信息区分不同的图像类别。图像分类是计算机视觉中重要的基本问题,根据图像的语义信息区分不同的图像类别,为图像打上不同的类别标签。图像分类是图像检测、实体分割、物体跟踪等其他高层视觉任务的基础,在很多领域有广泛的应用,包括安防领域的人脸识别、智能视频分析,交通领域的交通场景识别等。
物体识别,是对图片内容先进行目标定位再进行目标分类。物体识别是指根据图像的语义信息,将其中可能存在的不同物体经过检测定位框定后再进行分类和打标的过程。考虑到现实生活中的图片数据通常描述的是多种物体复合并存的场景,所以单独的图像分类往往难以有效应对。这时,物体识别借助分治的思路,先定位后分类,能够极大提升识别结果的准确性,在航天、医学、通信、工业自动化、机器人及军事领域有着广泛的应用。
文字识别,是对图片文字先进行文字定位再进行文字提取。文字识别(OCR)将图片上的文字内容智能识别为计算机可编辑的文本。根据实际使用的场景又可以分为印刷体文字识别和手写体文字识别。前者由于其文字的标准统一、风格固定,识别的准确性相对较高;后者因其本身带有一定的开放性和自由度,识别成本也相对较高。采用深度学习的文字识别技术借助其端到端的建模能力,能够有效替代人工信息录入,在票据、证件识别需 求频现的金融、保险等行业得到了显著的推广。
图像分割,是根据视觉特性对图片内容进行子区域的划分。图像分割是指将数字图像细分为多个图像子领域(像素的集合)的过程。图像分割的目的是简化或改变图像的表示形式,使得图像更容易理解和分析。图像分割通常用于定位图像中的物体和边界(线,曲线等)。更精确的,图像分割是对图像中的每个像素加标签的过程,这一过程使得具有相同标签的像素具有某种共同的视觉特性,例如颜色、亮度、纹理等。卫星图像中物体追踪定位、医学影像中的肿瘤定位、体积测量等都需要采用图像分割的技术。
特征点检测,是提取图片中具有显著视觉特性(如灰度等)的关键特征点。图像特征点指的是图像灰度值发生剧烈变化的点或者在图像边缘上曲率较大的点(即两个边缘的交点)。图像特征点能够反映图像的本质特征,能够标识图像中目标物体,并且能够通过特征点的匹配完成图像的匹配。其中,颜色、纹理作为图像全局表征能够辅助图像的理解,但容易受到环境的干扰。相对而言,斑点、角点等局部特征点往往对应图像中的线条、边缘等明暗变化的结构,受环境影响程度较小,能够有效应用于图像匹配、检索等应用场景。
在选取模型框架时,可以首先确定与用户的需求相匹配的任务类型,然后从与用户的需求相匹配的任务类型所对应的模型框架中选取模型框架。其中,可以通过多种方式确定与用户的需求相匹配的任务类型。例如,用户可以通过定义任务的形式,来表征其需求,可以从预先设定的多种任务类型中选取与用户定义的任务相匹配的任务类型,作为与用户的需求相匹配的任务类型。再例如,也可以为用户提供多种任务类型的介绍信息,以由用户根据自身需求,选取合适的任务类型。
因此,可以响应于用户的任务类型选择操作,从用户选择的任务类型所对应的模型框架中选取模型框架,也可以从一种或多种任务类型中选取与用户定义的任务相匹配的任务类型,并从选取的任务类型所对应的模型框架中选取模型框架。
作为发明的一个示例,在从与用户的需求相匹配的任务类型所对应的模型框架中选取模型框架时,可以随机选取,也可以由用户指定选取。
作为发明的另一个示例,对于与用户的需求相匹配的任务类型所对应的每个模型框架,还可以通过超参数寻优的方式,获取每个模型框架的最优超参数组合,并选取表现最好的模型框架及其最优超参数组合。例如,对于与用户的需求相匹配的任务类型所对应的每个模型框架,可以使用网格搜索(grid search)、随机搜索(random search)、贝叶斯优化等算法,设置不同的超参数组合,然后使用训练样本进行模型的训练,最后进行模型的测试,测试结果最好(如可以根据准确率、损失等测试指标进行评测)的那组超参数可以作为该模型框架下的最优超参数组合。然后,可以比较不同模型框架下的最优超参数组合,选取表现最好(如准确率高且损失小)的模型框架及其最优超参数组合。
模型框架为用于基于机器学习算法训练模型的框架,基于选取的模型框架,可以使用训练样本进行模型训练。例如,可以基于选取的模型框架及其最优超参数组合,使用训练样本进行模型训练。其中,在选取的模型框架的最优超参数组合未确定的情况下,可以使用网格搜索(grid search)、随机搜索(random search)、贝叶斯优化等算法,确定所选取的模型框架的最优超参数组合,关于最优超参数组合的寻找过程可以参见上文描述,此处不再赘述。
作为本公开的一个示例,由于不同的模型框架具有的参数量、运算消耗的时间以及理论可以达到的准确度不同,用户还可以定义模型的使用场景,如终端使用场景(运算速度快,准确率表现相对可以打折扣)、云端使用场景(运算速度可以较慢,准确率表现好)等各种场景。可以基于用户定义的模型使用场景,从与用户的需求相匹配的任务类型所对应的模型框架中选择与模型使用场景相匹配的模型框架。
图5示出了模型生成后的界面示意图。如图5所示,在成功训练出模型后,可以向用户展示模型的基本信息,如数据来源、任务类型、任务状态以及产出模型。其中,还可以向用户展示训练过程中所使用的参数配置信息。参数配置信息可以包括但不限于数据预处理参数、算法参数以及资源参数。其中,数据预处理参数主要包括针对数据执行的一种或多种预处理操作的参数,以图像数据为例,数据预处理参数可以包括随机裁剪、缩放、左 右翻转、上下翻转、旋转、超像素、灰度化、高斯模糊、均值模糊、锐化、逐点加噪、粗糙抛弃等数据预处理参数配置信息。算法参数可以是模型框架的超参数组合,其可以是通过超参数寻优的方式确定的最优超参数组合。资源参数可以包括CPU、内存等模型训练或者模型使用时的物理资源参数。
【模型解释】
在模型训练完成并上线后,用户可以使用模型对输入进行预测,以得到输出。其中,本公开还可以对输入中不同输入部分对输出的影响进行解释,以使得用户在使用机器学习模型针对输入得到输出后,还可以获知不同输入部分对输出的影响,从而可以在一定程度上理解模型主要是根据输入中哪些部分作出的预测,即输出,进而可以在一定程度上还可以增强机器学习模型的输出在用户层面的可信度。
图6示出了根据本公开示例性实施例的模型解释方法的流程图。图6所示的方法主要用于对模型针对输入(为了便于区分,也可称为“原输入”)得到的输出(为了便于区分,也可称为“原输出”)进行解释,具体地,可以用于对输入中不同输入部分对输出的影响进行解释。其中,此处述及的模型可以为机器学习模型,如可以包括基于神经网络的深度学习模型。下面主要以模型为图像模型,输入为图像,输出为图像的预测结果为例进行说明,应该知道,本公开的模型解释方法还可以适用于对其它格式的输入(如文本输入)进行预测的模型。
参见图6,在步骤S410,将输入划分成多个输入部分。
可以采取多种方式将输入划分成多个输入部分。以输入是图像为例,可以将图像划分成多个形状和大小相同或相似的区域,以得到多个输入部分。例如,可以将图像划分成N×M的网格。另外,也可以按照图像特征的相似程度,将图像划分成多个输入部分。其中,图像特征可以是指颜色、纹理、亮度等特征,同一输入部分中的像素具有相同或相似的图像特征。在根据图像特征划分输入部分时,同一输入部分中的像素可以具有相同或相似的图像特征且处于邻近位置。也就是说,在根据图像特征划分输入部分时,还可以考虑位置因素,将图像特征相同或相似且位置临近的像素划分到同一输入部分。当然,还可以有其它多种划分方式,此处不再赘述。
在步骤S420,针对每个输入部分,在保持其它输入部分不变的情况下,仅对该输入部分进行变换处理,以得到新输入。
每个新输入与原输入相比,仅其中一个输入部分是用进行了变换处理后的输入部分(可以称为“扰动输入”)代替的。针对每个输入部分,可以通过多种方式对其进行变换处理,以得到该输入部分的扰动输入(也可称为“噪声扰动”)。例如,可以在预定变换范围内对输入部分进行随机变换,以得到用于替换该输入部分的扰动部分。以输入为图像为例,可以在像素的取值范围内,对输入部分中每个像素的取值进行随机变换。其中,像素的取值范围是指像素值的范围,取值范围与比特每像素(BPP)有关,例如,对于8bpp,像素的取值个数是2
8,取值范围可以是在0-255之间;对于16bpp,像素的取值个数是2
16,取值范围可以是在0-65535之间。
在步骤S430,将每个新输入重新输入模型,以得到模型针对新输入得到的新输出。
在步骤S440,根据新输出与输出(也即原输出)之间的差异,确定不同的输入部分对输出的影响。
每个新输入可以看作是原输入中仅一个输入部分进行了变换处理后的输入,也即仅一个输入部分进行了扰动处理后的输入。如果新输出与原输出不同,可以认为新输入中进行了变换处理的输入部分对输出具有一定的影响,反之,如果新输出与原输出相同,可以认为新输入中进行了变换处理的输入部分对输出不那么重要,没有影响。
换言之,在新输出与输出相同的情况下,可以判定该新输出对应的新输入中进行了变换处理的输入部分对输出没有影响;并且/或者在新输出与输出不同的情况下,可以判定该新输出对应的新输入中进行了变换处理的输入部分对输出具有影响。
在步骤S450,将不同输入部分对输出的影响,以可理解的形式告知用户。
在确定了不同输入部分对输出的影响后,可以通过多种可被用户理解的方式将不同输 入部分对输出的影响告知用户。例如,可以将不同输入部分对输出的影响以文本(如列表)的形式告知用户,也可以在输入中标出不同输入部分对输出的影响,如可以按照影响大小,使用不同突出程度来突出呈现输入中不同输入部分的影响。以输入为图像为例,可以根据输入部分对输出的影响大小,生成图像的热力图,其中,输入部分在热力图中的突出程度与其对所述输出的影响成正比。
由此,用户在使用模型针对输入得到输出后,还可以获知不同输入部分对输出的影响,从而可以在一定程度上理解模型主要是根据输入中哪些部分作出的预测,即输出,进而还可以增强机器学习模型的输出在用户层面的可信度。
至此,结合图6就本公开的模型解释方法的基本流程做了详细说明。下面就确定不同输入部分的影响的具体实现原理进行示例性说明。
在本公开的一个实施例中,可以针对每个输入部分进行预定次数的变换处理,然后可以统计每个输入部分经过预定次数的变换处理后得到的预定数量个新输入分别输入机器学习模型后得到的新输出与输出相同的次数。其中,输入部分对输出的影响与统计得到的次数的大小成反比,即新输出与输出相同的次数越多,输入部分对输出的影响(也可以理解为重要性)越小。
在本公开的另一个实施例中,针对每个输入部分可以进行预定次数的变换处理以得到多个新输入,并且可以获取每个新输出的置信度。其中,置信度是指机器学习模型针对输入得到输出的置信度,也即机器学习模型针对输入得到输出的概率值、输出权重。可以在考虑置信度的情况下,计算每个输入部分对应的预定数量个新输出与原输出之间的差异,并根据该差异来确定不同的输入部分对输出的影响。作为示例,以二分类问题而言,可将代表两个类别的输出分别作为+1和-1,并将输出与置信度(即,概率值)的乘积作为参与比较的输出结果,在获得了预定数量个新输出的情况下,可累加每个输出对应的乘积,从而基于累加和或其均值来得到代表新输出整体上的输出结果,将该输出结果与原输出进行比较,作为二者的差异。由此,对于每个输入部分,可以根据其对应的预定数量个新输出整体上与原输出的差异,确定不同的输入部分对输出的影响。其中,输入部分对输出的影响,与差异的大小成正比,即差异越大,影响越大。
【模型更新】
在模型上线以为用户提供预测服务后,还可以接收用户针对预测服务的反馈信息,并根据反馈信息对模型进行更新。
作为本公开的一个示例,反馈信息可以包括对预测服务所提供的预测标注进行纠正的更新标注,可以基于预测标注所对应的数据及其更新标注来形成新的训练样本,并利用新的训练样本来更新模型。其中,更新标注可以是用户做出的。
作为本公开的另一个示例,反馈信息还可以仅包括对预测服务所提供的预测标注进行否决的拒绝信息,这种情况下,可以重新获取被否决的预测标注所对应的数据的标注结果,基于数据及其重新获取的标注结果来形成新的训练样本,并利用新的训练样本来更新模型。
也就是说,在使用模型的预测服务时,用户可以接受模型的预测或者不接受模型的预测,在用户不接受模型的预测时,可以将拒绝信息作为反馈信息进行反馈,也可以对预测标注进行纠正得到的更新标注作为反馈信息,根据用户的反馈信息可以实现模型的更新,以使得模型能够越来越接近用户预期。
图7示出了根据本公开示例性实施例的模型训练过程示意图。
如图7所示,首先可以执行步骤S510,判断用户上传的数据是否足够,例如,可以为用户提供数据上传接口,用户可以通过该接口上传数据。在用户未上传数据或者用户上传的数据的数据量低于预定阈值的情况下,可以认为用户的数据不足。
在判定用户的数据不足的情况下,可以执行步骤S530,发起数据采集。数据采集的过程可以参见上文图2所示,此处不再赘述。在数据采集完成后,可以执行步骤S540,发起数据标注。数据标注过程可以人工执行,例如,可以根据模型训练的预测目标,发布与预测目标相应的标注任务,由人工标注,然后获取采集到的数据的标注结果,并将数据的标注结果与数据关联地存储到用户数据库。
在判定用户上传的数据足够的情况下,可以执行步骤S520,进一步判断用户上传的数据中的标注数据是否足够,如果足够,则可以直接将数据存储到用户数据库。如果不足够,则可以执行步骤S540,发起数据标注。标注过程可以参见上文说明,此处不再赘述。
由此,对于没有数据或数据量不足的用户,在期望通过机器学习技术训练用于解决特定问题的模型时,可自动地实现应用于模型训练的数据的采集,可选地,还可进一步自动完成数据的标注。
如图7所示,数据平台可以维护公开数据库和用户数据库。公开数据库中的数据可以是完全对外开放的数据。经过标注后的数据及其标注结果可以关联地存储到用户数据库中。用户数据库的对外使用权限与用户的设置有关,在用户将其用户数据库设置为对外开放的情况下,用户数据库中的数据可被其他用户使用。由此,在执行步骤S530,发起数据采集时,不仅可以从公开数据库中采集数据,还可以从对外开放的其他用户的用户数据库中采集数据。
在得到数据及其标注结果后,可以执行步骤S550,进行模型训练。其中,在进行模型训练时,可以从模型框架库中选取与用户的需求相匹配的模型框架,并基于选取的模型框架进行模型训练,其中,模型框架库中可以包括多个分别与特定的任务类型相对应的模型框架。另外,在进行模型训练时,也可以从模型库中选取之前训练好的与用户的预测目标相匹配的模型,利用数据及其标注结果来更新选取到的模型,实现模型训练。模型库中的模型可以包括基于公开数据的模型和用户模型。基于公开数据的模型可以是基于公开数据训练得到的模型,用户模型可以是基于用户数据训练得到的模型,其可以是利用本公开训练得到的模型。基于公开数据的模型的使用权限可以是对所有用户开放,用户模型的对外使用权限可以与用户的设置相关,在从模型库中选取模型时,可以仅从权限为对外开放的模型中选取。
在模型训练完成后,可以将生成的模型作为用户模型进行保存,其中,模型的对外使用权限与用户的设置有关。可以响应于用户的模型应用请求,生成面向用户的应用程序编程接口,以便用户通过应用程序编程接口获取模型提供的预测服务。
如图7所示,用户在使用预测服务的时候,可以回传接受模型的预测或者不接受模型的预测。用户接受模型的预测的预测标注及数据可以保存到用户数据库,并且对于用户不接受的预测所对应的数据,可以重新发起数据标注,经过再次标注后交给模型着重学习,因此模型会越来越靠近用户预期。
图8示出了本公开实现为一种全流程自动学***台架构图。
如图8所示,本公开可以实现为一种全流程自动学***台可以由数据采集平台、数据标注平台、算法平台以及使用反馈平台构成。
数据采集平台可以为用户提供数据上传接口,接收用户上传的用于模型训练的数据。并且,数据采集平台还可以为用户提供数据采集服务,在用户的用于模型训练的数据不足的情况下,可以获取用户的数据采集需求,进行数据采集操作。例如,用户可以定义任务,比如“需要采集包含各种水果的图片”,数据采集平台可以根据用户输入的任务,来采集满足用户的需求的原始数据。其中,所采集的原始数据可以是不带标注结果的数据。数据采集过程可以参见上文图2描述,此处不再赘述。
数据标注平台可以为用户提供数据标注服务。数据标注平台的大致工作流程为,数据标注平台可以接收来自用户或数据采集平台的数据标注请求,并将期望标注的数据打包成标注任务,下发给一个或多个能够执行人工标注的标注者,由标注者对待标注的数据执行人为标注,然后数据标注平台可以整理人为标注结果,并保存或发送整理后的标注结果。
算法平台可以接收数据标注平台发送的数据及其标注结果,并利用数据及其标注结果自动进行模型训练。其中,关于模型训练过程可以参见上文图1中步骤S130的描述,此处不再赘述。
需要说明的是,数据标注平台在向标注者展示待标注对象时,还可以将辅助提示信息一并展示给标注者,以便标注者基于辅助提示信息在待标注对象中进行人工标注。其中,辅助提示信息可以由标注平台生成,也可以由算法平台生成辅助提示信息,并将其发送至 标注平台。
用户在使用预测服务的时候,可以向使用反馈平台回传接受模型的预测或者不接受模型的预测。用户不接受的预测所对应的数据可以反馈到数据标注平台,经过再次标注后交给模型着重学习,由此,模型会越来越靠近用户预期。
以面向图像数据为示例,本公开的平台可以是一个自动化、智能化的全流程计算机视觉平台,可以集数据采集、数据标注、模型自动生成、模型自动上线、模型更新及模型解释等诸多功能于一体。基于本公开,用户可以很简单地通过选择需要解决的问题模式,快速的获得模型的线上服务能力,把获取模型预测服务的门槛降低到不需要用户知晓算法知识的水平线上。
具体地,用户可以上传标注好的图像数据,也可以上传没有标注的图像数据通过平台进行标注,还可以发布采集任务和标注任务来获取标注数据的三种形式来获得数据x和学***台可以自动的拟合x与y之间的关系,并输出此拟合得到的模型到线上,用户可以通过http请求、grpc请求等各种方式,上传输入的图像x,服务返回对x的预测y。
综上,本公开可以通过定义任务类型自动的选择最优的模型框架和参数,用户无论有无标注,甚至有无数据,都可以通过在平台定义任务后通过众包的方式获取标注及数据,平台可以可解释化的评测训练结果模型,并自动上线成用户可以使用的web服务。
图9示出了根据本公开示例性实施例的执行机器学习过程的装置的框图。其中,用于实现标注的装置的功能模块可以由实现本公开原理的硬件、软件或硬件和软件的结合来实现。本领域技术人员可以理解的是,图9所描述的功能模块可以组合起来或者划分成子模块,从而实现上述发明的原理。因此,本文的描述可以支持对本文描述的功能模块的任何可能的组合、或者划分、或者更进一步的限定。下面仅就执行机器学习过程的装置可以具有的功能模块以及各功能模块可以执行的操作做简要说明,对于其中涉及的细节部分可以参见上文相关描述,这里不再赘述。
参见图9,执行机器学习过程的装置900可以包括数据获取模块910、标注结果获取模块920、选取模块930以及训练模块940。
数据获取模块910用于获取数据。如图10所示,数据获取模块910可以可选地包括需求获取模块911、解析模块913以及采集模块915。需求获取模块911可以用于获取用户的数据采集需求。解析模块913可以用于对数据采集需求进行解析,以确定适于采集的数据所具有的关键字。采集模块915可以用于采集具有关键字的数据。
标注结果获取模块920用于获取数据的标注结果,其中,标注结果获取模块920可以根据模型训练的预测目标,发布与预测目标相应的标注任务,以获取数据的标注结果。
如图11所示,标注结果获取模块920还可以可选地包括展示模块921、辅助提示信息获取模块923以及提供模块925。展示模块921用于向标注者展示待标注对象。辅助提示信息获取模块923用于获取用于提示待标注对象的标注结论的辅助提示信息。提供模块925用于向标注者提供辅助提示信息,以便标注者基于辅助提示信息在待标注对象中执行人为标注。
选取模块930用于选取与用户的需求相匹配的模型框架和/或与用户的预测目标相匹配的模型。其中,模型框架为用于基于机器学习算法训练模型的框架,选取模块930可以从与用户的需求相匹配的任务类型所对应的模型框架中选取模型框架,并且/或者,选取模块930可以从之前训练好的模型中选取与用户的预测目标相匹配的模型。
如图9所示,装置900还可以可选地包括图中虚线框所示的设定模块950。设定模块950用于预先设定一种或多种任务类型,并且针对每种所述任务类型,设置一个或多个模型框架。
作为本公开的一个示例,所述之前训练好的模型可以是基于相应模型框架进行模型训练得到的。选取模块930可以先从之前训练好的模型中搜索适用于用户的预测目标的模型,然后再从搜索出的模型中进一步选取基于与用户的需求相匹配的任务类型所对应的模型框架训练得到的模型。或者,选取模块930也可以先从之前训练好的模型中选取基于与用户 的需求相匹配的任务类型所对应的模型框架训练得到的模型,然后再从选取出的模型中进一步搜索适用于用户的预测目标的模型。
训练模块940用于基于选取到的模型框架和/或模型,利用数据及其标注结果来进行模型训练。在本公开中,训练模块940可以基于选取的模型框架,利用数据及其标注结果进行模型训练,或者训练模块940也可以利用数据及其标注结果来更新选取到的模型,或者训练模块940也可以在选取到与用户的预测目标相匹配的模型的情况下,利用数据及其标注结果来更新选取到的模型,并且在没有选取到与用户的预测目标相匹配的模型的情况下,基于选取的模型框架,利用数据及其标注结果进行模型训练。
如图9所示,装置900还可以可选地包括图中虚线框所示的存储模块970和/或保存模块975。存储模块970用于将所述数据及其标注结果存储在与所述用户对应的用户数据库中。保存模块975用于保存训练得到的模型。其中,训练得到的模型和/或用户数据库的对外使用权限与用户的设置有关。作为示例,在用户数据库设置为对外开放的情况下,用户数据库中的数据可被其他用户使用,并且/或者,在模型设置为对外开放的情况下,模型可被其他用户使用。
如图9所示,装置900还可以可选地包括图中虚线框所示的接口生成模块980。接口生成模块980用于在模型训练完成后,响应于用户的模型应用请求,生成面向用户的应用程序编程接口,以便用户通过应用程序编程接口获取模型提供的预测服务。
如图9所示,装置900还可以可选地包括图中虚线框所示的反馈信息接收模块985和模型更新模块990。反馈信息接收模块985用于接收用户针对预测服务的反馈信息,模型更新模块990用于根据反馈信息,对模型进行更新。作为本公开的一个示例,反馈信息可以包括对预测服务所提供的预测标注进行纠正的更新标注,其中,模型更新模块990可以基于预测标注所对应的数据及其更新标注来形成新的训练样本,并利用新的训练样本来更新模型。作为本公开的另一个示例,反馈信息也可以包括对预测服务所提供的预测标注进行否决的拒绝信息,其中,模型更新模块990可以重新获取被否决的预测标注所对应的数据的标注结果,基于数据及其重新获取的标注结果来形成新的训练样本,并利用新的训练样本来更新模型。
如图9所示,装置900还可以可选地包括图中虚线框所示的解释模块995。解释模块995用于在所述用户使用所述模型对输入进行预测,以得到输出后,对所述输入中不同输入部分对所述输出的影响进行解释。
如图12所示,解释模块995可以可选地包括划分模块9951、变换处理模块9953、运算模块9955、影响确定模块9957以及告知模块9959。
划分模块9951用于将输入划分成多个输入部分。变换处理模块9953用于针对每个输入部分,在保持其它输入部分不变的情况下,仅对该输入部分进行变换处理,以得到新输入。运算模块9955用于将每个新输入重新输入模型进行预算,以得到模型针对新输入得到的新输出。影响确定模块9957用于根据新输出与输出之间的差异,确定不同的输入部分对输出的影响。告知模块9959用于将不同输入部分对输出的影响,以可理解的形式告知用户。
如图9所示,装置900还可以可选地包括图中虚线框所示的调整模块997。调整模块997用于根据用户通过所述应用程序编程接口发起的预测请求量来动态地调整利用模型提供预测服务所使用的资源。
图13示出了根据本公开一实施例可用于实现上述方法的数据处理的计算设备的结构示意图。
参见图13,计算设备1200包括存储器1210和处理器1220。
处理器1220可以是一个多核的处理器,也可以包含多个处理器。在一些实施例中,处理器1220可以包含一个通用的主处理器以及一个或多个特殊的协处理器,例如图形处理器(GPU)、数字信号处理器(DSP)等等。在一些实施例中,处理器1220可以使用定制的电路实现,例如特定用途集成电路(ASIC,Application Specific Integrated Circuit)或者现场可编程逻辑门阵列(FPGA,Field Programmable Gate Arrays)。
存储器1210可以包括各种类型的存储单元,例如***内存、只读存储器(ROM), 和永久存储装置。其中,ROM可以存储处理器1220或者计算机的其他模块需要的静态数据或者指令。永久存储装置可以是可读写的存储装置。永久存储装置可以是即使计算机断电后也不会失去存储的指令和数据的非易失性存储设备。在一些实施方式中,永久性存储装置采用大容量存储装置(例如磁或光盘、闪存)作为永久存储装置。另外一些实施方式中,永久性存储装置可以是可移除的存储设备(例如软盘、光驱)。***内存可以是可读写存储设备或者易失性可读写存储设备,例如动态随机访问内存。***内存可以存储一些或者所有处理器在运行时需要的指令和数据。此外,存储器1210可以包括任意计算机可读存储媒介的组合,包括各种类型的半导体存储芯片(DRAM,SRAM,SDRAM,闪存,可编程只读存储器),磁盘和/或光盘也可以采用。在一些实施方式中,存储器1210可以包括可读和/或写的可移除的存储设备,例如激光唱片(CD)、只读数字多功能光盘(例如DVD-ROM,双层DVD-ROM)、只读蓝光光盘、超密度光盘、闪存卡(例如SD卡、min SD卡、Micro-SD卡等等)、磁性软盘等等。计算机可读存储媒介不包含载波和通过无线或有线传输的瞬间电子信号。
存储器1210上存储有可处理代码,当可处理代码被处理器1220处理时,可以使处理器1220执行上文述及的方法。
作为示例,所述处理器可实现为计算装置,而存储器可实现为至少一个存储指令的存储装置,相应地,所述计算设备可实现为包括至少一个计算装置和至少一个存储指令的存储装置的***,其中,所述指令在被所述至少一个计算装置运行时,促使所述至少一个计算装置执行用于执行机器学习过程的相关步骤。
上文中已经参考附图详细描述了根据本公开的执行机器学习过程的方法、装置以及计算设备。
此外,根据本公开的方法还可以实现为一种计算机程序或计算机程序产品,该计算机程序或计算机程序产品包括用于执行本公开的上述方法中限定的上述各步骤的计算机程序代码指令。例如,可实现为一种存储指令的计算机可读存储介质,其中,当所述指令被至少一个计算装置运行时,促使所述至少一个计算装置执行本公开的上述方法中限定的上述各步骤。
或者,本公开还可以实施为一种非暂时性机器可读存储介质(或计算机可读存储介质、或机器可读存储介质),其上存储有可执行代码(或计算机程序、或计算机指令代码),当可执行代码(或计算机程序、或计算机指令代码)被电子设备(或计算设备、服务器等)的处理器执行时,使处理器执行根据本公开的上述方法的各个步骤。
本领域技术人员还将明白的是,结合这里的公开所描述的各种示例性逻辑块、模块、电路和算法步骤可以被实现为电子硬件、计算机软件或两者的组合。
附图中的流程图和框图显示了根据本公开的多个实施例的***和方法的可能实现的体系架构、功能和操作。在这点上,流程图或框图中的每个方框可以代表一个模块、程序段或代码的一部分,模块、程序段或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意,在有些作为替换的实现中,方框中所标记的功能也可以以不同于附图中所标记的顺序发生。例如,两个连续的方框实际上可以基本并行地执行,它们有时也可以按相反的顺序执行,这依所涉及的功能而定。也要注意的是,框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合,可以用执行规定的功能或操作的专用的基于硬件的***来实现,或者可以用专用硬件与计算机指令的组合来实现。
以上已经描述了本公开的各实施例,上述说明是示例性的,并非穷尽性的,并且也不限于所披露的各实施例。在不偏离所说明的各实施例的范围和精神的情况下,对于本技术领域的普通技术人员来说许多修改和变更都是显而易见的。例如,对于本公开的技术方案而言,可能会在不同场合下提到相应的机器学习模型,这些模型可能相同、相似或不同,对于各个模型的含义和限定,本领域技术人员可根据说明书公开的内容来了解。本文中所用术语的选择,旨在最好地解释各实施例的原理、实际应用或对市场中的技术的改进,或者使本技术领域的其它普通技术人员能理解本文披露的各实施例。
Claims (40)
- 一种由至少一个计算装置执行机器学习过程的方法,包括:获取数据;获取数据的标注结果;以及选取与用户的需求相匹配的模型框架和与用户的预测目标相匹配的模型之中的至少一个,基于选取到的模型框架和模型之中的所述至少一个,利用所述数据及其标注结果来进行模型训练,其中,所述模型框架为用于基于机器学习算法训练模型的框架。
- 根据权利要求1所述的方法,其中,所述选取与用户的需求相匹配的模型框架和与用户的预测目标相匹配的模型之中的至少一个的步骤包括以下步骤中的至少一个:从与所述用户的需求相匹配的任务类型所对应的模型框架中选取模型框架;以及从之前训练好的模型中选取与用户的预测目标相匹配的模型。
- 根据权利要求2所述的方法,还包括:预先设定一种或多种任务类型,并且针对每种所述任务类型,设置一个或多个模型框架。
- 根据权利要求2所述的方法,其中,所述之前训练好的模型是基于相应模型框架进行模型训练得到的,所述从之前训练好的模型中选取与用户的预测目标相匹配的模型的步骤包括:从之前训练好的模型中搜索适用于所述用户的预测目标的模型,并且从搜索出的模型中进一步选取基于与所述用户的需求相匹配的任务类型所对应的模型框架训练得到的模型;或者从之前训练好的模型中选取基于与所述用户的需求相匹配的任务类型所对应的模型框架训练得到的模型,并且从选取出的模型中进一步搜索适用于所述用户的预测目标的模型。
- 根据权利要求1所述的方法,其中,所述利用所述数据及其标注结果来进行模型训练的步骤包括:基于选取的模型框架,利用所述数据及其标注结果进行模型训练;或者利用所述数据及其标注结果来更新选取到的模型;或者在选取到与用户的预测目标相匹配的模型的情况下,利用所述数据及其标注结果来更新选取到的模型,并且在没有选取到与用户的预测目标相匹配的模型的情况下,基于选取的模型框架,利用所述数据及其标注结果进行模型训练。
- 根据权利要求1所述的方法,其中,所述获取数据的步骤包括:获取用户的数据采集需求;对所述数据采集需求进行解析,以确定适于采集的数据所具有的关键字;以及采集具有所述关键字的数据。
- 根据权利要求1所述的方法,其中,所述获取数据的标注结果的步骤包括:根据模型训练的预测目标,发布与预测目标相应的标注任务,以获取数据的标注结果。
- 根据权利要求7所述的方法,其中,所述获取数据的标注结果的步骤还包括:向标注者展示待标注对象;获取用于提示待标注对象的标注结论的辅助提示信息;以及向标注者提供所述辅助提示信息,以便所述标注者基于所述辅助提示信息在待标注对象中执行人为标注。
- 根据权利要求1所述的方法,还包括:将所述数据及其标注结果存储在与所述用户对应的用户数据库中,其中,用户数据库的对外使用权限与所述用户的设置有关。
- 根据权利要求9所述的方法,其中,在所述用户数据库设置为对外开放的情况下,所述用户数据库中的数据可被其他用户使用。
- 根据权利要求1所述的方法,还包括:保存训练得到的模型,其中,训练得到的模型的对外使用权限与所述用户的设置有关。
- 根据权利要求11所述的方法,其中,在所述模型设置为对外开放的情况下,所述模型可被其他用户使用。
- 根据权利要求1所述的方法,还包括:在模型训练完成后,响应于用户的模型应用请求,生成面向用户的应用程序编程接口,以便用户通过所述应用程序编程接口获取模型提供的预测服务。
- 根据权利要求13所述的方法,还包括:接收用户针对预测服务的反馈信息;以及根据所述反馈信息,对所述模型进行更新。
- 根据权利要求14所述的方法,其中,所述反馈信息包括对预测服务所提供的预测标注进行纠正的更新标注,其中,对所述模型进行更新的步骤包括:基于预测标注所对应的数据及其更新标注来形成新的训练样本,并利用所述新的训练样本来更新模型。
- 根据权利要求14所述的方法,其中,所述反馈信息包括对预测服务所提供的预测标注进行否决的拒绝信息,其中,对所述模型进行更新的步骤包括:重新获取被否决的预测标注所对应的数据的标注结果,基于所述数据及其重新获取的标注结果来形成新的训练样本,并利用所述新的训练样本来更新模型。
- 根据权利要求13所述的方法,还包括:在所述用户使用所述模型对输入进行预测,以得到输出后,对所述输入中不同输入部分对所述输出的影响进行解释。
- 根据权利要求17所述的方法,其中,所述对所述输入中不同输入部分对所述输出的影响进行解释的步骤包括:将所述输入划分成多个输入部分;针对每个所述输入部分,在保持其它输入部分不变的情况下,仅对该输入部分进行变换处理,以得到新输入;将每个新输入重新输入所述模型,以得到所述模型针对所述新输入得到的新输出;根据所述新输出与所述输出之间的差异,确定不同的输入部分对所述输出的影响;以及将不同输入部分对所述输出的影响,以可理解的形式告知用户。
- 根据权利要求13所述的方法,还包括:根据用户通过所述应用程序编程接口发起的预测请求量来动态地调整利用模型提供预测服务所使用的资源。
- 一种包括至少一个计算装置和至少一个存储指令的存储装置的***,其中,所述指令在被所述至少一个计算装置运行时,促使所述至少一个计算装置执行用于执行机器学习过程的以下步骤:获取数据;获取数据的标注结果;选取与用户的需求相匹配的模型框架和与用户的预测目标相匹配的模型之中的至少一个,其中,所述模型框架为用于基于机器学习算法训练模型的框架;以及基于选取到的模型框架和模型之中的所述至少一个,利用所述数据及其标注结果来进行模型训练。
- 根据权利要求20所述的装置,其中,所述选取与用户的需求相匹配的模型框架和与用户的预测目标相匹配的模型之中的至少一个的步骤包括以下步骤中的至少一个:从与所述用户的需求相匹配的任务类型所对应的模型框架中选取模型框架;以及从之前训练好的模型中选取与用户的预测目标相匹配的模型。
- 根据权利要求21所述的装置,其中,所述指令在被所述至少一个计算装置运行时,将促使所述至少一个计算装置还执行以下步骤:预先设定一种或多种任务类型,并且针对每种所述任务类型,设置一个或多个模型框架。
- 根据权利要求21所述的装置,其中,所述之前训练好的模型是基于相应模型框架进行模型训练得到的,所述从之前训练好的模型中选取与用户的预测目标相匹配的模型的步骤包括:从之前训练好的模型中搜索适用于所述用户的预测目标的模型,并且从搜索出的模型中进一步选取基于与所述用户的需求相匹配的任务类型所对应的模型框架训练得到的模型,或者从之前训练好的模型中选取基于与所述用户的需求相匹配的任务类型所对应的模型框架训练得到的模型,并且从选取出的模型中进一步搜索适用于所述用户的预测目标的模型。
- 根据权利要求20所述的装置,其中,所述利用所述数据及其标注结果来进行模型训练的步骤包括:所述训练模块基于选取的模型框架,利用所述数据及其标注结果进行模型训练,或者所述训练模块利用所述数据及其标注结果来更新选取到的模型,或者所述训练模块在选取到与用户的预测目标相匹配的模型的情况下,利用所述数据及其标注结果来更新选取到的模型,并且在没有选取到与用户的预测目标相匹配的模型的情况下,基于选取的模型框架,利用所述数据及其标注结果进行模型训练。
- 根据权利要求20所述的装置,其中,所述获取数据的步骤包括:获取用户的数据采集需求;对所述数据采集需求进行解析,以确定适于采集的数据所具有的关键字;以及采集具有所述关键字的数据。
- 根据权利要求20所述的装置,其中,所述获取数据的标注结果的步骤包括:根据模型训练的预测目标,发布与预测目标相应的标注任务,以获取数据的标注结果。
- 根据权利要求26所述的装置,其中,所述获取数据的标注结果的步骤还包括:向标注者展示待标注对象;用于提示待标注对象的标注结论的辅助提示信息;以及向标注者提供所述辅助提示信息,以便所述标注者基于所述辅助提示信息在待标注对象中执行人为标注。
- 根据权利要求20所述的装置,其中,所述指令在被所述至少一个计算装置运行时,将促使所述至少一个计算装置还执行以下步骤:将所述数据及其标注结果存储在与所述用户对应的用户数据库中,其中,用户数据库的对外使用权限与所述用户的设置有关。
- 根据权利要求28所述的装置,其中,在所述用户数据库设置为对外开放的情况下,所述用户数据库中的数据可被其他用户使用。
- 根据权利要求20所述的装置,其中,所述指令在被所述至少一个计算装置运行时,将促使所述至少一个计算装置还执行以下步骤:保存训练得到的模型,其中,训练得到的模型的对外使用权限与所述用户的设置有关。
- 根据权利要求30所述的装置,其中,在所述模型设置为对外开放的情况下,所述模型可被其他用户使用。
- 根据权利要求20所述的装置,其中,所述指令在被所述至少一个计算装置运行时,将促使所述至少一个计算装置还执行以下步骤:在模型训练完成后,响应于用户的模型应用请求,生成面向用户的应用程序编程接口,以便用户通过所述应用程序编程接口获取模型提供的预测服务。
- 根据权利要求32所述的装置,其中,所述指令在被所述至少一个计算装置运行时,将促使所述至少一个计算装置还执行以下步骤:接收用户针对预测服务的反馈信息;以及根据所述反馈信息,对所述模型进行更新。
- 根据权利要求33所述的装置,其中,所述反馈信息包括对预测服务所提供的预测标注进行纠正的更新标注,其中,对所述模型进行更新的步骤包括:基于预测标注所对应的数据及其更新标注来形成新的训练样本,并利用所述新的训练样本来更新模型。
- 根据权利要求33所述的装置,其中,所述反馈信息包括对预测服务所提供的预测标注进行否决的拒绝信息,其中,对所述模型进行更新的步骤包括:重新获取被否决的预测标注所对应的数据的标注结果,基于所述数据及其重新获取的标注结果来形成新的训练样本,并利用所述新的训练样本来更新模型。
- 根据权利要求32所述的装置,其中,所述指令在被所述至少一个计算装置运行时,将促使所述至少一个计算装置还执行以下步骤:在所述用户使用所述模型对输入进行预测,以得到输出后,对所述输入中不同输入部分对所述输出的影响进行解释。
- 根据权利要求36所述的装置,其中,所述对所述输入中不同输入部分对所述输出的影响进行解释的步骤包括:将所述输入划分成多个输入部分;针对每个所述输入部分,在保持其它输入部分不变的情况下,仅对该输入部分进行变换处理,以得到新输入;将每个新输入重新输入所述模型进行预算,以得到所述模型针对所述新输入得到的新输出;根据所述新输出与所述输出之间的差异,确定不同的输入部分对所述输出的影响;以及将不同输入部分对所述输出的影响,以可理解的形式告知用户。
- 根据权利要求32所述的装置,其中,所述指令在被所述至少一个计算装置运行时,将促使所述至少一个计算装置还执行以下步骤:根据用户通过所述应用程序编程接口发起的预测请求量来动态地调整利用模型提供预测服务所使用的资源。
- 一种执行机器学习过程的装置,包括:数据获取模块,用于获取数据;标注结果获取模块,用于获取数据的标注结果;选取模块,用于选取与用户的需求相匹配的模型框架和与用户的预测目标相匹配的模型之中的至少一个,其中,模型框架为用于基于机器学习算法训练模型的框架;以及训练模块,用于基于选取到的模型框架和模型之中的所述至少一个,利用数据及其标注结果来进行模型训练。
- 一种存储指令的计算机可读存储介质,其中,当所述指令被至少一个计算装置运行时,促使所述至少一个计算装置执行如权利要求1至19中任一项所述的方法。
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP19831144.1A EP3819828A4 (en) | 2018-07-05 | 2019-07-02 | METHOD AND APPARATUS FOR IMPLEMENTING MACHINE LEARNING METHOD, DEVICE AND RECORDING MEDIUM |
SG11202100004XA SG11202100004XA (en) | 2018-07-05 | 2019-07-02 | Machine learning process implementation method and apparatus, device, and storage medium |
US17/257,897 US20210271809A1 (en) | 2018-07-05 | 2019-07-02 | Machine learning process implementation method and apparatus, device, and storage medium |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201810730414.8 | 2018-07-05 | ||
CN201810730414.8A CN110210624A (zh) | 2018-07-05 | 2018-07-05 | 执行机器学习过程的方法、装置、设备以及存储介质 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2020007287A1 true WO2020007287A1 (zh) | 2020-01-09 |
Family
ID=67779781
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2019/094363 WO2020007287A1 (zh) | 2018-07-05 | 2019-07-02 | 执行机器学习过程的方法、装置、设备以及存储介质 |
Country Status (5)
Country | Link |
---|---|
US (1) | US20210271809A1 (zh) |
EP (1) | EP3819828A4 (zh) |
CN (1) | CN110210624A (zh) |
SG (1) | SG11202100004XA (zh) |
WO (1) | WO2020007287A1 (zh) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20210192394A1 (en) * | 2019-12-19 | 2021-06-24 | Alegion, Inc. | Self-optimizing labeling platform |
CN113836443A (zh) * | 2021-09-28 | 2021-12-24 | 土巴兔集团股份有限公司 | 一种文章审核方法及其相关设备 |
Families Citing this family (33)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11797902B2 (en) * | 2018-11-16 | 2023-10-24 | Accenture Global Solutions Limited | Processing data utilizing a corpus |
WO2020196385A1 (ja) * | 2019-03-25 | 2020-10-01 | 富士フイルム株式会社 | 画像処理装置、画像処理方法及びプログラム、撮影装置 |
CN110634141B (zh) * | 2019-09-19 | 2022-02-11 | 南京邮电大学 | 基于改进直觉模糊c均值聚类的图像分割方法及存储介质 |
CN112632179A (zh) * | 2019-09-24 | 2021-04-09 | 北京国双科技有限公司 | 模型构建方法、装置、存储介质及设备 |
CN112580912A (zh) * | 2019-09-30 | 2021-03-30 | 北京国双科技有限公司 | 预算审核方法、装置、电子设备和存储介质 |
KR20210050129A (ko) * | 2019-10-28 | 2021-05-07 | 삼성에스디에스 주식회사 | 객체 탐지를 위한 기계 학습 장치 및 방법 |
CN110991649A (zh) * | 2019-10-28 | 2020-04-10 | 中国电子产品可靠性与环境试验研究所((工业和信息化部电子第五研究所)(中国赛宝实验室)) | 深度学习模型搭建方法、装置、设备和存储介质 |
KR102176458B1 (ko) * | 2019-11-18 | 2020-11-09 | 셀렉트스타 주식회사 | 데이터 라벨링을 위한 바운딩 박스 그리기 방법 및 장치 |
CN111324732B (zh) * | 2020-01-21 | 2024-04-02 | 中信百信银行股份有限公司 | 模型训练方法、文本处理方法、装置及电子设备 |
CN113496232B (zh) * | 2020-03-18 | 2024-05-28 | 杭州海康威视数字技术股份有限公司 | 标签校验方法和设备 |
US11941496B2 (en) * | 2020-03-19 | 2024-03-26 | International Business Machines Corporation | Providing predictions based on a prediction accuracy model using machine learning |
JP2021149844A (ja) * | 2020-03-23 | 2021-09-27 | 富士フイルムビジネスイノベーション株式会社 | 情報処理装置及び情報処理プログラム |
CN111523422B (zh) * | 2020-04-15 | 2023-10-10 | 北京华捷艾米科技有限公司 | 一种关键点检测模型训练方法、关键点检测方法和装置 |
CN111369011A (zh) * | 2020-04-16 | 2020-07-03 | 光际科技(上海)有限公司 | 机器学习模型应用的方法、装置、计算机设备和存储介质 |
US20210342736A1 (en) * | 2020-04-30 | 2021-11-04 | UiPath, Inc. | Machine learning model retraining pipeline for robotic process automation |
CN112036441A (zh) * | 2020-07-31 | 2020-12-04 | 上海图森未来人工智能科技有限公司 | 机器学习物体检测结果的反馈标注方法和装置、存储介质 |
CN114067350A (zh) * | 2020-07-31 | 2022-02-18 | 伊姆西Ip控股有限责任公司 | 模型更新方法、设备和计算机程序产品 |
TWI787669B (zh) * | 2020-11-16 | 2022-12-21 | 國立陽明交通大學 | 基於模型處方的自動機器學習之系統與方法 |
CN114577481B (zh) * | 2020-12-02 | 2024-01-12 | 新奥新智科技有限公司 | 燃气内燃机的污染指标监测方法及装置 |
CN112419077A (zh) * | 2020-12-04 | 2021-02-26 | 上海商汤智能科技有限公司 | 数据处理方法及装置、电子设备和存储介质 |
US11841925B1 (en) * | 2020-12-10 | 2023-12-12 | Amazon Technologies, Inc. | Enabling automatic classification for multi-label classification problems with label completion guarantees |
CN112733454B (zh) * | 2021-01-13 | 2024-04-30 | 新奥新智科技有限公司 | 一种基于联合学习的设备预测性维护方法及装置 |
CN112508723B (zh) * | 2021-02-05 | 2024-02-02 | 北京淇瑀信息科技有限公司 | 基于自动择优建模的金融风险预测方法、装置和电子设备 |
CN112988727B (zh) * | 2021-03-25 | 2022-09-16 | 北京百度网讯科技有限公司 | 数据标注方法、装置、设备、存储介质及计算机程序产品 |
US20220343153A1 (en) * | 2021-04-26 | 2022-10-27 | Micron Technology, Inc. | Artificial neural network retraining in memory |
CN113221564B (zh) * | 2021-04-29 | 2024-03-01 | 北京百度网讯科技有限公司 | 训练实体识别模型的方法、装置、电子设备和存储介质 |
CN113392263A (zh) * | 2021-06-24 | 2021-09-14 | 上海商汤科技开发有限公司 | 一种数据标注方法及装置、电子设备和存储介质 |
DE102021116779A1 (de) | 2021-06-30 | 2023-01-05 | Bayerische Motoren Werke Aktiengesellschaft | Verfahren zum Bereitstellen eines prädizierten, aktuellen Fahrziels an einen Nutzer eines Fahrzeugs, computerlesbares Medium, System, Fahrzeug, und mobiles Endgerät |
CN114219971A (zh) * | 2021-12-13 | 2022-03-22 | 腾讯科技(深圳)有限公司 | 一种数据处理方法、设备以及计算机可读存储介质 |
CN114428677B (zh) * | 2022-01-28 | 2023-09-12 | 北京百度网讯科技有限公司 | 任务处理方法、处理装置、电子设备及存储介质 |
CN114118449B (zh) * | 2022-01-28 | 2022-10-04 | 深圳佑驾创新科技有限公司 | 基于偏标记学习模型的图片标签识别方法、介质及设备 |
CN114245206B (zh) * | 2022-02-23 | 2022-07-15 | 阿里巴巴达摩院(杭州)科技有限公司 | 视频处理方法及装置 |
CN114911813B (zh) * | 2022-06-27 | 2023-09-26 | 芯砺智能科技(上海)有限公司 | 车载感知模型的更新方法、装置、电子设备及存储介质 |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20150054824A1 (en) * | 2013-08-21 | 2015-02-26 | Canon Kabushiki Kaisha | Object detection method, object detection device, and image pickup device |
CN106779166A (zh) * | 2016-11-23 | 2017-05-31 | 北京师范大学 | 一种基于数据驱动的知识点掌握状态的预测***及方法 |
CN106779079A (zh) * | 2016-11-23 | 2017-05-31 | 北京师范大学 | 一种基于多模型数据驱动的知识点掌握状态的预测***及方法 |
CN106909931A (zh) * | 2015-12-23 | 2017-06-30 | 阿里巴巴集团控股有限公司 | 一种用于机器学习模型的特征生成方法、装置和电子设备 |
CN107247972A (zh) * | 2017-06-29 | 2017-10-13 | 哈尔滨工程大学 | 一种基于众包技术的分类模型训练方法 |
CN107316007A (zh) * | 2017-06-07 | 2017-11-03 | 浙江捷尚视觉科技股份有限公司 | 一种基于深度学习的监控图像多类物体检测与识别方法 |
Family Cites Families (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7853539B2 (en) * | 2005-09-28 | 2010-12-14 | Honda Motor Co., Ltd. | Discriminating speech and non-speech with regularized least squares |
US20160358099A1 (en) * | 2015-06-04 | 2016-12-08 | The Boeing Company | Advanced analytical infrastructure for machine learning |
CN105550746B (zh) * | 2015-12-08 | 2018-02-02 | 北京旷视科技有限公司 | 机器学习模型的训练方法和训练装置 |
US11080616B2 (en) * | 2016-09-27 | 2021-08-03 | Clarifai, Inc. | Artificial intelligence model and data collection/development platform |
CN108229686B (zh) * | 2016-12-14 | 2022-07-05 | 阿里巴巴集团控股有限公司 | 模型训练、预测方法、装置、电子设备及机器学习平台 |
CN107273492B (zh) * | 2017-06-15 | 2021-07-23 | 复旦大学 | 一种基于众包平台处理图像标注任务的交互方法 |
US11301684B1 (en) * | 2017-09-29 | 2022-04-12 | Amazon Technologies, Inc. | Vision-based event detection |
CN108197664B (zh) * | 2018-01-24 | 2020-09-04 | 北京墨丘科技有限公司 | 模型获取方法、装置、电子设备及计算机可读存储介质 |
US10877927B2 (en) * | 2018-05-31 | 2020-12-29 | Microsofttechnology Licensing, Llc | Distributed computing system with a synthetic data as a service asset assembly engine |
US11120364B1 (en) * | 2018-06-14 | 2021-09-14 | Amazon Technologies, Inc. | Artificial intelligence system with customizable training progress visualization and automated recommendations for rapid interactive development of machine learning models |
-
2018
- 2018-07-05 CN CN201810730414.8A patent/CN110210624A/zh active Pending
-
2019
- 2019-07-02 EP EP19831144.1A patent/EP3819828A4/en active Pending
- 2019-07-02 US US17/257,897 patent/US20210271809A1/en active Pending
- 2019-07-02 SG SG11202100004XA patent/SG11202100004XA/en unknown
- 2019-07-02 WO PCT/CN2019/094363 patent/WO2020007287A1/zh active Application Filing
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20150054824A1 (en) * | 2013-08-21 | 2015-02-26 | Canon Kabushiki Kaisha | Object detection method, object detection device, and image pickup device |
CN106909931A (zh) * | 2015-12-23 | 2017-06-30 | 阿里巴巴集团控股有限公司 | 一种用于机器学习模型的特征生成方法、装置和电子设备 |
CN106779166A (zh) * | 2016-11-23 | 2017-05-31 | 北京师范大学 | 一种基于数据驱动的知识点掌握状态的预测***及方法 |
CN106779079A (zh) * | 2016-11-23 | 2017-05-31 | 北京师范大学 | 一种基于多模型数据驱动的知识点掌握状态的预测***及方法 |
CN107316007A (zh) * | 2017-06-07 | 2017-11-03 | 浙江捷尚视觉科技股份有限公司 | 一种基于深度学习的监控图像多类物体检测与识别方法 |
CN107247972A (zh) * | 2017-06-29 | 2017-10-13 | 哈尔滨工程大学 | 一种基于众包技术的分类模型训练方法 |
Non-Patent Citations (1)
Title |
---|
See also references of EP3819828A4 * |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20210192394A1 (en) * | 2019-12-19 | 2021-06-24 | Alegion, Inc. | Self-optimizing labeling platform |
CN113836443A (zh) * | 2021-09-28 | 2021-12-24 | 土巴兔集团股份有限公司 | 一种文章审核方法及其相关设备 |
Also Published As
Publication number | Publication date |
---|---|
EP3819828A4 (en) | 2022-03-30 |
CN110210624A (zh) | 2019-09-06 |
US20210271809A1 (en) | 2021-09-02 |
EP3819828A1 (en) | 2021-05-12 |
SG11202100004XA (en) | 2021-02-25 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2020007287A1 (zh) | 执行机器学习过程的方法、装置、设备以及存储介质 | |
Sener et al. | Temporal aggregate representations for long-range video understanding | |
JP6844301B2 (ja) | 予測分析のための時系列のデータ・セットを生成する方法及びデータ処理装置 | |
Issa et al. | Research ideas for artificial intelligence in auditing: The formalization of audit and workforce supplementation | |
Li et al. | Tobler’s First Law in GeoAI: A spatially explicit deep learning model for terrain feature detection under weak supervision | |
CN110909820A (zh) | 基于自监督学习的图像分类方法及*** | |
US11887215B2 (en) | Image processing apparatus and method for style transformation | |
CN108550065B (zh) | 评论数据处理方法、装置及设备 | |
CN113688851B (zh) | 数据标注方法和装置和精细粒度识别方法和装置 | |
US11532036B2 (en) | Digital image ordering using object position and aesthetics | |
WO2024051609A1 (zh) | 广告创意数据选取方法及装置、模型训练方法及装置、设备、存储介质 | |
Ghosh et al. | Automated detection and classification of pavement distresses using 3D pavement surface images and deep learning | |
US11625612B2 (en) | Systems and methods for domain adaptation | |
Grechikhin et al. | User modeling on mobile device based on facial clustering and object detection in photos and videos | |
US20200410298A1 (en) | Automatic metadata detector based on images | |
CN114119136A (zh) | 一种产品推荐方法、装置、电子设备和介质 | |
Takimoto et al. | Image aesthetics assessment based on multi-stream CNN architecture and saliency features | |
Lin et al. | An analysis of English classroom behavior by intelligent image recognition in IoT | |
Kharchevnikova et al. | Efficient video face recognition based on frame selection and quality assessment | |
Mar et al. | Cow detection and tracking system utilizing multi-feature tracking algorithm | |
Heidari et al. | Forest roads damage detection based on deep learning algorithms | |
CN116681961A (zh) | 基于半监督方法和噪声处理的弱监督目标检测方法 | |
US11615618B2 (en) | Automatic image annotations | |
CN112507912B (zh) | 一种识别违规图片的方法及装置 | |
US11715023B2 (en) | Methods and apparatuses for training one or more model parameters of a predictive parking difficulty model |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 19831144 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2019831144 Country of ref document: EP |