CN109460396B - Model processing method and device, storage medium and electronic equipment - Google Patents
Model processing method and device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN109460396B CN109460396B CN201811191482.8A CN201811191482A CN109460396B CN 109460396 B CN109460396 B CN 109460396B CN 201811191482 A CN201811191482 A CN 201811191482A CN 109460396 B CN109460396 B CN 109460396B
- Authority
- CN
- China
- Prior art keywords
- model
- training
- information
- configuration
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 34
- 238000012549 training Methods 0.000 claims abstract description 115
- 238000012545 processing Methods 0.000 claims abstract description 51
- 238000004458 analytical method Methods 0.000 claims abstract description 30
- 238000000034 method Methods 0.000 claims abstract description 27
- 238000012795 verification Methods 0.000 claims description 18
- 230000008569 process Effects 0.000 claims description 12
- 230000002159 abnormal effect Effects 0.000 claims description 10
- 238000012360 testing method Methods 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 3
- 238000012856 packing Methods 0.000 claims 1
- 238000010586 diagram Methods 0.000 description 11
- 238000010276 construction Methods 0.000 description 5
- 230000005856 abnormality Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000006978 adaptation Effects 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000003066 decision tree Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000001965 increasing effect Effects 0.000 description 2
- 238000003062 neural network model Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000012706 support-vector machine Methods 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000007477 logistic regression Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a model processing method and device, a storage medium and electronic equipment, which are applied to a stand-alone environment and relate to the technical field of computers. The model processing device comprises: the configuration acquisition module is used for acquiring a configuration file of the model; the information analysis module is used for analyzing the configuration file to obtain configuration information; the model training module is used for constructing a model according to the model parameters in the configuration information and acquiring a training set according to the training set path in the configuration information so as to train the model; and the database is used for storing the configuration information and training results of the model. The method and the device can better realize the management of the model in a single machine environment.
Description
Technical Field
The present disclosure relates to the field of computer technology, and in particular, to a model processing method, a model processing apparatus, a storage medium, and an electronic device.
Background
With the development of computer technology, industries may utilize models to analyze data and solve various prediction problems. Establishment of the model idea greatly improves the efficiency of processing the problem and reduces the cost of manual repeated participation.
The process of modeling and analyzing data is often performed in an online state. Currently, in a stand-alone environment, there is no unified model parameter and version management. On the one hand, when the model has problems, the model is difficult to trace back; on the other hand, because there is no better management mechanism, the construction of the current model cannot be realized by efficiently utilizing the historical model parameters; on the other hand, there is no visual comparison method between model effects of different model versions, and a user cannot clearly view a comparison result of model prediction effects.
It should be noted that the information disclosed in the above background section is only for enhancing understanding of the background of the present disclosure and thus may include information that does not constitute prior art known to those of ordinary skill in the art.
Disclosure of Invention
The disclosure aims to provide a model processing method, a model processing device, a storage medium and an electronic device, so as to realize model management in a single machine environment at least to a certain extent, and realize model backtracking and model comparison.
According to an aspect of the present disclosure, there is provided a model processing apparatus, applied in a stand-alone environment, including: the configuration acquisition module is used for acquiring a configuration file of the model; the information analysis module is used for analyzing the configuration file to obtain configuration information; the model training module is used for constructing a model according to the model parameters in the configuration information and acquiring a training set according to the training set path in the configuration information so as to train the model; and the database is used for storing the configuration information and training results of the model.
In an exemplary embodiment of the present disclosure, the model processing apparatus further includes: the information verification module is used for verifying the configuration information; and the model training module is used for constructing a model according to model parameters in the configuration information when the configuration information is successfully checked, and acquiring a training set according to a training set path in the configuration information so as to train the model.
In an exemplary embodiment of the present disclosure, the model processing apparatus further includes: the data prediction module is used for obtaining data to be predicted, predicting the data to be predicted by adopting a trained model, and storing a prediction result into the database.
In an exemplary embodiment of the present disclosure, the model processing apparatus further includes: and the abnormality processing module is used for judging whether the model is abnormal in the training and/or predicting process, and executing one or more operations of initializing the model, retraining and/or predicting and sending alarm information when the model is abnormal.
In an exemplary embodiment of the present disclosure, the model processing apparatus further includes: and the analysis and comparison module is used for acquiring one or more model prediction results from the database, feeding the one or more model prediction results back to the user side so as to be analyzed and compared by the user, and modifying model parameters according to the analysis and comparison results.
In an exemplary embodiment of the present disclosure, the model processing apparatus further includes: and the time control module is used for executing operations of building a model, training the model and/or predicting data at a preset time.
In an exemplary embodiment of the present disclosure, the configuration obtaining module for obtaining a configuration file of a model includes: the configuration acquisition module is used for acquiring a configuration file and a training script which are packaged and uploaded by a user; the model training module is used for responding to a training instruction of a user, constructing a model according to model parameters in the configuration information, acquiring a training set according to a training set path in the configuration information, and training the model by utilizing the training set and executing the training script.
According to one aspect of the present disclosure, there is provided a model processing method, applied in a stand-alone environment, including: acquiring a configuration file of a model; analyzing the configuration file to obtain configuration information; constructing a model according to model parameters in the configuration information, and acquiring a training set according to a training set path in the configuration information to train the model; and storing the configuration information and the training result of the model into a database.
In an exemplary embodiment of the present disclosure, the model processing method further includes: checking the configuration information; and when the verification is successful, constructing a model according to the model parameters in the configuration information.
In an exemplary embodiment of the present disclosure, the model processing method further includes: and obtaining data to be predicted, predicting the data to be predicted by adopting a trained model, and storing a prediction result into a database.
In an exemplary embodiment of the present disclosure, the model processing method further includes: judging whether the model is abnormal in the training and/or predicting process, and executing one or more operations of initializing the model, retraining and/or predicting and sending out alarm information when the model is abnormal.
In an exemplary embodiment of the present disclosure, the model processing method further includes: and obtaining one or more model prediction results from the database, feeding back the one or more model prediction results to the user side so as to enable the user to analyze and compare, and modifying model parameters according to the analysis and comparison results.
In an exemplary embodiment of the present disclosure, the model processing method further includes: operations to build the model, train the model, and/or predict the data are performed at predetermined times.
In one exemplary embodiment of the present disclosure, obtaining the configuration file of the model includes obtaining a configuration file and a training script uploaded by a user in a package; the training method comprises the steps of responding to a training instruction of a user, constructing a model according to model parameters in configuration information, acquiring a training set according to a training set path in the configuration information, and training the model by utilizing the training set and executing a training script.
According to an aspect of the present disclosure, there is provided a storage medium having stored thereon a computer program which, when executed by a processor, implements the model processing method of any one of the above.
According to one aspect of the present disclosure, there is provided an electronic device including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the model processing method of any of the above via execution of the executable instructions.
In the technical schemes provided by some embodiments of the present disclosure, on one hand, the present disclosure better implements management of a model in a single environment by means of a configuration acquisition module, an information analysis module, a model training module and a database that are constructed in the single environment, and can implement backtracking of the model by means of configuration information and model training results stored in the database; on the other hand, the configuration information stored in the database can be modified to quickly realize the construction of a new model, so that the time is saved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure. It will be apparent to those of ordinary skill in the art that the drawings in the following description are merely examples of the disclosure and that other drawings may be derived from them without undue effort. In the drawings:
fig. 1 schematically shows a block diagram of a model processing apparatus according to a first exemplary embodiment of the present disclosure;
FIG. 2 schematically illustrates a block diagram of a model processing apparatus according to a second exemplary embodiment of the present disclosure;
FIG. 3 schematically illustrates a block diagram of a model processing apparatus according to a third exemplary embodiment of the present disclosure;
FIG. 4 schematically shows a block diagram of a model processing apparatus according to a fourth exemplary embodiment of the present disclosure;
FIG. 5 schematically shows a block diagram of a model processing apparatus according to a fifth exemplary embodiment of the present disclosure;
FIG. 6 schematically shows a block diagram of a model processing apparatus according to a sixth exemplary embodiment of the present disclosure;
FIG. 7 schematically illustrates a flow chart of a model processing method according to an exemplary embodiment of the present disclosure;
FIG. 8 illustrates a schematic diagram of a storage medium according to an exemplary embodiment of the present disclosure; and
Fig. 9 schematically illustrates a block diagram of an electronic device according to an exemplary embodiment of the present disclosure.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments may be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the present disclosure. One skilled in the relevant art will recognize, however, that the aspects of the disclosure may be practiced without one or more of the specific details, or with other methods, components, devices, steps, etc. In other instances, well-known technical solutions have not been shown or described in detail to avoid obscuring aspects of the present disclosure.
Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus a repetitive description thereof will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software or in one or more hardware modules or integrated circuits or in different networks and/or processor devices and/or microcontroller devices.
The flow diagrams depicted in the figures are exemplary only and not necessarily all steps are included. For example, some steps may be decomposed, and some steps may be combined or partially combined, so that the order of actual execution may be changed according to actual situations.
The model processing device is applied to a single machine environment. Fig. 1 schematically shows a block diagram of a model processing apparatus according to a first exemplary embodiment of the present disclosure. Referring to fig. 1, the model processing apparatus 1 may include a configuration acquisition module 11, an information parsing module 12, a model training module 13, and a database 19, wherein:
a configuration acquisition module 11, configured to acquire a configuration file of the model;
the information parsing module 12 may be configured to parse the configuration file to obtain configuration information;
the model training module 13 may be configured to construct a model according to model parameters in the configuration information, and acquire a training set according to a training set path in the configuration information to train the model;
A database 19 may be used to store the configuration information and training results of the model.
In the model processing device 1 of the exemplary embodiment of the present disclosure, on the one hand, the present disclosure better implements management of a model in a single environment through a configuration acquisition module, an information analysis module, a model training module, and a database that are constructed in the single environment, and can implement backtracking of the model through configuration information and model training results stored in the database; on the other hand, the configuration information stored in the database can be modified to quickly realize the construction of a new model, so that the time is saved.
The respective constituent parts of the model processing apparatus 1 of the present disclosure will be described below.
In an exemplary embodiment of the present disclosure, the configuration obtaining module 11 may be connected to a user terminal, and the user may input parameter information required for the model through a front end interface (e.g., a web page) of the user terminal, where the parameter information may be configured as a configuration file, and a format of the configuration file may be json format, for example. If the configuration file is noted as conf, the configuration file may include, but is not limited to, the following information:
In particular, the configuration file may contain information related to the model configuration rights, such as a user name and password; information related to model parameters, such as algorithms employed by the model, may include, but are not limited to, SVM (Support Vector Machine ), logistic regression algorithm, GBDT (Gradient Boosting Decision Tree, gradient-lifting decision tree), etc.; information related to the data used by the model, e.g., the path of the training set, the path of the test set, the path of the data to be predicted, etc. In addition, the configuration file may also include other information such as "run mode", model id, as described above.
However, it should be understood that the above description of the configuration file is merely exemplary, and the present disclosure does not impose particular limitations on the format and content of the configuration file.
After the user inputs the model configuration information, the generated configuration file may be sent to the configuration acquisition module 11 of the model processing apparatus through the user side.
In addition, the user may input model configuration information in real time. However, the configuration information may also be stored in advance in the user terminal by the user, and the user terminal transmits the configuration information to the configuration acquisition module 11 when a predetermined event occurs, wherein the predetermined event may include a predetermined time set by the user himself, when the unit for building the model is idle, and the like.
After acquiring the configuration file of the model, the configuration acquisition module 11 may send the configuration file to the information analysis module 12. The information parsing module 12 may parse the configuration file to obtain the specific configuration information. For example, the information parsing module 12 may be provided with a tool for parsing json files, by means of which the json formatted configuration files may be parsed. Specifically, the tool may be an existing parsing tool, or may be a tool that a developer develops according to an actual service requirement, which is not particularly limited in this exemplary embodiment.
After the information parsing module 12 parses the configuration file, the model training module 13 may obtain the model parameters in the parsed configuration information. Specifically, after the information analysis module 12 obtains the configuration information, the information analysis module 12 may directly send all the configuration information to the model training module 13, however, the information analysis module 12 may also send only the model parameters in the configuration information to the model training module.
The model training module 13 may construct a model according to model parameters, for example, a neural network model, where the model parameters may specifically further include information such as a size and a dimension of a convolution kernel of each convolution layer in the neural network.
After the model is built, the model training module 13 may further acquire a training set from the training set paths in the configuration information, where a process of acquiring the training set paths is similar to the process of acquiring the model parameters described above, which is not described herein. Next, the model training module 13 may train the constructed model using the acquired training set. In addition, it is easy to understand that the model training module 13 may also obtain a path of the test set from the configuration information, and obtain the test set of the model according to the path, so as to test the trained model.
It should be noted that the traffic is different, and the paths of the training set and the test set are also different. The specific storage locations of the training set and the test set are not particularly limited by the present disclosure.
Referring to fig. 1, in performing the above-described acquisition profiles, parsing of profiles, model building and/or model training tests, data received, generated, transferred by the modules may be stored in a database 19. Specifically, the database 19 may store configuration information obtained by the information parsing module 12 parsing the configuration file, and training results obtained by the model training module 13 training the constructed model. In addition, the database 19 may also store the time of model construction, directly store configuration files, and the like.
In addition, in view of the requirement that the database 19 is easy to use, has strong scalability, is easy to maintain, etc., the database 19 of the present disclosure is a mondab database. However, the database 19 may also be other types of databases.
In the model processing apparatus of the second exemplary embodiment of the present disclosure, referring to fig. 2, the model processing apparatus 2 may include an information verification module 14 in addition to the configuration acquisition module 11, the information parsing module 12, the model training module 13, and the database 19.
After the information parsing module 12 obtains the configuration information, the configuration information may be sent to the information verification module 14. The information verification module 14 may verify the configuration information, and in particular, may verify the rights of the user. For example, it may be determined whether the user is in a pre-configured white list, and if the user is in the white list, the user is instructed to comply with the rights to model and manage the model. The whitelist may be preset and may be stored, for example, in the information verification module 14 for verification purposes. In addition, the information verification module 14 may also verify whether the model id meets the format requirement, whether it is repeated with the existing model, and so on. This is not particularly limited in the present exemplary embodiment.
If the configuration information verification is successful, the model training module 13 may perform the procedure it performs as described above. In addition, configuration information may be sent by the information verification module 14 to the model training module 13. However, the configuration information may also be sent by the information parsing module 12 to the model training module 13, in which case the information verification module 14 only functions as a verification, and no information transfer.
If the configuration information verification fails, the information verification module 14 may directly send an alarm message to the user side to prompt that the configuration information is wrong, thereby prompting the user to upload the reconfiguration file.
In the model processing apparatus of the third exemplary embodiment of the present disclosure, referring to fig. 3, the model processing apparatus 3 may include a model prediction module 15 in addition to the configuration acquisition module 11, the information parsing module 12, the model training module 13, and the database 19.
The model prediction module 15 may acquire data to be predicted. The data to be predicted can be data uploaded by a user in real time, and in addition, the configuration file contains a path of the predicted data, and the model prediction module can acquire the data to be predicted according to the path.
The model prediction module 15 may obtain a trained model from the model training module 13 or the database 11 and use the model to predict the data to be predicted. After the prediction is completed, the model prediction module 15 may store the prediction result to the database 19.
In the model processing apparatus of the fourth exemplary embodiment of the present disclosure, referring to fig. 4, the model processing apparatus 4 may include an abnormality processing module 16 in addition to the configuration acquisition module 11, the information analysis module 12, the model training module 13, the database 19, and the model prediction module 15.
Exception handling module 16 may determine whether the model is abnormal during training and/or prediction, which may include a program run error to exit. When the anomaly handling module 16 determines an anomaly, one or more of initializing the model, retraining and/or predicting may be performed, issuing an alert message. Wherein, initializing the model may refer to controlling the model training module 13 to reuse model parameters for modeling; retraining may refer to controlling model training module 13 to retrain the model with the training set; re-predicting may refer to controlling the model prediction module 15 to re-predict the data to be predicted; sending the alarm information may refer to directly sending information of abnormal processing procedure to the user side and/or the developer, so as to remind the user and/or the developer of performing an error checking operation.
In addition, the exception handling module 16 may store the information that generated the error in the database 19.
In the model processing apparatus of the fifth exemplary embodiment of the present disclosure, referring to fig. 5, the model processing apparatus 5 may include an analysis comparison module 17 in addition to the configuration acquisition module 11, the information analysis module 12, the model training module 13, the database 19, and the model prediction module 15.
The analysis comparison module 17 may obtain one or more model predictions from the database 19, wherein the model predictions are sent to the database 19 by the model prediction module 15. Then, the analysis and comparison module 17 may send the model prediction result to the user side, the user side may use software or means of manual analysis to perform analysis and comparison on the model prediction result, and send the analysis and comparison result to the analysis and comparison module 17, where the analysis and comparison module 17 may modify the model parameters according to the analysis and comparison result sent by the user side.
Taking a neural network model as an example, when a user finds that the difference between the model prediction result and the expected result is large, the dimension of the convolution kernel can be increased, and parameter information for increasing the dimension of the convolution kernel is sent to the model processing device, the analysis and comparison module 17 can send the parameter information to the model training module 13, the model training module 13 can retrain the model, and the model prediction module 15 can further predict the data again. In addition, the analysis and comparison module 17 may also directly send the parameter information to the model prediction module 15, so that the model prediction module 15 may directly predict after modifying the model parameters.
In the model processing apparatus of the sixth exemplary embodiment of the present disclosure, referring to fig. 6, the model processing apparatus 6 may include a time control module 18 in addition to the configuration acquisition module 11, the information parsing module 12, the model training module 13, the database 19, and the model prediction module 15.
The time control module 18 may perform operations to build models, train models, and/or predict data at predetermined times. Specifically, the predetermined time may be set by the developer himself, and the time unit of the predetermined time may be minutes, hours, days, weeks, months, or the like. For example, training of the model may be set to begin at 2 a.m. each day to avoid taking up system resources.
According to further embodiments, the configuration acquisition module 11 may be configured to acquire a configuration file and a training script uploaded by a user in a package. Thereby, data loss during transmission can be avoided.
In this case, the model training module 13 may construct a model according to model parameters in the configuration information in response to a training instruction of the user, and acquire a training set according to a training set path in the configuration information, and execute a training script to train the model.
In addition, the present disclosure may also include reconstructing a new model using the historical model information in database 19. In this case, the model can be built faster by modifying only some parameters, so that the time is saved greatly.
Model processing means within the scope of the present disclosure are described above by way of example. It should be understood that, although the information verification module 14 is described in the model processing apparatus 2, the information verification module 14 may also be included in the model processing apparatus 3 to the model processing apparatus 6, and similarly, the abnormality processing module 16, the analysis comparison module 17, the time control module 18 may also be included in other model processing apparatuses.
Further, in this example embodiment, a model processing method is also provided.
Fig. 7 schematically shows a flow chart of a model processing method according to an exemplary embodiment of the present disclosure. Referring to fig. 7, a model processing method of an exemplary embodiment of the present disclosure may include:
S72, acquiring a configuration file of the model;
s74, analyzing the configuration file to obtain configuration information;
S76, constructing a model according to model parameters in the configuration information, and acquiring a training set according to a training set path in the configuration information so as to train the model;
s78, storing the configuration information and training results of the model into a database.
In the model processing method provided by some embodiments of the present disclosure, on one hand, the present disclosure better implements management of a model in a stand-alone environment, and may implement backtracking of the model through configuration information and model training results stored in a database; on the other hand, the configuration information stored in the database can be modified to quickly realize the construction of a new model, so that the time is saved.
According to an exemplary embodiment of the present disclosure, the model processing method further includes: checking the configuration information; and when the verification is successful, constructing a model according to the model parameters in the configuration information.
According to an exemplary embodiment of the present disclosure, the model processing method further includes: and obtaining data to be predicted, predicting the data to be predicted by adopting a trained model, and storing a prediction result into a database.
According to an exemplary embodiment of the present disclosure, the model processing method further includes: judging whether the model is abnormal in the training and/or predicting process, and executing one or more operations of initializing the model, retraining and/or predicting and sending out alarm information when the model is abnormal.
According to an exemplary embodiment of the present disclosure, the model processing method further includes: and obtaining one or more model prediction results from the database, feeding back the one or more model prediction results to the user side so as to enable the user to analyze and compare, and modifying model parameters according to the analysis and comparison results.
According to an exemplary embodiment of the present disclosure, the model processing method further includes: operations to build the model, train the model, and/or predict the data are performed at predetermined times.
According to an exemplary embodiment of the present disclosure, obtaining a configuration file of a model includes obtaining a configuration file and a training script uploaded by a user in a package; the training method comprises the steps of responding to a training instruction of a user, constructing a model according to model parameters in configuration information, acquiring a training set according to a training set path in the configuration information, and training the model by utilizing the training set and executing a training script.
Since the specific process of the model processing method in the embodiment of the present invention is the same as the description corresponding to the above model processing device, the description is omitted here.
In an exemplary embodiment of the present disclosure, a computer-readable storage medium having stored thereon a program product capable of implementing the method described above in the present specification is also provided. In some possible embodiments, the various aspects of the invention may also be implemented in the form of a program product comprising program code for causing a terminal device to carry out the steps according to the various exemplary embodiments of the invention as described in the "exemplary methods" section of this specification, when said program product is run on the terminal device.
Referring to fig. 8, a program product 800 for implementing the above-described method according to an embodiment of the present invention is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The computer readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).
In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.
Those skilled in the art will appreciate that the various aspects of the invention may be implemented as a system, method, or program product. Accordingly, aspects of the invention may be embodied in the following forms, namely: an entirely hardware embodiment, an entirely software embodiment (including firmware, micro-code, etc.) or an embodiment combining hardware and software aspects may be referred to herein as a "circuit," module "or" system.
An electronic device 900 according to such an embodiment of the invention is described below with reference to fig. 9. The electronic device 900 shown in fig. 9 is merely an example, and should not be construed as limiting the functionality and scope of use of embodiments of the present invention.
As shown in fig. 9, the electronic device 900 is embodied in the form of a general purpose computing device. Components of electronic device 900 may include, but are not limited to: the at least one processing unit 910, the at least one storage unit 920, a bus 930 connecting the different system components (including the storage unit 920 and the processing unit 910), and a display unit 940.
Wherein the storage unit stores program code that is executable by the processing unit 910 such that the processing unit 910 performs steps according to various exemplary embodiments of the present invention described in the above-described "exemplary methods" section of the present specification. For example, the processing unit 910 may perform steps S72 to S78 as shown in fig. 7.
The storage unit 920 may include readable media in the form of volatile storage units, such as Random Access Memory (RAM) 9201 and/or cache memory 9202, and may further include Read Only Memory (ROM) 9203.
The storage unit 920 may also include a program/utility 9204 having a set (at least one) of program modules 9205, such program modules 9205 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.
The bus 930 may be one or more of several types of bus structures including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
The electronic device 900 may also communicate with one or more external devices 1000 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 900, and/or with any device (e.g., router, modem, etc.) that enables the electronic device 900 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 950. Also, electronic device 900 may communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet, through network adapter 960. As shown, the network adapter 960 communicates with other modules of the electronic device 900 over the bus 930. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with electronic device 900, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.
From the above description of embodiments, those skilled in the art will readily appreciate that the example embodiments described herein may be implemented in software, or may be implemented in software in combination with the necessary hardware. Thus, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD-ROM, a U-disk, a mobile hard disk, etc.) or on a network, including several instructions to cause a computing device (may be a personal computer, a server, a terminal device, or a network device, etc.) to perform the method according to the embodiments of the present disclosure.
Furthermore, the above-described drawings are only schematic illustrations of processes included in the method according to the exemplary embodiment of the present invention, and are not intended to be limiting. It will be readily appreciated that the processes shown in the above figures do not indicate or limit the temporal order of these processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, for example, among a plurality of modules.
It should be noted that although in the above detailed description several modules or units of a device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit in accordance with embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into a plurality of modules or units to be embodied.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any adaptations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It is to be understood that the present disclosure is not limited to the precise arrangements and instrumentalities shown in the drawings, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims (7)
1. A model processing method applied in a stand-alone environment, comprising:
acquiring a configuration file and a training script which are uploaded by a user in a packing way; the configuration file is generated by a user side according to parameter information required by the model, the parameter information is input by the user through a front end interface of the user side, the configuration file comprises information related to model configuration authority, information related to model parameters and information related to data used by the model, and the information related to the data used by the model comprises a path of a training set, a path of a testing set and a path of data to be predicted;
Analyzing the configuration file to obtain configuration information;
responding to a training instruction of a user, constructing a model according to model parameters in the configuration information, acquiring a training set according to a training set path in the configuration information, and training the model by utilizing the training set and executing the training script;
storing the configuration information and training results of the model into a database;
the model processing method further comprises the following steps:
obtaining data to be predicted, predicting the data to be predicted by adopting a trained model, and storing a prediction result into the database;
And obtaining one or more model prediction results from the database, feeding the one or more model prediction results back to the user side so as to be analyzed and compared by the user, and modifying model parameters according to the analysis and comparison results.
2. The model processing method according to claim 1, characterized in that the model processing method further comprises:
checking the configuration information;
And when the verification is successful, constructing a model according to the model parameters in the configuration information.
3. The model processing method according to claim 1, characterized in that the model processing method further comprises:
Judging whether the model is abnormal in the training and/or predicting process, and executing one or more operations of initializing the model, retraining and/or predicting and sending out alarm information when the model is abnormal.
4. The model processing method according to claim 1, characterized in that the model processing method further comprises:
operations to build the model, train the model, and/or predict the data are performed at predetermined times.
5. A model processing apparatus for use in a stand-alone environment, comprising:
The configuration acquisition module is used for acquiring a configuration file and a training script which are packaged and uploaded by a user; the configuration file is generated by a user side according to parameter information required by the model, the parameter information is input by the user through a front end interface of the user side, the configuration file comprises information related to model configuration authority, information related to model parameters and information related to data used by the model, and the information related to the data used by the model comprises a path of a training set, a path of a testing set and a path of data to be predicted;
The information analysis module is used for analyzing the configuration file to obtain configuration information;
The model training module is used for responding to a training instruction of a user, constructing a model according to model parameters in the configuration information, acquiring a training set according to a training set path in the configuration information, and training the model by utilizing the training set and executing the training script;
the database is used for storing the configuration information and training results of the model;
the model prediction module is used for obtaining data to be predicted, predicting the data to be predicted by adopting a trained model, and storing a prediction result into the database;
And the analysis and comparison module is used for acquiring one or more model prediction results from the database, feeding the one or more model prediction results back to the user side so as to be analyzed and compared by the user, and modifying model parameters according to the analysis and comparison results.
6. A computer-readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the model processing method of any one of claims 1 to 4.
7. An electronic device, comprising:
a processor; and
A memory for storing executable instructions of the processor;
Wherein the processor is configured to perform the model processing method of any one of claims 1 to 4 via execution of the executable instructions.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811191482.8A CN109460396B (en) | 2018-10-12 | 2018-10-12 | Model processing method and device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811191482.8A CN109460396B (en) | 2018-10-12 | 2018-10-12 | Model processing method and device, storage medium and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109460396A CN109460396A (en) | 2019-03-12 |
| CN109460396B true CN109460396B (en) | 2024-06-04 |
Family
ID=65607600
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811191482.8A Active CN109460396B (en) | 2018-10-12 | 2018-10-12 | Model processing method and device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109460396B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109992258A (en) * | 2019-04-02 | 2019-07-09 | 北京市天元网络技术股份有限公司 | A kind of flexible implementation model configuration method and device |
| CN113011138B (en) * | 2019-12-19 | 2023-09-15 | 北京懿医云科技有限公司 | Information processing method, information processing device, electronic equipment and storage medium |
| CN111367891A (en) * | 2020-03-30 | 2020-07-03 | 中国建设银行股份有限公司 | Method, device and equipment for calling modeling intermediate data and readable storage medium |
| CN115146691A (en) * | 2021-03-30 | 2022-10-04 | 华为技术有限公司 | Method, device and system for managing and controlling model training |
| CN113806180B (en) * | 2021-09-23 | 2022-08-12 | 腾云悦智科技(深圳)有限责任公司 | Unsupervised intelligent noise reduction processing method |
| CN114816506A (en) * | 2022-04-21 | 2022-07-29 | 北京金堤科技有限公司 | Model feature rapid processing method and device, storage medium and electronic equipment |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2004287094A (en) * | 2003-03-20 | 2004-10-14 | Fujitsu Ltd | Method and program for presenting operation procedure in on-line learning system |
| JP2013245927A (en) * | 2012-05-29 | 2013-12-09 | Japan Radio Co Ltd | Training support system |
| US8775341B1 (en) * | 2010-10-26 | 2014-07-08 | Michael Lamport Commons | Intelligent control with hierarchical stacked neural networks |
| CN106548675A (en) * | 2016-11-08 | 2017-03-29 | 湖南拓视觉信息技术有限公司 | Virtual military training method and device |
| WO2017193769A1 (en) * | 2016-05-11 | 2017-11-16 | 星环信息科技（上海）有限公司 | Method and apparatus for conversion between machine learning models |
| CN107678951A (en) * | 2017-09-21 | 2018-02-09 | 平安科技（深圳）有限公司 | Test exemple automation management method, device, equipment and storage medium |
| CN108509453A (en) * | 2017-02-27 | 2018-09-07 | 华为技术有限公司 | A kind of information processing method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10803401B2 (en) * | 2016-01-27 | 2020-10-13 | Microsoft Technology Licensing, Llc | Artificial intelligence engine having multiple independent processes on a cloud based platform configured to scale |
-
2018
- 2018-10-12 CN CN201811191482.8A patent/CN109460396B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2004287094A (en) * | 2003-03-20 | 2004-10-14 | Fujitsu Ltd | Method and program for presenting operation procedure in on-line learning system |
| US8775341B1 (en) * | 2010-10-26 | 2014-07-08 | Michael Lamport Commons | Intelligent control with hierarchical stacked neural networks |
| JP2013245927A (en) * | 2012-05-29 | 2013-12-09 | Japan Radio Co Ltd | Training support system |
| WO2017193769A1 (en) * | 2016-05-11 | 2017-11-16 | 星环信息科技（上海）有限公司 | Method and apparatus for conversion between machine learning models |
| CN106548675A (en) * | 2016-11-08 | 2017-03-29 | 湖南拓视觉信息技术有限公司 | Virtual military training method and device |
| CN108509453A (en) * | 2017-02-27 | 2018-09-07 | 华为技术有限公司 | A kind of information processing method and device |
| CN107678951A (en) * | 2017-09-21 | 2018-02-09 | 平安科技（深圳）有限公司 | Test exemple automation management method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109460396A (en) | 2019-03-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109460396B (en) | Model processing method and device, storage medium and electronic equipment | |
| US11481313B2 (en) | Testing framework for automation objects | |
| US20230367559A1 (en) | Development environment for real-time dataflow programming language | |
| US11947943B2 (en) | Industrial automation smart object inheritance | |
| CN109683953B (en) | Method and device for processing configuration file based on visual interface | |
| US11163536B2 (en) | Maintenance and commissioning | |
| US11836686B2 (en) | Feedback based automated maintenance system | |
| US11593562B2 (en) | Advanced machine learning interfaces | |
| US20230305813A1 (en) | Continuous integration and development of code in a secure environment | |
| EP4137937A1 (en) | Industrial automation project library cross sharing | |
| US20230007894A1 (en) | Intelligent Dynamic Web Service Testing Apparatus in a Continuous Integration and Delivery Environment | |
| US10789563B2 (en) | Building confidence of system administrator in productivity tools and incremental expansion of adoption | |
| CN111523676B (en) | Method and device for assisting machine learning model to be online | |
| EP3336705A1 (en) | Certification process for cloud platform | |
| EP4057086A1 (en) | Industrial automation smart object inheritance break and singleton creation | |
| CN111209018B (en) | Method and device for processing application upgrading prompt information and electronic equipment | |
| CN113126881B (en) | System configuration method, device, equipment, readable storage medium and distributed storage system | |
| US11797277B2 (en) | Neural network model conversion method server, and storage medium | |
| US11687441B2 (en) | Intelligent dynamic web service testing apparatus in a continuous integration and delivery environment | |
| US11916773B1 (en) | Data center monitoring and management operation including data center analytics outlier detection operation | |
| US12039292B2 (en) | Maintenance and commissioning | |
| CN114980183B (en) | Network element configuration state monitoring method, device, system, medium and electronic equipment | |
| CN113220507B (en) | Version consistency checking method and device for double-master control system | |
| CN117591104B (en) | Model generation method and device, electronic equipment and storage medium | |
| US20230091963A1 (en) | Industrial automation project design telemetry |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |