CN108984257A - A kind of machine learning platform for supporting custom algorithm component - Google Patents
A kind of machine learning platform for supporting custom algorithm component Download PDFInfo
- Publication number
- CN108984257A CN108984257A CN201810737289.3A CN201810737289A CN108984257A CN 108984257 A CN108984257 A CN 108984257A CN 201810737289 A CN201810737289 A CN 201810737289A CN 108984257 A CN108984257 A CN 108984257A
- Authority
- CN
- China
- Prior art keywords
- machine learning
- algorithm
- client
- learning platform
- custom algorithm
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/451—Execution arrangements for user interfaces
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Stored Programmes (AREA)
Abstract
The present invention discloses a kind of machine learning platform for supporting custom algorithm component, which provides unified access standard for the algorithm assembly that client is encapsulated in docker image, by incoming parameter in need defined by way of order line；Algorithm assembly defined interface is set, and client can configure packaged docker image to an algorithm assembly of platform；Incoming parameter is needed to configure to described in defined interface.Advantage of the present invention is as follows: one, supporting client's custom algorithm component, theoretically support all algorithm frames；Two, scalability is strong, for emerging algorithm, does not need to do additional adaptation；Three, safely controllable, it is provided by client oneself, algorithm platform is only responsible for scheduling.
Description
Technical field
The present invention relates to a kind of machine learning platform more particularly to a kind of machine learning for supporting custom algorithm component are flat
Platform.
Background technique
Current machine learning platform support many algorithms frame, such as ScikitLearn, XGBoost, Spark ML,
TensorFlow, Caffe, BigDL, but still cannot include whole algorithm frames.These algorithm frames are due to various originals
Because can only independent operating, cannot be integrated into machine learning platform.For example, there is License in MatLab；Client is certainly
The algorithm of oneself exploitation, the algorithm frame that do not supported based on platform.Therefore, a kind of flexible scheme is needed, permission is provided by client
Algorithm assembly is scheduled in algorithm platform.
Summary of the invention
It is an object of the invention to by a kind of machine learning platform for supporting custom algorithm component, to solve above carry on the back
The problem of scape technology segment is mentioned.
To achieve this purpose, the present invention adopts the following technical scheme:
A kind of machine learning platform for supporting custom algorithm component, the platform are that client is encapsulated in docker image
Algorithm assembly unified access standard is provided, by incoming parameter in need defined by way of order line；Setting is calculated
Method component defined interface, client can configure packaged docker image to an algorithm assembly of platform；It is making by oneself
Incoming parameter is needed to configure to described in adopted interface.
Particularly, the machine learning platform for supporting custom algorithm component provides the realization of directed acyclic graph (DAG),
The scheduling of client's custom algorithm docker image is realized by independent docker cluster.
Particularly, the Docker cluster supports swarm, kubernetes and mesos.
Particularly, the machine learning platform for supporting custom algorithm component will be objective by managing process (manager)
The customized docker image in family is submitted to docker cluster operation as Job, executes normal flow created,
launching,running,finished。
Particularly, the state of the regular check job of the managing process (manager), when there is timeout, failed,
When stopped state, by its kill and recycle.
The machine learning platform advantage proposed by the present invention for supporting custom algorithm component is as follows: one, supporting client to make by oneself
Adopted algorithm assembly theoretically supports all algorithm frames；Two, scalability is strong, for emerging algorithm, does not need to do additional
Adaptation；Three, safely controllable, it is provided by client oneself, algorithm platform is only responsible for scheduling.
Detailed description of the invention
Fig. 1 is model prediction flow diagram provided in an embodiment of the present invention；
Fig. 2 is the machine learning platform algorithmic dispatching principle provided in an embodiment of the present invention for supporting custom algorithm component
Figure.
Specific embodiment
Present invention will be further explained below with reference to the attached drawings and examples.It is understood that tool described herein
Body embodiment is used only for explaining the present invention rather than limiting the invention.It also should be noted that for the ease of retouching
It states, only some but not all contents related to the present invention are shown in the drawings.Unless otherwise defined, used herein all
Technical and scientific term have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.Herein at this
The term used in the description of invention, which is only for the purpose of describing specific embodiments, is not intended to limit this hair
It is bright.Term " and or " used herein includes any and all combinations of one or more related listed items.
Algorithm assembly is encapsulated in docker image by client in the present embodiment.Support the machine of custom algorithm component
Learning platform is specifically used for: unified access standard is provided for the algorithm assembly that client is encapsulated in docker image, by institute
Incoming parameter in need is defined by way of order line；Algorithm assembly defined interface is set, and client can will be packaged
Docker image is configured to an algorithm assembly of platform；Incoming parameter is needed to match to described in defined interface
It sets.
Specifically, in the present embodiment, the scheduling for algorithm supports the machine learning platform of custom algorithm component to mention
For the realization of directed acyclic graph (DAG), as shown in Figure 1, " GBDT classification " component is just after the completion of " data fractionation " component executes
It can start to execute.And " GBDT classification " component is realized by the calling of docker image, other algorithm assemblies such as " data
Fractionation " " prediction " component is realized by the distributed algorithm frame that platform provides.Therefore, in the present embodiment by independent
The scheduling of docker cluster realization client's custom algorithm docker image.Wherein, the Docker cluster support swarm,
Kubernetes and mesos.
The life cycle of client's custom algorithm includes: 1. creation state created in the present embodiment；2. opening
Dynamic state launching；3. operating status running；4. status of fail failed；5. halted state stopped；6. completing
State finished；7. timeout mode timeout.As shown in Fig. 2, platform will be objective by managing process (manager) when work
The customized docker image in family is submitted to docker cluster operation as Job, executes normal flow created,
Launching, running, finished, but each link is likely to mistake occur.In order to guarantee that resource recycles in time,
The state of the regular check job of the managing process (manager), when there is timeout, failed, when stopped state,
By its kill and recycle.It should be noted that each english nouns are the conventional techniques term of computer field in the present embodiment,
It is unique in the paraphrase of computer field, it is therefore, just no longer superfluous herein to those skilled in the art without any objection
It states.
Technical solution advantage proposed by the present invention is as follows: one, supporting client's custom algorithm component, theoretically support all
Algorithm frame；Two, scalability is strong, for emerging algorithm, does not need to do additional adaptation；Three, safely controllable, certainly by client
Oneself provides, and algorithm platform is only responsible for scheduling.
Those of ordinary skill in the art will appreciate that realizing that all parts in above-described embodiment are can to pass through computer
Program is completed to instruct relevant hardware, and the program can be stored in a computer-readable storage medium, the program
When being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein, the storage medium can for magnetic disk, CD, only
Read storage memory (Read-Only Memory, ROM) or random access memory (Random Access Memory, RAM)
Deng.
The technical principle of the invention is described above in combination with a specific embodiment.These descriptions are intended merely to explain of the invention
Principle, and shall not be construed in any way as a limitation of the scope of protection of the invention.Based on the explanation herein, the technology of this field
Personnel can associate with other specific embodiments of the invention without creative labor, these modes are fallen within
Within protection scope of the present invention.
Claims (5)
1. a kind of machine learning platform for supporting custom algorithm component, which is characterized in that the platform is encapsulated in for client
Algorithm assembly in docker image provides unified access standard, by the incoming parameter in need side that passes through order line
Formula definition；Algorithm assembly defined interface is set, and client can configure packaged docker image to a calculation of platform
Method component；Incoming parameter is needed to configure to described in defined interface.
2. the machine learning platform according to claim 1 for supporting custom algorithm component, which is characterized in that the support
The machine learning platform of custom algorithm component provides the realization of directed acyclic graph, realizes client by independent docker cluster
The scheduling of custom algorithm docker image.
3. the machine learning platform according to claim 2 for supporting custom algorithm component, which is characterized in that described
Docker cluster supports swarm, kubernetes and mesos.
4. the machine learning platform according to claim 3 for supporting custom algorithm component, which is characterized in that the support
The machine learning platform of custom algorithm component is submitted to by managing process using the customized docker image of client as Job
Docker cluster operation, executes normal flow created, launching, running, finished.
5. the machine learning platform according to claim 4 for supporting custom algorithm component, which is characterized in that the management
The state of the regular check job of process, when there is timeout, failed when stopped state, by its kill and is recycled.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810737289.3A CN108984257A (en) | 2018-07-06 | 2018-07-06 | A kind of machine learning platform for supporting custom algorithm component |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810737289.3A CN108984257A (en) | 2018-07-06 | 2018-07-06 | A kind of machine learning platform for supporting custom algorithm component |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108984257A true CN108984257A (en) | 2018-12-11 |
Family
ID=64536402
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810737289.3A Pending CN108984257A (en) | 2018-07-06 | 2018-07-06 | A kind of machine learning platform for supporting custom algorithm component |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108984257A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109635170A (en) * | 2018-12-13 | 2019-04-16 | 成都四方伟业软件股份有限公司 | Algorithm cut-in method, device, electronic equipment and readable storage medium storing program for executing |
| CN109816114A (en) * | 2018-12-29 | 2019-05-28 | 大唐软件技术股份有限公司 | A kind of generation method of machine learning model, device |
| CN110046046A (en) * | 2019-04-09 | 2019-07-23 | 中国科学院计算机网络信息中心 | A kind of distributed hyperparameter optimization system and method based on Mesos |
| CN110471767A (en) * | 2019-08-09 | 2019-11-19 | 上海寒武纪信息科技有限公司 | A kind of dispatching method of equipment |
| CN112906907A (en) * | 2021-03-24 | 2021-06-04 | 成都工业学院 | Method and system for hierarchical management and distribution of machine learning pipeline model |
| US11748128B2 (en) | 2019-12-05 | 2023-09-05 | International Business Machines Corporation | Flexible artificial intelligence agent infrastructure for adapting processing of a shell |
| US11797820B2 (en) | 2019-12-05 | 2023-10-24 | International Business Machines Corporation | Data augmented training of reinforcement learning software agent |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170034023A1 (en) * | 2015-07-27 | 2017-02-02 | Datagrid Systems, Inc. | Techniques for evaluating server system reliability, vulnerability and component compatibility using crowdsourced server and vulnerability data |
| CN107169575A (en) * | 2017-06-27 | 2017-09-15 | 北京天机数测数据科技有限公司 | A kind of modeling and method for visualizing machine learning training pattern |
| CN107451663A (en) * | 2017-07-06 | 2017-12-08 | 阿里巴巴集团控股有限公司 | Algorithm assembly, based on algorithm assembly modeling method, device and electronic equipment |
| CN107948035A (en) * | 2017-11-28 | 2018-04-20 | 广州供电局有限公司 | Business data service bus construction method based on the interaction of private clound big data |
-
2018
- 2018-07-06 CN CN201810737289.3A patent/CN108984257A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170034023A1 (en) * | 2015-07-27 | 2017-02-02 | Datagrid Systems, Inc. | Techniques for evaluating server system reliability, vulnerability and component compatibility using crowdsourced server and vulnerability data |
| CN107169575A (en) * | 2017-06-27 | 2017-09-15 | 北京天机数测数据科技有限公司 | A kind of modeling and method for visualizing machine learning training pattern |
| CN107451663A (en) * | 2017-07-06 | 2017-12-08 | 阿里巴巴集团控股有限公司 | Algorithm assembly, based on algorithm assembly modeling method, device and electronic equipment |
| CN107948035A (en) * | 2017-11-28 | 2018-04-20 | 广州供电局有限公司 | Business data service bus construction method based on the interaction of private clound big data |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109635170A (en) * | 2018-12-13 | 2019-04-16 | 成都四方伟业软件股份有限公司 | Algorithm cut-in method, device, electronic equipment and readable storage medium storing program for executing |
| CN109816114A (en) * | 2018-12-29 | 2019-05-28 | 大唐软件技术股份有限公司 | A kind of generation method of machine learning model, device |
| CN110046046A (en) * | 2019-04-09 | 2019-07-23 | 中国科学院计算机网络信息中心 | A kind of distributed hyperparameter optimization system and method based on Mesos |
| CN110471767A (en) * | 2019-08-09 | 2019-11-19 | 上海寒武纪信息科技有限公司 | A kind of dispatching method of equipment |
| CN110471767B (en) * | 2019-08-09 | 2021-09-03 | 上海寒武纪信息科技有限公司 | Equipment scheduling method |
| US11748128B2 (en) | 2019-12-05 | 2023-09-05 | International Business Machines Corporation | Flexible artificial intelligence agent infrastructure for adapting processing of a shell |
| US11797820B2 (en) | 2019-12-05 | 2023-10-24 | International Business Machines Corporation | Data augmented training of reinforcement learning software agent |
| CN112906907A (en) * | 2021-03-24 | 2021-06-04 | 成都工业学院 | Method and system for hierarchical management and distribution of machine learning pipeline model |
| CN112906907B (en) * | 2021-03-24 | 2024-02-23 | 成都工业学院 | Method and system for layering management and distribution of machine learning pipeline model |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108984257A (en) | A kind of machine learning platform for supporting custom algorithm component | |
| US10467048B2 (en) | Techniques for virtual machine migration | |
| CN110401700B (en) | Model loading method and system, control node and execution node | |
| CN103530898B (en) | A kind of animation processing method and system based on 3D real-time renderings | |
| EP3278220B1 (en) | Power aware scheduling and power manager | |
| US10467054B2 (en) | Resource management method and system, and computer storage medium | |
| CN102937916B (en) | Regularly iterator | |
| CN105955809B (en) | Thread scheduling method and system | |
| WO2016112625A1 (en) | System realizing script operation in preset embedded system | |
| CN108351840B (en) | Vehicle control device | |
| US20220283846A1 (en) | Pod deployment method and apparatus | |
| EP2458501A1 (en) | Method of operating a communication device and related communication device | |
| CN108509284A (en) | A kind of tree shaped model task management system applied to functional expression programming | |
| CN112433823A (en) | Apparatus and method for dynamically virtualizing physical card | |
| CN108806679A (en) | voice awakening method and device | |
| WO2020062277A1 (en) | Management method and apparatus for computing resources in data pre-processing phase of neural network | |
| CN106843890B (en) | Sensor network, node and operation method thereof based on intelligent decision | |
| US8806078B2 (en) | Information processing device and program product | |
| US9043625B2 (en) | Processor bridge power management | |
| US8352055B2 (en) | Method for implementing production processes and system for executing the method | |
| KR101607209B1 (en) | Self adaptive system and method using ontology | |
| CN109709806B (en) | Self-adaptive data acquisition system | |
| CN106453656A (en) | Cluster host selection method and device | |
| CN109522000A (en) | Embedded GUI System and computer program product based on Linux | |
| KR20110064832A (en) | Task management apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20181211 |