WO2024001594A1 - 算子开发、处理方法、装置、电子设备、***及存储介质 - Google Patents
算子开发、处理方法、装置、电子设备、***及存储介质 Download PDFInfo
- Publication number
- WO2024001594A1 WO2024001594A1 PCT/CN2023/094913 CN2023094913W WO2024001594A1 WO 2024001594 A1 WO2024001594 A1 WO 2024001594A1 CN 2023094913 W CN2023094913 W CN 2023094913W WO 2024001594 A1 WO2024001594 A1 WO 2024001594A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- operator
- target
- framework
- description file
- standard
- Prior art date
Links
- 238000011161 development Methods 0.000 title claims abstract description 106
- 238000000034 method Methods 0.000 title claims abstract description 63
- 238000003672 processing method Methods 0.000 title claims abstract description 13
- 238000004806 packaging method and process Methods 0.000 claims abstract description 9
- 238000012360 testing method Methods 0.000 claims description 40
- 230000007246 mechanism Effects 0.000 claims description 23
- 238000012545 processing Methods 0.000 claims description 14
- 101001072091 Homo sapiens ProSAAS Proteins 0.000 claims description 13
- 102100036366 ProSAAS Human genes 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 12
- 238000006243 chemical reaction Methods 0.000 claims description 6
- 230000003993 interaction Effects 0.000 claims description 4
- 238000000605 extraction Methods 0.000 description 24
- 238000010586 diagram Methods 0.000 description 19
- 230000006870 function Effects 0.000 description 16
- 238000010801 machine learning Methods 0.000 description 16
- 238000012549 training Methods 0.000 description 15
- 238000004891 communication Methods 0.000 description 7
- 239000003795 chemical substances by application Substances 0.000 description 5
- 238000011156 evaluation Methods 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000013461 design Methods 0.000 description 3
- 238000003032 molecular docking Methods 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000001902 propagating effect Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- RYGMFSIKBFXOCR-UHFFFAOYSA-N Copper Chemical compound [Cu] RYGMFSIKBFXOCR-UHFFFAOYSA-N 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 229910052802 copper Inorganic materials 0.000 description 1
- 239000010949 copper Substances 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000005538 encapsulation Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000000275 quality assurance Methods 0.000 description 1
- 230000000246 remedial effect Effects 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 238000013515 script Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000003238 somatosensory effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/20—Software design
- G06F8/24—Object-oriented
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/20—Software design
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/31—Programming languages or programming paradigms
- G06F8/315—Object-oriented languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/38—Creation or generation of source code for implementing user interfaces
Definitions
- the present disclosure relates to the field of artificial intelligence technology, and more specifically, to an operator development, processing method, device, electronic equipment, system and storage medium.
- One purpose of the embodiments of the present disclosure is to provide a new technical solution for operator development.
- an operator development method which method includes:
- the description file of the target operator package is determined.
- the description file is configured to describe the basic capabilities and operating capabilities of the operator.
- the operating capabilities at least include framework information and development language, so The above frame information is configured to indicate the operating frame supported by the operator;
- the operator standard template that matches the description file; wherein the operator standard template at least includes: the operator description file and the operator source code;
- the target operator is packaged and stored in the standard operator library.
- an operator processing method which method includes:
- the target operator from the standard operator library; wherein the target operator at least includes the operator's description file and the operator source code; the description file is configured to describe the operator's basic capabilities and operating capabilities, and the operating capabilities At least including framework information and a development language, the framework information is configured to indicate the running framework supported by the operator;
- the processed target operator corresponding to the target execution frame is sent to the target execution frame to run the target operator in the target execution frame.
- an operator development device which device includes:
- the first determination module is configured to determine the description file of the target operator package based on the user's first operation information on the development interface.
- the description file is configured to describe the basic capabilities and operating capabilities of the operator.
- the operating capabilities are at least Including framework information and development language, the framework information is configured to indicate the running framework supported by the operator;
- the second determination module is configured to determine the operator standard template that matches the description file; wherein the operator standard template at least includes: the operator description file and the operator source code;
- a filling module configured to fill in the operator standard template based on the user's second operation information on the development interface to obtain the target operator;
- the packaging module is configured to package the target operator and store it in the standard operator library.
- an operator processing device which device includes:
- the acquisition module is configured to obtain the target operator from the standard operator library; wherein the target operator at least includes the operator's description file and the operator source code; the description file is configured to describe the operator's basic capabilities and operation Capabilities, the running capabilities include at least framework information and development language, the framework information is configured to indicate the running framework supported by the operator;
- a conversion module configured to convert the description file of the target operator according to the protocol supported by the target operation framework, and obtain the processed target operator corresponding to the target operation framework;
- a sending module configured to send the processed target operator corresponding to the target running frame to the target running frame, so as to run the target operator in the target running frame.
- an electronic device including at least one computing device and at least one storage device, wherein the at least one storage device is configured to store instructions configured to control the at least one storage device.
- a computing device performs the method according to the first aspect above; or, the device implements the device according to the second aspect above through the computing device and the storage device.
- a computer-readable storage medium is provided with a computer program stored thereon, and when executed by a processor, the computer program implements any one of the first aspect or the second aspect. Methods.
- an operator operating system characterized in that the system includes: an agent configured to act as an agent for interaction with other systems, and the other systems include: an operator warehouse, an operator Sub-run framework; status tracker, configured to track and record task status; task configurator, configured to monitor tasks and configure tasks.
- One beneficial effect of this disclosure is that it develops target operators through the description file of the target operator package and the operator standard template, where the description file of the target operator is configured to describe the basic capabilities and operating capabilities of the operator,
- the running capability at least includes framework information and development language.
- the framework information is configured to indicate the running framework supported by the operator.
- the operator standard template includes the operator description file and the operator source code.
- Figure 1 is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present disclosure
- Figure 2 is a schematic flowchart of an operator development method according to an embodiment of the present disclosure
- Figure 3 is a schematic flowchart of an operator development method according to an example of the present disclosure
- Figure 4 is a schematic flowchart of an operator processing method according to an embodiment of the present disclosure
- Figure 5 is a schematic structural diagram of an operator operating system according to an embodiment of the present disclosure.
- Figure 6 is a schematic structural diagram of an operator operating system according to another embodiment of the present disclosure.
- Figure 7 is a schematic structural diagram of an operator operating system according to another embodiment of the present disclosure.
- Figure 8 is a schematic diagram of operator operation data flow according to an example of the present disclosure.
- Figure 9 is a functional block diagram of an operator development device according to an embodiment of the present disclosure.
- Figure 10 is a functional block diagram of an operator processing device according to an embodiment of the present disclosure.
- FIG. 11 is a schematic block diagram of an electronic device according to an embodiment of the present disclosure.
- any specific values are to be construed as illustrative only and not as limiting. Accordingly, other examples of the exemplary embodiments may have different values.
- the method of the embodiment of the present disclosure may be implemented by at least one electronic device, that is, the device 900 for implementing the operator development method or the device 1000 for implementing the operator processing method may be arranged on the at least one electronic device.
- Figure 1 shows the hardware structure of any electronic device.
- the electronic device shown in Figure 1 may be a portable computer, a desktop computer, a workstation, a server, etc., or any other device having a computing device such as a processor and a storage device such as a memory, and is not limited here.
- the electronic device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, and so on.
- the processor 1100 is configured to execute the computer program.
- the computer program can be written using instruction sets of architectures such as x86, Arm, RISC, MIPS, SSE, etc.
- the memory 1200 includes, for example, ROM (Read Only Memory), RAM (Random Access Memory), nonvolatile memory such as a hard disk, and the like.
- the interface device 1300 includes, for example, a USB interface, a headphone interface, and the like.
- the communication device 1400 is capable of wired or wireless communication, for example, and may specifically include Wifi communication, Bluetooth communication, 2G/3G/4G/5G communication, etc.
- the display device 1500 is, for example, a liquid crystal display screen, a touch display screen, or the like.
- the input device 1600 may include, for example, a touch screen, a keyboard, somatosensory input, and the like.
- the electronic device 1000 can output voice information through the speaker 1700, and can collect voice information through the microphone 1800, etc.
- the memory 1200 of the electronic device 1000 is configured to store instructions, and the instructions are configured to control the processor 1100 to operate to execute the operator development method or algorithm of the embodiments of the present disclosure. sub-processing method. Skilled personnel can design instructions based on the solutions disclosed in this disclosure. How instructions control the processor to operate is well known in the art and will not be described in detail here.
- an electronic device includes at least one computing device and at least one storage device, the at least one storage device configured to store instructions configured to control the at least one computing device to perform execution in accordance with the present disclosure. Methods of any embodiment.
- the device may include at least one electronic device 1000 shown in FIG. 1 to provide at least one computing device such as a processor and at least one storage device such as a memory, which is not limited here.
- FIG. 2 is a schematic flowchart of an operator development method according to an embodiment of the present disclosure. The method is executed by the electronic device 1000. As shown in Figure 2, the method may include the following steps S2100 to S2400:
- Step S2100 Determine the description file of the target operator package based on the user's first operation information on the development interface.
- the operator can be a functional module developed by the developer according to the needs.
- the operator can include but is not limited to at least one of the following: model training operator, model prediction operator, feature extraction operator, and model evaluation operator.
- the operator can also be a custom operator developed by the developer according to the customer's needs. The functions and application scenarios of the custom operator can be customized and designed by the customer. This embodiment is not limited here.
- the description file is configured to describe the basic capabilities and running capabilities of the operator.
- the running capabilities include at least framework information and development language, and the framework information is configured to indicate the running framework supported by the operator.
- the format of the description file is yaml.
- the description file can be named Operator.yaml to distinguish it from other files.
- the framework information is configured to describe the operating framework supported by the operator.
- the operating framework involved in the embodiment of the present disclosure may include but is not limited to at least one of the following: spark, flink, horovod, dask, mars, mindspore, local, k8s, hadoop, SAAS.
- different operating frameworks can be defined in the description file as needed, so that the developed target operator can be adapted to the defined operating framework.
- the developed target operator can run on the two or more operating frameworks at the same time, thereby enabling the target operator to run across operating frameworks. .
- the operator development method provided by the embodiments of the present disclosure lowers the threshold for operator use.
- the operator has high portability, can run in a variety of operating frameworks, and has a wide range of applications.
- the development language involved in the embodiment of the present disclosure may include but is not limited to at least one of the following: Python language, golang language, c language, c++ language, and Java language.
- different development languages can be defined in the description file as needed, and then operators can be developed based on the defined development languages, which can be compatible with operators developed in multiple languages.
- the operating capabilities of the operator may also include but are not limited to at least one of the following: operating type, scale, load mode, underlying platform, management mode, dependency, output data set, status, and architecture.
- the operation type may include but is not limited to online type operators and batch type operators.
- the online type operator can usually provide online prediction services.
- the online type operator may include but is not limited to services and streaming methods. This batch type operator can usually provide batch prediction services.
- the scale can include but is not limited to stand-alone operators and distributed operators.
- Load modes can include but are not limited to remote and local.
- the underlying platform can include but is not limited to at least one of the following: k8s (kubernetes), yarn, and hadoop.
- Management modes can include but are not limited to hosting (watch through admission hook, non-invasive, suitable for native k8s types), status reporting (non-native frameworks, third-party k8s CRD or other frameworks), etc.
- Architectures can include but are not limited to x86, arm.
- Dependencies can include but are not limited to ray, dask, and horovod.
- the status may include but is not limited to synchronization status information syncStatus and running status runningPhase, where synchronization status information syncStatus may include but is not limited to unsynced, synchronized, synchronized, and unknown.
- the running status runningPhase can include but is not limited to pending, running, succeeded, skipped, failed, and error.
- the basic capabilities of the operator include but are not limited to: input, output, and operating parameters.
- the input is configured to describe the input information when the operator is running.
- the input may include but is not limited to source type, configuration, and input data.
- the output is configured to describe the output information when the operator is running.
- the output may include but is not limited to output type, configuration, and output data.
- Running parameters are necessary business parameters required when the operator is running.
- the necessary business parameters can include but are not limited to the number of concurrency, the number of training rounds, and the model type. In this embodiment, by defining the necessary business parameters required when the operator is running in the description file, the operator can run stably, avoid operator failure, and thereby improve the operator's operation capability.
- each item in the description file can also be modified based on the actual situation.
- the content is adaptively increased or decreased.
- the user's first operation information on the development interface may be information related to the user's selection operation, determination operation, or editing operation for the description file output by the electronic device.
- the description file output by the electronic device may be is an initial description file.
- the initial description file may be a file pre-stored in the electronic device.
- Developers can directly perform a determination operation to determine the initial description file as the description file of the target operator package; or, developers can edit the initial description file according to their own needs to obtain the description file of the target operator package.
- the electronic device can output multiple description files, and the user can select on the interface to determine the description file of the target operator package. This method of obtaining the description file of the target operator package based on the description file output by the electronic device has high operator development efficiency.
- the user's first operation information on the development interface can also be the description file of the target operator package manually written by the user.
- This method of manually writing the description file of the target operator package is more in line with user needs. It can also meet the user's custom design needs, and provide users with remedial measures to customize the design description file when the preset description file of the electronic device does not meet the user's needs.
- Step S2200 Determine the operator standard template that matches the description file.
- the operator standard template includes at least: the operator description file and the operator source code. It can be understood that the operator standard template can be embodied as scaffolding, and the scaffolding can be called the code directory structure and entry function. In this embodiment, the scaffold plays the role of the operator standard template.
- the operator source code is the code written to realize the operator function.
- the feature extraction operator is configured to extract features from the input data.
- the source code of the feature extraction operator is the code written to implement the feature extraction function from the input data.
- the model training operator is configured to perform model training based on the results of feature extraction to obtain a machine learning model.
- the source code of the model training operator is to implement the function of performing model training based on the results of feature extraction to obtain a machine learning model. code written.
- the model evaluation operator is configured to evaluate the performance of a trained machine learning model.
- the source code of the model evaluation operator is written to implement the function of evaluating the performance of a trained machine learning model. code.
- the operator standard template in addition to the description file and the operator source code, also includes at least one of the following: an operator description document and an operator test file.
- operator documentation is usually written to facilitate customers and ordinary people to use, run, and debug the operator normally, thereby lowering the threshold for operator application.
- the operator's test files can include but are not limited to test cases and test report templates.
- a test case is a description of the test task for an operator, which embodies the test plan, method and strategy. Its content can include but is not limited to test goals, test environment, input data, test steps, expected results, and test scripts.
- the test report template is the test report obtained by executing the test case against the operator. In this embodiment, by specifying the test file of the operator in the operator standard template, the quality of the operator can be improved, so that after the target operator is developed, the operator rating can be performed through a combination of automatic testing and manual review.
- step S2200 of determining the operator standard template that matches the description file may further include: selecting from the preset operator standard templates that matches the framework information and the development language. Operator standard template.
- the preset operator standard template includes multiple operator standard templates, such as but not limited to operator standard template 1 and operator standard template 2 .
- operator standard template 1 and operator standard template 2 any two different development languages have different categories of operator standard templates corresponding to them, and the operator standard templates of different categories are different.
- the major items in the operator standard template (description file , operator source code, documentation, test files), the content contained in them or the content generation methods are different.
- the development language Python corresponds to a category of operator standard templates
- the development language C++ corresponds to a category of operator standard templates, and the content or content generation methods of each major item in the operator standard templates of these two categories are different.
- the framework information and development language have a mapping relationship with the operator standard template. That is to say, if the framework information and development language are different, the corresponding operator standard templates are different. In this way, when determining the description file of the target operator package In this case, the operator standard template corresponding to the framework information and development language can be determined from the preset operator standard template according to the framework information and development language in the description file, and the operator standard can be accurately determined template.
- step S2200 of determining the operator standard template that matches the description file may further include: generating an operator standard template that matches the framework information and the development language.
- the electronic device when the description file of the target operator package is determined, can also automatically generate an operator standard template that matches the framework information and development language directly based on the framework information and development language in the description file. , which can efficiently generate operator standard templates.
- Step S2300 Fill in the operator standard template based on the user's second operation information on the development interface to obtain the target operator.
- the operator standard template can be filled in to obtain the target operator based on the user's second operation information on the development interface.
- Step S2400 Package the target operator and store it in the standard operator library.
- packaging the target operator and storing it in the standard operator library in step S2400 may further include: verifying whether the target operator satisfies the operator admission mechanism; when the target operator satisfies the In the operator admission mechanism, the target operator is packaged and stored in the standard operator library.
- the operator admission mechanism can be an automatic testing mechanism.
- the operator admission mechanism may be related to the operator's test file.
- the operator's test file can be used to test the operator. When it passes the test, it can be determined that the operator access mechanism is met.
- the operator access mechanism can also be manually controlled by business personnel.
- the target operator after the target operator is obtained, it can be determined through manual control whether the target operator satisfies the operator admission mechanism. That is, it is provided for business personnel to manually control the quality of the operator. a possibility.
- the automatic test mechanism and the manual control mechanism can also be combined and applied.
- the target operator satisfies the operator admission mechanism.
- the operator access mechanism can also be set according to actual needs. This embodiment does not limit the specific content of the operator access mechanism. In other words, after obtaining the target operator, it does not directly incorporate the target operator into the standard operator library, but first verifies the target operator based on the operator access mechanism, and if the verification passes, Only then will the target operators be packaged and stored in the standard operator library to further improve the quality of the operators, ensure the consistency of the operator levels in the standard operator library, and accordingly reduce the cost of operator development.
- standardized operators can run both in the cluster and locally, and have the same functional experience, which reduces the learning cost for operator developers who need to learn multiple distributed clusters.
- the target operator does not meet the operator admission mechanism, it will be returned to the developer so that the developer can modify or adjust it.
- the target operator adjusted by the developer can still be used as shown in Figure 2. for development.
- the target operator is developed through the description file of the target operator package and the operator standard template, wherein the description file of the target operator is configured to describe the basic capabilities and operating capabilities of the operator, and the operating capabilities At least the framework information and development language are included.
- the framework information is configured to indicate the running framework supported by the operator.
- the operator standard template includes the operator description file and the operator source code.
- the operator development method may include the following steps S310 to S390:
- Step S310 Generate a description text of the target operator package based on the first operation information of the operator developer on the development interface. Operator.yaml.
- Step S320 Automatically generate or manually write scaffolding code based on the development language and running framework defined in the description file Operator.yaml.
- Step S330 perform opctl local debugging on the scaffolding code.
- Step S340 Fill in the scaffolding code to obtain the target operator.
- Step S350 Package the target operator and upload it to the continuous integration tool (CI tool).
- CI tool continuous integration tool
- Step S360 Perform automated testing on the target operator in the CI tool based on the operator test file to obtain automated test results.
- Step S370 The operator developer debugs the target operator in the OpHub system based on the automated test results, and writes test files for the debugged target operator.
- Step S380 After verifying in the CI tool that the target operator satisfies the operator admission mechanism, quality assurance personnel (QA personnel) manually test the target operator based on the written test file, and if the test passes, The packaged target operator is released through the CI tool.
- quality assurance personnel QA personnel
- Step S390 Drop the target operator into the standard operator library, that is, the product library.
- the benefits brought by operator standardization to R&D during the R&D process include: First, it develops target operators based on generating scaffolding code, which can reduce duplication of work when developing operators and improve the efficiency of operators.
- the adaptability and scope of use greatly reduce the difficulty of machine learning implementation and reduce development costs and time costs.
- full life cycle management improves operator development efficiency and reduces development time by 50% compared to traditional operator development.
- standardized operators can run in clusters or locally, which reduces the learning cost for operator developers to learn multiple distributed clusters.
- the benefits of operator standardization to product managers include but are not limited to: testing the operator through the operator's test file before dropping it into the product library, and designing easy-to-use operator access through access to automated testing.
- the mechanism improves operator quality.
- FIG. 4 is a schematic flowchart of an operator processing method according to an embodiment of the present disclosure.
- the method is executed by the electronic device 1000. Specifically, it may be executed by the OpHub system in the electronic device, or it may also be executed by other modules or systems in the electronic device. , wherein these modules or systems can schedule and apply operators according to the method shown in Figure 4.
- the method can include the following steps S4100 to S4300:
- Step S4100 Obtain the target operator from the standard operator library.
- the target operator may be one or more operators in the standard operator library developed according to the first embodiment.
- the required one or more target operators can be obtained from the standard operator library.
- the target operator can be developed based on the local integrated development environment (local IDE) or the network integrated development environment (web IDE) and uploaded to the standard operator library (git repo), so that the system (such as OpHub) can be downloaded from the git rego Get the target operator.
- the target operator at least includes the operator's description file and the operator source code; the description file is configured to describe the operator's basic capabilities and operating capabilities, and the operating capabilities at least include framework information and development language.
- the framework information Configured to indicate the operating framework supported by the operator.
- the running framework supported by the framework information may include but is not limited to at least one of the following: spark, flink, horovod, dask, mars, mindspore, local, k8s, hadoop, and SAAS.
- spark flink
- horovod dask
- mars mindspore
- local k8s
- hadoop and SAAS.
- SAAS SAAS
- the development language involved in the embodiment of the present disclosure may include but is not limited to at least one of the following: Python language, golang language, c language, c++ language, and Java language.
- different development languages can be defined in the description file as needed, and then operators can be developed based on the defined development languages, which can be compatible with operators developed in multiple languages.
- the operating capabilities of the operator may also include but are not limited to at least one of the following: operating type, scale, load mode, The underlying platform, management model, dependencies, output data dataset, status, and architecture.
- operating type e.g., operating type, scale, load mode
- the underlying platform e.g., management model, dependencies, output data dataset, status, and architecture.
- the basic capabilities of the operator include but are not limited to: input, output, and operating parameters.
- input, output, and operating parameters For details, please refer to the introduction of the operator development embodiment. This embodiment will not be described in detail here.
- the operator source code is the code written to realize the operator function.
- the feature extraction operator is configured to extract features from the input data.
- the source code of the feature extraction operator is the code written to implement the feature extraction function from the input data.
- the model training operator is configured to perform model training based on the results of feature extraction to obtain a machine learning model.
- the source code of the model training operator is to implement the function of performing model training based on the results of feature extraction to obtain a machine learning model. code written.
- the model evaluation operator is configured to evaluate the performance of a trained machine learning model.
- the source code of the model evaluation operator is written to implement the function of evaluating the performance of a trained machine learning model. code.
- Step S4200 Convert the description file of the target operator according to the protocol supported by the target execution framework to obtain the processed target operator corresponding to the target execution framework.
- the target running framework is the running framework supported by the operator indicated by the operator's framework information in the operator's description file. It can be understood that there can be only one target execution framework, or there can be multiple target execution frameworks. For details, please refer to the above introduction.
- step S4200 converting the description file of the target operator according to the protocol supported by the target operation framework, and obtaining the processed target operator corresponding to the target operation framework may further include: according to The protocol supported by the target execution framework parses the description file of the target standard into a second file readable by the protocol; according to the second file, obtains the processed target corresponding to the target execution framework operator.
- the target operating framework in order to ensure that the target operating framework can run the target operator normally, it is necessary to first parse the target operator's description file into its protocol according to the protocols supported by the target operating framework. A readable file, and then the processed target operator corresponding to the target execution framework is obtained based on the file.
- the provided running frameworks can include local, k8s, hadoop, SAAS... (not exhaustive).
- the target running frameworks can be hadoop and SAAS, and the features can be extracted according to the protocols supported by hadoop.
- the description file of the operator is parsed into a readable file, and the processed feature extraction operator corresponding to hadoop is obtained.
- the description file of the feature extraction operator is parsed into a readable file, and SAAS is obtained.
- the corresponding processed feature extraction operator is configured to track the task running status during the operator's execution of the task.
- the task tracking framework can include but is not limited to: xxx Platform, tfBorad, ES, Grafana, log, tracing.
- Step S4300 Send the processed target operator corresponding to the target execution frame to the corresponding execution frame, so as to run the target operator in the corresponding execution frame.
- the processed feature extraction operator can be sent to hadoop so that the processed feature extraction operator can be run in hadoop. And after obtaining the processed feature extraction operator corresponding to SAAS, the processed feature extraction operator can be sent to SAAS so that the processed feature extraction operator can be run in SAAS.
- the target operator obtained is obtained from a standard operator library.
- the target operator includes a description file of the operator.
- the description file is configured to describe the operator's operating capability, and the operating capability at least includes Framework information and development language, the framework information is configured to indicate the running framework supported by the operator. That is, the obtained target operator can configure the framework information according to the needs, which improves the adaptability of the operator.
- the target operator can be made to have cross-operation capabilities. The ability of the framework, that is, the target operator can be adapted to different operating frameworks, and then run in different operating frameworks to achieve cross-operating frameworks.
- the description file of the target operator is converted according to the protocol supported by the target execution framework, and the description file of the target operator is obtained according to the protocol supported by the target execution framework.
- the operator processing method in the embodiment of the present disclosure may further include the following steps: S5100 ⁇ step S5200:
- Step S5100 Parse the description file of the target operator into a first file readable by the orchestration framework through the adapter corresponding to the orchestration framework.
- the orchestration framework can include but is not limited to at least one of the following: workflow, argo workflow, airflow, and kubeflow.
- an adapter executor in a scenario involving multiple target operators for orchestration and scheduling, an adapter executor will be involved.
- the adapter corresponds to the orchestration framework one-to-one.
- the adapter is configured to realize the interaction between the orchestration framework and the system.
- the adapter corresponding to the orchestration framework Specifically, it is configured as follows: According to the requirements of the orchestration framework, the description file of each target operator is converted into a file readable by the orchestration framework.
- the provided orchestration frameworks include kubeflow, airflow, argo workflow, and xxx workflow.
- operator developers can select the orchestration framework workflow.
- the target operators include feature extraction operators and model training algorithms. Here, you can convert the description file of the feature extraction operator and the description file of the model training operator according to the adapter corresponding to the workflow.
- Step S5200 The arrangement framework arranges the target operator based on the first file corresponding to the target operator, and obtains the arranged target operator.
- the orchestration framework workflow arranges the feature extraction operator and the model training operator based on the first file corresponding to the feature extraction operator and the first file corresponding to the model training operator, and extracts the arranged feature Operators and model training operators are protocol converted through the OpHub system and sent to the running framework for execution.
- the system can convert the description files of the target operators according to the protocol supported by the target operation framework to obtain the processed target corresponding to the target operation framework. operator, and sends the processed target operator to the corresponding running framework to run the target operator in the corresponding running framework.
- the operator runtime is decoupled from the orchestration and scheduling, different adapters are developed for different orchestration frameworks, and the target operators are submitted to the system through the adapters.
- an embodiment of the present disclosure also provides an operator operating system.
- the operator operating system 600 includes an agent 610, a status tracker 620, and a task configurator 630.
- the operator operating system 600 may be an OpHub system.
- the agent (transmitter) 610 is configured to act as a proxy for interaction with other systems.
- the other systems include: an operator warehouse and an operator execution framework.
- the operator warehouse is the standard operator library described in the above embodiment, and the operator warehouse is the nexus repository shown in Figure 7.
- the operator running framework may include but is not limited to at least one of the following: spark, flink, horovod, dask, mars, and mindspore.
- Figure 7 shows the running framework kubernetes.
- the other system may further include a service framework and a management framework.
- the service framework is the data framework that the operator needs to depend on.
- the service framework can include but is not limited to elasticsearch and MySQL as shown in Figure 7.
- the management framework is configured as a management task, and the management framework may be the distributed version control system git shown in Figure 7.
- a status tracker (watcher) 620 is configured to track and record task status.
- the status tracker 620 can monitor tasks in kubernetes and record them to elasticsearch and MySQL.
- the status tracker 620 includes a log monitoring collection component or can connect to a third party.
- Task configurator 630 (addmission webhook) is configured to monitor tasks and configure tasks. Referring to Figure 7, the task configurator 630 can monitor tasks submitted to kubernetes and perform unified configuration at the cluster level and namespace level.
- agent 610 status tracker 620 and task configurator 630 provide the ability to expose data, and the storage of specific data can be flexibly selected according to on-site dependencies.
- FIG. 8 Combined with Figure 8, a schematic diagram of the operator operation data flow is shown.
- the operator execution task is first submitted, and the task is submitted to the operator operation framework platform, and the task is executed in the operator operation framework.
- information can be obtained while the operator is running through meta/status, log, and trace in the status tracker 620. Collect and view operator runtime information.
- the embodiment of the present disclosure also provides an operator development device 900.
- the operator development device 900 includes a first determination module 910, a second determination module 920, a filling module 930, and a packaging module 940.
- the first determination module 910 is configured to determine the description file of the target operator package based on the user's first operation information on the development interface.
- the description file is configured to describe the basic capabilities and operating capabilities of the operator.
- the operating capabilities It includes at least framework information and a development language, and the framework information is configured to indicate a running framework supported by the operator.
- the second determination module 920 is configured to determine the operator standard template that matches the description file; wherein the operator standard template at least includes: the operator description file and the operator source code.
- the filling module 930 is configured to fill in the operator standard template based on the user's second operation information on the development interface to obtain the target operator.
- the packaging module 940 is configured to package the target operator and store it in the standard operator library.
- the second determination module 920 is specifically configured as:
- a matching operator standard template is generated.
- the operator standard template also includes at least one of the following: an operator description document and an operator test file.
- the basic capabilities of the operator include: input, output, and operating parameters.
- the operator operation capability also includes at least one of the following: operation type, scale, load mode, underlying platform, management mode, dependency, output data set, status, and architecture.
- the running framework supported by the framework information includes at least one of the following: spark, flink, horovod, dask, mars, and mindspore.
- the packaging module 940 is specifically configured to: verify whether the target operator satisfies the operator admission mechanism; when the target operator satisfies the operator admission mechanism, the The target operator is packaged and stored in the standard operator library.
- the target operator is developed through the description file of the target operator package and the operator standard template, wherein the description file of the target operator is configured to describe the basic capabilities and operating capabilities of the operator, and the operating capabilities At least the framework information and development language are included.
- the framework information is configured to indicate the running framework supported by the operator.
- the operator standard template includes the operator description file and the operator source code.
- An embodiment of the present disclosure also provides an operator processing device 1000.
- the operator processing device 1000 includes an acquisition module 1010, a conversion module 1020, and a sending module 1030.
- the acquisition module 1010 is configured to obtain the target operator from the standard operator library; wherein the target operator at least includes the operator's description file and the operator source code; the description file is configured to describe the operator's basic capabilities and Running capabilities include at least framework information and development language, and the framework information is configured to indicate the running framework supported by the operator.
- the conversion module 1020 is configured to convert the description file of the target operator according to the protocol supported by the target execution framework, and obtain the processed target operator corresponding to the target execution framework.
- the sending module 1030 is configured to send the processed target operator corresponding to the target execution frame to the target execution frame, so as to run the target operator in the target execution frame.
- the device further includes an orchestration module (not shown in the figure), specifically configured to:
- the adapter corresponding to the arrangement framework parses the description file of the target operator into a first file readable by the arrangement framework; the arrangement framework, based on the first file corresponding to the target operator, analyzes the The target operator is arranged, and the arranged target operator is obtained.
- the conversion module 1020 is specifically configured to: parse the description file of the target standard into a second file readable by the protocol according to the protocol supported by the target execution framework; The second file is used to obtain the processed target operator corresponding to the target operation framework.
- the operator standard template also includes at least one of the following: an operator description document and an operator test file.
- the basic capabilities of the operator include: input, output, and operating parameters.
- the operator operation capability further includes at least one of the following: operation type, scale, load mode, underlying platform, management mode, and dependencies.
- the running framework supported by the framework information includes at least one of the following: spark, flink, horovod, dask, mars, and mindspore.
- the target operator obtained is obtained from a standard operator library.
- the target operator includes a description file of the operator.
- the description file is configured to describe the operator's operating capability, and the operating capability at least includes Framework information and development language, the framework information is configured to indicate the running framework supported by the operator. That is, the obtained target operator can configure the framework information according to the needs, which improves the adaptability of the operator.
- the target operator can be made to have cross-operation capabilities. The ability of the framework, that is, the target operator can be adapted to different operating frameworks, and then run in different operating frameworks to achieve cross-operating frameworks.
- the embodiment of the present disclosure provides an electronic device 1100.
- the electronic device 1100 may be a server or a terminal device.
- the server may be an integrated server or a distributed server across multiple computers or computer data centers.
- Terminal devices can be mobile phones, tablets, laptops, etc.
- the electronic device 1100 includes a processor 1110 and a memory 1120.
- the memory 1120 is configured to store executable instructions; the processor 1110 is configured to run the electronic device 1100 according to the control of the instructions to execute any implementation according to the present disclosure. Examples of operator development methods or operator processing methods.
- the electronic device may be an electronic product having a processor and memory.
- it can be a smartphone, laptop computer, desktop computer, tablet computer, etc.
- Embodiments of the present disclosure provide a computer-readable storage medium on which a computer program is stored. When executed by a processor, the computer program implements the operator development method or operator processing method provided in any of the foregoing embodiments.
- the present disclosure may be a system, method, and/or computer program product.
- a computer program product may include a computer-readable storage medium having thereon computer-readable program instructions for causing a processor to implement aspects of the present disclosure.
- Computer-readable storage media may be tangible devices that can retain and store instructions for use by an instruction execution device.
- the computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or Flash memory), Static Random Access Memory (SRAM), Compact Disk Read Only Memory (CD-ROM), Digital Versatile Disk (DVD), Memory Stick, Floppy Disk, Mechanical Coding Device, such as a printer with instructions stored on it.
- RAM random access memory
- ROM read-only memory
- EPROM erasable programmable read-only memory
- Flash memory Static Random Access Memory
- CD-ROM Compact Disk Read Only Memory
- DVD Digital Versatile Disk
- Memory Stick
- Computer-readable storage media are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or through electrical wires. transmitted electrical signals.
- Computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to various computing/processing devices, or to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, and/or a wireless network.
- Networks can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, Gateway computers and/or edge servers.
- a network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage on a computer-readable storage medium in the respective computing/processing device .
- Computer program instructions for performing operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or instructions in one or more programming languages.
- the computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server implement.
- the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as an Internet service provider through the Internet). connect).
- LAN local area network
- WAN wide area network
- an external computer such as an Internet service provider through the Internet. connect
- an electronic circuit such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA)
- the electronic circuit can Computer readable program instructions are executed to implement various aspects of the disclosure.
- These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, thereby producing a machine that, when executed by the processor of the computer or other programmable data processing apparatus, , resulting in an apparatus that implements the functions/actions specified in one or more blocks in the flowchart and/or block diagram.
- These computer-readable program instructions can also be stored in a computer-readable storage medium. These instructions cause the computer, programmable data processing device and/or other equipment to work in a specific manner. Therefore, the computer-readable medium storing the instructions includes An article of manufacture that includes instructions that implement aspects of the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.
- Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other equipment, causing a series of operating steps to be performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented process , thereby causing instructions executed on a computer, other programmable data processing apparatus, or other equipment to implement the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams.
- each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions that embody one or more elements for implementing the specified logical function(s).
- Executable instructions may occur out of the order noted in the figures. For example, two consecutive blocks may actually execute substantially in parallel, or they may sometimes execute in the reverse order, depending on the functionality involved.
- each block of the block diagram and/or flowchart illustration, and combinations of blocks in the block diagram and/or flowchart illustration can be implemented by special purpose hardware-based systems that perform the specified functions or acts. , or can be implemented using a combination of specialized hardware and computer instructions. It is well known to those skilled in the art that implementation through hardware, implementation through software, and implementation through a combination of software and hardware are all equivalent.
- the target operator is developed through the description file of the target operator package and the operator standard template
- the description file of the target operator is configured to describe the basic capabilities and operating capabilities of the operator.
- the operating capabilities include at least framework information and development language.
- the framework information is configured to indicate the operating framework supported by the operator.
- the operator standard template Includes operator description files and operator source code.
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Computing Systems (AREA)
- Stored Programmes (AREA)
Abstract
本公开实施例公开一种算子开发、处理方法、装置、电子设备、***及存储介质。该方法包括:基于用户在开发界面上的第一操作信息,确定目标算子包的描述文件,所述描述文件被配置为描述算子基础能力与运行能力,所述运行能力至少包括框架信息与开发语言,所述框架信息被配置为指示算子所支持的运行框架;确定与所述描述文件相匹配的算子标准模板;其中,所述算子标准模板至少包括:算子的描述文件与算子源代码;基于所述用户在所述开发界面上的第二操作信息,填充所述算子标准模板,得到所述目标算子;将所述目标算子打包并存入标准算子库。
Description
本公开要求于2022年06月29日提交中国专利局,申请号为202210762431.6,申请名称为“算子开发、处理方法、装置、电子设备、***及存储介质”的中国专利申请的优先权,其全部内容通过引用结合在本公开中。
本公开涉及人工智能技术领域,更具体的,涉及一种算子开发、处理方法、装置、电子设备、***及存储介质。
在机器学习领域,模型的产生往往只是机器学习落地应用的第一步。模型产生之后往往需要经过服务化封装、参数调优、数据对接、编排框架对接、运行框架对接等诸多复杂的流程,而且每个流程往往需要有相关技术背景的工程师反复沟通协助才能完成。
为方便协作,目前在机器学习领域抽象出算子的概念,并将算子作为运行在运行框架的最小单元。相关技术中,其是通过SDK(Software Development Kit)的方式进行算子的开发,且算子的开发往往与某一运行框架绑定,当需要将算子与其他运行框架进行绑定时,这种SDK开发方式对算子本身的逻辑有较强侵入性,对原有逻辑有一定的改造成本,且每次开发算子时需要进行大量的重复工作,增加了开发成本和时间成本。
发明内容
本公开实施例的一个目的是提供一种算子开发的新的技术方案。
根据本公开的第一方面,提供了一种算子开发方法,该方法包括:
基于用户在开发界面上的第一操作信息,确定目标算子包的描述文件,所述描述文件被配置为描述算子基础能力与运行能力,所述运行能力至少包括框架信息与开发语言,所述框架信息被配置为指示算子所支持的运行框架;
确定与所述描述文件相匹配的算子标准模板;其中,所述算子标准模板至少包括:算子的描述文件与算子源代码;
基于所述用户在所述开发界面上的第二操作信息,填充所述算子标准模板,得到所述目标算子;
将所述目标算子打包并存入标准算子库。
根据本公开的第二方面,提供了一种算子处理方法,该方法包括:
从标准算子库获取目标算子;其中,所述目标算子至少包括算子的描述文件与算子源代码;所述描述文件被配置为描述算子基础能力与运行能力,所述运行能力至少包括框架信息与开发语言,所述框架信息被配置为指示算子所支持的运行框架;
根据目标运行框架所支持的协议,将所述目标算子的描述文件进行转换,获得与所述目标运行框架对应的处理后的目标算子;
将与所述目标运行框架对应的处理后的目标算子发送至所述目标运行框架,以在所述目标运行框架中运行所述目标算子。
根据本公开的第三方面,提供了一种算子开发装置,该装置包括:
第一确定模块,被配置为基于用户在开发界面上的第一操作信息,确定目标算子包的描述文件,所述描述文件被配置为描述算子基础能力与运行能力,所述运行能力至少包括框架信息与开发语言,所述框架信息被配置为指示算子所支持的运行框架;
第二确定模块,被配置为确定与所述描述文件相匹配的算子标准模板;其中,所述算子标准模板至少包括:算子的描述文件与算子源代码;
填充模块,被配置为基于所述用户在所述开发界面上的第二操作信息,填充所述算子标准模板,得到所述目标算子;
打包模块,被配置为将所述目标算子打包并存入标准算子库。
根据本公开的第四方面,提供了一种算子处理装置,该装置包括:
获取模块,被配置为从标准算子库获取目标算子;其中,所述目标算子至少包括算子的描述文件与算子源代码;所述描述文件被配置为描述算子基础能力与运行能力,所述运行能力至少包括框架信息与开发语言,所述框架信息被配置为指示算子所支持的运行框架;
转换模块,被配置为根据目标运行框架所支持的协议,将所述目标算子的描述文件进行转换,获得与所述目标运行框架对应的处理后的目标算子;
发送模块,被配置为将与所述目标运行框架对应的处理后的目标算子发送至所述目标运行框架,以在所述目标运行框架中运行所述目标算子。
根据本公开的第五方面,提供了一种包括至少一个计算装置和至少一个存储装置的电子设备,其中,所述至少一个存储装置被配置为存储指令,所述指令被配置为控制所述至少一个计算装置执行根据以上第一方面所述的方法;或者,所述设备通过所述计算装置和所述存储装置实现根据以上第二方面所述的装置。
根据本公开的第六方面,提供了一种计算机可读存储介质,其上存储有计算机程序,所述计算机程序在被处理器执行时实现如第一方面或第二方面中任一项所述的方法。
根据本公开的第七方面,提供了一种算子运行***,其特征在于,所述***包括:代理器,被配置为代理与其他***的交互,所述其他***包括:算子仓库、算子运行框架;状态***,被配置为跟踪任务状态并记录;任务配置器,被配置为监听任务并作任务配置。
本公开的一个有益效果在于,其是通过目标算子包的描述文件和算子标准模板进行目标算子的开发,其中,目标算子的描述文件被配置为描述算子基础能力与运行能力,运行能力至少包括框架信息与开发语言,框架信息被配置为指示算子所支持的运行框架,其中,算子标准模板包括算子的描述文件和算子源代码。具体进行目标算子开发时,需要确定出目标算子包的描述文件,然后确定与该描述文件匹配的算子标准模板,并填充该算子标准模板以得到目标算子。即,其是通过配置算子标准模板的方式进行目标算子的开发,能够降低开发算子时的重复工作,提升了算子的适配性,大幅降低机器学习落地难度,降低了开发成本和时间成本。
被结合在说明书中并构成说明书的一部分的附图示出了本公开的实施例,并且连同其说明一起用于解释本公开的原理。
图1是根据本公开实施例的电子设备的硬件结构示意图;
图2是根据本公开实施例的算子开发方法的流程示意图;
图3是根据本公开一个例子的算子开发方法的流程示意图;
图4是根据本公开实施例的算子处理方法的流程示意图;
图5是根据本公开实施例的算子运行***结构示意图;
图6是根据本公开另一实施例的算子运行***结构示意图;
图7是根据本公开另一实施例的算子运行***结构示意图;
图8是根据本公开一个例子的算子运行数据流示意图;
图9是根据本公开实施例的算子开发装置的原理框图;
图10是根据本公开实施例的算子处理装置的原理框图;
图11是根据本公开实施例的电子设备的原理框图。
现在将参照附图来详细描述本公开的各种示例性实施例。应注意到:除非另外具体说明,否则在这些实施例中阐述的部件和步骤的相对布置、数字表达式和数值不限制本公开的范围。
以下对至少一个示例性实施例的描述实际上仅仅是说明性的,决不作为对本公开及其应用或使用的任何限制。
对于相关领域普通技术人员已知的技术、方法和设备可能不作详细讨论,但在适当情况下,所述技术、方法和设备应当被视为说明书的一部分。
在这里示出和讨论的所有例子中,任何具体值应被解释为仅仅是示例性的,而不是作为限制。因此,示例性实施例的其它例子可以具有不同的值。
应注意到:相似的标号和字母在下面的附图中表示类似项,因此,一旦某一项在一个附图中被定义,则在随后的附图中不需要对其进行进一步讨论。
<硬件配置>
本公开实施例的方法可以由至少一台电子设备实施,即,用于实施算子开发方法的装置900或者实施算子处理方法的装置1000可以布置在该至少一台电子设备上。图1示出了任意电子设备的硬件结构。图1所示的电子设备可以是便携式电脑、台式计算机、工作站、服务器等,也可以是任意的具有处理器等计算装置和存储器等存储装置的其他设备,在此不做限定。
如图1所示,该电子设备1000可以包括处理器1100、存储器1200、接口装置1300、通信装置1400、显示装置1500、输入装置1600、扬声器1700、麦克风1800等等。其中,处理器1100被配置为执行计算机程序。该计算机程序可以采用比如x86、Arm、RISC、MIPS、SSE等架构的指令集编写。存储器1200例如包括ROM(只读存储器)、RAM(随机存取存储器)、诸如硬盘的非易失性存储器等。接口装置1300例如包括USB接口、耳机接口等。通信装置1400例如能够进行有线或无线通信,具体地可以包括Wifi通信、蓝牙通信、2G/3G/4G/5G通信等。显示装置1500例如是液晶显示屏、触摸显示屏等。输入装置1600例如可以包括触摸屏、键盘、体感输入等。电子设备1000可以通过扬声器1700输出语音信息,及可以通过麦克风1800采集语音信息等。
图1所示的电子设备仅仅是说明性的并且决不意味着对本公开、其应用或使用的任何限制。应用于本公开的实施例中,电子设备1000的所述存储器1200被配置为存储指令,所述指令被配置为控制所述处理器1100进行操作以执行本公开实施例的算子开发方法或者算子处理方法。技术人员可以根据本公开所公开方案设计指令。指令如何控制处理器进行操作,这是本领域公知,故在此不再详细描述。
在一个实施例中,提供了一种包括至少一个计算装置和至少一个存储装置的电子设备,该至少一个存储装置被配置为存储指令,该指令被配置为控制该至少一个计算装置执行根据本公开任意实施例的方法。
该设备可以包括至少一台图1所示的电子设备1000,以提供至少一个例如是处理器的计算装置和至少一个例如是存储器的存储装置,在此不做限定。
<方法实施例一>
图2是根据本公开实施例的算子开发方法的流程示意图,该方法由电子设备1000执行,如图2所示,该方法可以包括如下步骤S2100~S2400:
步骤S2100,基于用户在开发界面上的第一操作信息,确定目标算子包的描述文件。
算子可以是开发者根据需求开发出的功能模组,该算子可以包括但不限于以下至少一种:模型训练算子、模型预测算子、特征提取算子、模型评估算子,当然,算子也可以是开发者根据客户需求所开发出的自定义算子,自定义算子的功能和应用场景均可由客户自定义设计,本实施例在此不做限定。
所述描述文件被配置为描述算子基础能力与运行能力,所述运行能力至少包括框架信息与开发语言,所述框架信息被配置为指示算子所支持的运行框架。该描述文件的格式为yaml,例如,该描述文件可以命名为Operator.yaml,以便于与其他文件进行区分。
其中,所述框架信息被配置为描述算子所支持的运行框架,本公开实施例所涉及到的运行框架可以包括但不限于以下至少一项:spark、flink、horovod、dask、mars、mindspore、local、k8s、hadoop、SAAS。本实施例中,可以根据需要在描述文件定义不同的运行框架,进而使得所开发出的目标算子能够与所定义的运行框架适配。更进一步地,也可以在描述文件中定义有两个以上的运行框架时,所开发出的目标算子可以同时运行在该两个以上运行框架上,进而使得目标算子能够实现跨运行框架运行。换言之,通过本公开实施例所提供的算子开发方法,降低了算子使用门槛,算子的可迁移性较高,能够运行于多种运行框架,适用范围较广。
其中,本公开实施例所涉及到的所述开发语言可以包括但不限于以下至少一项:Python语言、golang语言、c语言、c++语言、Java语言。本实施例中,可以根据需要在描述文件中定义不同的开发语言,进而基于所定义的开发语言进行算子的开发,能兼容多语言开发的算子。
其中,所述算子的运行能力还可以包括但不限于以下至少一项:运行类型、规模、负载模式、底层平台、管理模式、依赖、输出数据数据集、状态、架构。运行类型可以包括但不限于在线类型算子和批量类型算子,该在线类型算子通常能够提供在线预测服务,在线类型算子可以包括但不限于服务、流式。该批量类型算子通常能够提供批量预测服务。
规模可以包括但不限于单机算子、分布式算子。负载模式可以包括但不限于远程(remote)、本地(local)。底层平台可以包括但不限于以下至少一项:k8s(kubernetes)、yarn、hadoop。管理模式可以包括但不限于托管(通过admission hook方式watch,无侵入性,适合原生k8s类型)、状态上报(非原生框架,第三方k8s CRD或者其他框架)等。架构可以包括但不限于x86、arm。
依赖可以包括但不限于ray、dask、horovod。状态可以包括但不限于同步状态信息syncStatus和运行状态runningPhase,其中,同步状态信息syncStatus可以包括但不限于未同步unsynced、同时发生syncing、同步synced、未知unknown。运行状态runningPhase可以包括但不限于等待pending、运行running、成功succeeded、转移skipped、失败failed、错误error。
其中,所述算子基础能力包括但不限于:输入、输出、运行参数。该输入被配置为描述算子运行时的输入信息,该输入可以包括但不限于源类型、配置、输入数据。该输出被配置为描述算子运行时的输出信息,该输出可以包括但不限于输出类型、配置、输出数据。运行参数是算子运行时所需要的必要业务参数,该必要业务参数可以包括但不限于并发数、训练轮数和模型类型。本实施例中,通过在描述文件中定义算子运行时所需要的必要业务参数,可以使得算子稳定的运行,避免算子运行失败、进而提升算子的运行能力。
应当理解,前述针对描述文件中所包含的各项内容,以及,各项内容所支持的具体内容,都仅为示例性的,实际场景中,还可以基于实际情况,对描述文件中的各项内容进行适应性的增减。
在一个例子中,用户在开发界面上的第一操作信息可以是用户针对电子设备所输出的描述文件的选择操作、确定操作或编辑操作等操作的相关信息,例如,电子设备输出的描述文件可以是初始描述文件,该初始描述文件可以是预先存储至电子设备中的文件,在此,
开发者可以直接进行确定操作,将该初始描述文件确定为目标算子包的描述文件;或者,开发者可以根据自身需求对该初始描述文件进行编辑操作,进而得到目标算子包的描述文件。又例如,电子设备可以输出多种描述文件,用户就可以在该界面上进行选择,以确定目标算子包的描述文件。这种基于电子设备所输出的描述文件去得到目标算子包的描述文件的方式,算子开发效率高。
在一个例子中,用户在开发界面上的第一操作信息也可以是用户手动编写的目标算子包的描述文件,这种通过手动编写目标算子包的描述文件的方式,更加符合用户需求,也能够满足用户的自定义设计需求,以及,在电子设备预设的描述文件不满足用户需求时,为用户提供自定义设计描述文件的补救措施。
在执行基于用户在开发界面上的第一操作信息,确定目标算子包的描述文件之后,进入:
步骤S2200,确定与所述描述文件相匹配的算子标准模板。
算子标准模板至少包括:算子的描述文件与算子源代码。可以理解的是,算子标准模板可以具体化为脚手架,脚手架可以称之为代码目录结构和入口函数,本实施例中,脚手架所起的作用为算子标准模板。
其中,算子源代码是为实现算子功能所编写的代码。例如,特征提取算子被配置为对输入的数据进行特征提取,对应的,特征提取算子源代码即为实现对输入的数据进行特征提取这一功能所编写的代码。又例如,模型训练算子被配置为根据特征提取的结果进行模型训练得到机器学习模型,对应的,模型训练算子源代码即为实现根据特征提取的结果进行模型训练得到机器学习模型这一功能所编写的代码。再例如,模型评估算子被配置为对训练好的机器学习模型的性能进行评估,对应的,模型评估算子源代码即为实现对训练好的机器学习模型的性能进行评估这一功能所编写的代码。
在以上的任意一个实施例中,算子标准模板除包括描述文件与算子源代码之外,还包括进一步以下至少一项:算子的说明文档、算子的测试文件。
其中,算子的说明文档通常是为方便客户、普通人员能够正常使用、运行、调试算子所编写的文档,进而降低算子应用的门槛。
其中,算子的测试文件可以包括但不限于测试用例和测试报告模板。测试用例是针对算子进行测试任务的描述,体现测试方案、方法和策略,其内容可以包括但不限于测试目标、测试环境、输入数据、测试步骤、预期结果、测试脚本。测试报告模板是针对算子执行测试用例所得到的测试报告。本实施例中,通过在算子标准模板规定算子的测试文件,可以提升算子质量,以便在开发出目标算子之后,可通过自动测试和人工审核相结合的方式进行算子评级。
在一个实施例中,本步骤S2200确定与所述描述文件相匹配的算子标准模板可以进一步包括:在预设的算子标准模板中,选择与所述框架信息与所述开发语言相匹配的算子标准模板。
本实施例中,预设的算子标准模板中包括多个算子标准模板,例如但不限于包括算子标准模板1、算子标准模板2。可以理解的是,任意两种不同的开发语言,其各自对应的算子标准模板的类别不同,且不同类别的算子标准模板不同,具体而言,算子标准模板中各大项(描述文件、算子源代码、说明文档、测试文件)里面包含的内容或内容的生成方式不同。例如,开发语言python对应一个类别的算子标准模板,开发语言c++对应一个类别的算子标准模板,且这两种类别的算子标准模板中各大项里面包含的内容或内容的生成方式不同。
本实施例中,框架信息和开发语言与算子标准模板具有映射关系,也就是说,框架信息和开发语言不同,所对应的算子标准模板不同,这样,在确定目标算子包的描述文件的情况下,便可根据描述文件中的框架信息和开发语言,从预设的算子标准模板中确定出与该框架信息和开发语言对应的算子标准模板,可以准确的确定出算子标准模板。
在一个实施例中,本步骤S2200确定与所述描述文件相匹配的算子标准模板也可以进一步包括:基于所述框架信息与所述开发语言,生成与之匹配的算子标准模板。
本实施例中,在确定目标算子包的描述文件的情况下,电子设备也可以直接根据描述文件中的框架信息和开发语言,自动生成与该框架信息和开发语言相匹配的算子标准模板,可以高效的生成算子标准模板。
在执行确定与所述描述文件相匹配的算子标准模板之后,进入:
步骤S2300,基于所述用户在所述开发界面上的第二操作信息,填充所述算子标准模板,得到所述目标算子。
本实施例中,在确定出与所述描述文件相匹配的算子标准模板之后,便可基于用户在开发界面上的第二操作信息,填充该算子标准模板以得到目标算子。
在执行基于所述用户在所述开发界面上的第二操作信息,填充所述算子标准模板,得到所述目标算子之后,进入:
步骤S2400,将所述目标算子打包并存入标准算子库。
本实施例中,本步骤S2400中将所述目标算子打包并存入标准算子库可以进一步包括:验证所述目标算子是否满足算子准入机制;当所述目标算子满足所述算子准入机制时,将所述目标算子打包并存入标准算子库。
示例性的,算子准入机制可以为一种自动测试机制。一种可能的实施例中,算子准入机制可以与算子的测试文件相关。在实际应用中,可以利用算子的测试文件对算子进行测试,当其通过测试,则可以确定满足该算子准入机制。
或者,算子准入机制也可以由业务人员进行人为把控。另一种可能的实施例中,当得到目标算子后,可通过人为把控的方式,来确定目标算子是否满足算子准入机制,即,为业务人员人为把控算子质量,提供了一种可能。
当然,具体应用时,自动测试机制与人为把控机制也可以组合应用,当二者均通过时,则目标算子满足算子准入机制。
此外,算子准入机制也可以是根据实际需求进行设置,本实施例并不限定算子准入机制的具体内容。也就是说,其在得到目标算子之后,并不是直接将目标算子并入标准算子库,而是先基于算子准入机制对目标算子进行验证,并在验证通过的情况下,才将目标算子打包并存入标准算子库,进一步提升算子质量,保证标准算子库中的算子水平的一致性,也会相应的降低算子开发成本。
而且,标准化后的算子既可以在集群中运行,又可以在本地运行,且有同样的功能体验,这降低了算子研发人员需要学习多种分布式集群的学习成本。
此外,若目标算子不满足算子准入机制,则退回开发人员处,以便于开发人员对其进行修改或调整,开发人员调整后的目标算子,仍可按照图2所示的方法来进行开发。
根据本公开实施例,其是通过目标算子包的描述文件和算子标准模板进行目标算子的开发,其中,目标算子的描述文件被配置为描述算子基础能力与运行能力,运行能力至少包括框架信息与开发语言,框架信息被配置为指示算子所支持的运行框架,其中,算子标准模板包括算子的描述文件和算子源代码。具体进行目标算子开发时,需要确定出目标算子包的描述文件,然后确定与该描述文件匹配的算子标准模板,并填充该算子标准模板以得到目标算子。即,其是通过配置算子标准模板的方式进行目标算子的开发,能够降低开发算子时的重复工作,提升了算子的适配性,大幅降低机器学习落地难度,降低了开发成本和时间成本。
需要说明的是,本公开实施例中所涉及到的算子,还可被用于隐私计算、多方安全计算、联邦学习、匿踪(隐匿)查询、安全(隐私)求交等场景。
接下来示出一个例子的算子开发方法的流程示意图,该例子中,结合图3所示,该算子开发方法可以包括如下步骤S310~步骤S390:
步骤S310,基于算子开发者在开发界面上的第一操作信息,生成目标算子包的描述文
件Operator.yaml。
步骤S320,根据描述文件Operator.yaml中定义的开发语言和运行框架,自动生成或手动编写脚手架代码。
步骤S330,对脚手架代码进行opctl本地调试。
步骤S340,填充脚手架代码得到目标算子。
步骤S350,将目标算子打包并上传至持续集成工具(CI工具)。
步骤S360,基于算子的测试文件在CI工具中对目标算子进行自动化测试,得到自动化测试结果。
步骤S370,算子开发者基于自动化测试结果在OpHub***中对目标算子进行调试,并针对调试后的目标算子编写测试文件。
步骤S380,在CI工具中验证目标算子满足算子准入机制的情况下,质量保证人员(QA人员)基于所编写的测试文件对目标算子进行人工测试,并在测试通过的情况下,通过CI工具发版(release)打包后的目标算子。
步骤S390,将目标算子落入标准算子库即制品库。
根据本例子,研发过程中,算子标准化对研发带来的好处包括:首先,其基于生成脚手架代码的方式进行目标算子的开发,能够降低开发算子时的重复工作,提升了算子的适配性和使用范围,大幅降低机器学习落地难度,降低了开发成本和时间成本。而且,全生命周期管理,提升算子开发效率,相比于传统算子开发,开发时间缩小50%。其次,标准化后的算子既可以在集群中运行,也可以在本地运行,这降低了算子开发者需要学习多种分布式集群的学习成本。
算子标准化对产品管理人员带来的好处包括但不限于:通过算子的测试文件对算子进行测试后才落入制品库,通过接入自动化测试的方式,设计易用的算子准入机制进提升算子质量。
<方法实施例二>
图4是根据本公开实施例的算子处理方法的流程示意图,该方法由电子设备1000执行,具体可以是由电子设备中的OpHub***执行,或者,也可以电子设备中的其他模块或***执行,其中,这些模块或***可以按照如图4所示的方法来对算子进行调度应用,具体而言,该方法可以包括如下步骤S4100~S4300:
步骤S4100,从标准算子库获取目标算子。
目标算子可以是根据上述实施例一开发得到的标准算子库中的其中一个或多个算子。本实施例中,可以从标准算子库中获取所需要的一个或多个目标算子。参照图5,可以基于本地集成开发环境(local IDE)或者网络集成开发环境(web IDE)开发目标算子并上传至标准算子库(git repo),以便本***(例如OpHub)可以从git rego中获取目标算子。
所述目标算子至少包括算子的描述文件与算子源代码;所述描述文件被配置为描述算子基础能力与运行能力,所述运行能力至少包括框架信息与开发语言,所述框架信息被配置为指示算子所支持的运行框架。
其中,所述框架信息所支持的运行框架可以包括但不限于以下至少一项:spark、flink、horovod、dask、mars、mindspore、local、k8s、hadoop、SAAS。本实施例中,在描述文件中定义两个以上的运行框架时,所开发出的目标算子可以同时运行在该两个以上运行框架上,进而使得目标算子能够实现跨运行框架运行,提升适用范围和适配性,分布式算子开发的开发体验提升。
其中,本公开实施例所涉及的所述开发语言可以包括但不限于以下至少一项:Python语言、golang语言、c语言、c++语言、Java语言。本实施例中,可以根据需要在描述文件中定义不同的开发语言,进而基于所定义的开发语言进行算子的开发,能兼容多语言开发的算子。
所述算子的运行能力还可以包括但不限于以下至少一项:运行类型、规模、负载模式、
底层平台、管理模式、依赖、输出数据数据集、状态、架构。具体可以参照算子开发实施例的介绍,本实施例在此不做赘述。
所述算子基础能力包括但不限于:输入、输出、运行参数。具体可以参照算子开发实施例的介绍,本实施例在此不做赘述。
算子源代码是为实现算子功能所编写的代码。例如,特征提取算子被配置为对输入的数据进行特征提取,对应的,特征提取算子源代码即为实现对输入的数据进行特征提取这一功能所编写的代码。又例如,模型训练算子被配置为根据特征提取的结果进行模型训练得到机器学习模型,对应的,模型训练算子源代码即为实现根据特征提取的结果进行模型训练得到机器学习模型这一功能所编写的代码。再例如,模型评估算子被配置为对训练好的机器学习模型的性能进行评估,对应的,模型评估算子源代码即为实现对训练好的机器学习模型的性能进行评估这一功能所编写的代码。
在执行从标准算子库获取目标算子之后,进入:
步骤S4200,根据目标运行框架所支持的协议,将所述目标算子的描述文件进行转换,获得与所述目标运行框架对应的处理后的目标算子。
目标运行框架为算子的描述文件中算子的框架信息所指示的算子所支持的运行框架。可以理解的是,目标运行框架可以仅有一个,目标运行框架也可以有多个,具体可以参照上述介绍。
本实施例中,本步骤S4200中根据目标运行框架所支持的协议,将所述目标算子的描述文件进行转换,获得与所述目标运行框架对应的处理后的目标算子可以进一步包括:根据所述目标运行框架所支持的协议,将所述目标标准的描述文件解析为所述协议可读的第二文件;根据所述第二文件,得到与所述目标运行框架对应的处理后的目标算子。
可以理解的是,由于不同运行框架均有所支持的协议,为了保证目标运行框架能够正常运行目标算子,需要先根据目标运行框架所支持的协议,将目标算子的描述文件解析为其协议可读的文件,然后根据该文件得到目标运行框架对应的处理后的目标算子。参照图5,所提供的运行框架可以包括有local、k8s、hadoop、SAAS……(不作穷举),在此,目标运行框架可以为hadoop和SAAS,便可根据hadoop所支持的协议该特征提取算子的描述文件解析为其可读的文件,得到hadoop对应的处理后的特征提取算子,以及根据SAAS所支持的协议该特征提取算子的描述文件解析为其可读的文件,得到SAAS对应的处理后的特征提取算子。可以理解的是,图5中的任务追踪被配置为在算子执行任务过程中,对任务运行状态进行追踪,该任务追踪框架可以包括但不限于:xxx Platform、tfBorad、ES、Grafana、log、tracing。
步骤S4300,将与所述目标运行框架对应的处理后的目标算子发送至对应的运行框架,以在对应的运行框架中运行所述目标算子。
继续上述示例,得到hadoop对应的处理后的特征提取算子,便可将处理后的特征提取算子发送至hadoop,以便在hadoop运行处理后的特征提取算子。以及得到SAAS对应的处理后的特征提取算子,便可将处理后的特征提取算子发送至SAAS,以便在SAAS运行处理后的特征提取算子。
根据本公开实施例,其所获取的目标算子是从标准算子库中获取的,该目标算子包括算子的描述文件,描述文件被配置为描述算子运行能力,且运行能力至少包括框架信息与开发语言,框架信息被配置为指示算子所支持的运行框架。即,其所获取的目标算子可以根据需求进行框架信息的配置,提升了算子的适配性,并且,在描述文件中定义有不同运行框架的情况下,可以使得目标算子具备跨运行框架的能力,即该目标算子可以与不同的运行框架适配,进而运行在不同的运行框架中,实现跨运行框架。
在一个实施例中,在目标算子包括多个的情况下,在执行以上步骤S3200根据目标运行框架所支持的协议,将所述目标算子的描述文件进行转换,获得与所述目标运行框架对应的处理后的目标算子之前,本公开实施例的算子处理方法还可以进一步包括如下步骤
S5100~步骤S5200:
步骤S5100,通过与编排框架对应的适配器,将所述目标算子的描述文件解析为所述编排框架可读的第一文件。
编排框架可以包括但不限于以下至少一项:workflow、argo workflow、airflow、kubeflow。
本实施例中,涉及多个目标算子进行编排调度的场景中,会涉及到适配器executor,适配器与编排框架一一对应,适配器被配置为实现编排框架与本***的交互,编排框架对应的适配器具体被配置为:按照编排框架的要求,对每一个目标算子的描述文件进行转换,使其成为编排框架可读的文件。示例性地,参照图5,所提供的编排框架包括有kubeflow、airflow、argo workflow、xxx workflow,在此,算子开发者可以选择编排框架workflow,目标算子包括特征提取算子和模型训练算子,在此,可以根据workflow对应的适配器分别将特征提取算子的描述文件进行转换,以及将模型训练算子的描述文件进行转换。
步骤S5200,所述编排框架基于所述目标算子对应的所述第一文件,对所述目标算子进行编排,获得编排后的目标算子。
继续上述示例,编排框架workflow基于特征提取算子对应的第一文件、以及基于模型训练算子对应的第一文件,对特征提取算子和模型训练算子进行编排,并将编排后的特征提取算子和模型训练算子通过OpHub***进行协议转化并发送至运行框架运行。
可以理解的是,在对多个目标算子进行编排之后,便可由本***根据目标运行框架所支持的协议,将目标算子的描述文件进行转换,获得与目标运行框架对应的处理后的目标算子,并将处理后的目标算子发送至对应的运行框架,以在对应的运行框架中运行目标算子。
根据本公开实施例,算子运行时与编排调度解耦,针对不同的编排框架,开发不同的适配器,通过适配器将目标算子提交给***。
<算子运行***>
本公开实施例还提供一种算子运行***,参照图6,该算子运行***600包括代理器610、状态***620和任务配置器630。该算子运行***600可以是OpHub***。
本实施例中,代理器(transmitter)610,被配置为代理与其他***的交互,所述其他***包括:算子仓库、算子运行框架。
该算子仓库即为以上实施例所述的标准算子库,该算子仓库为图7所示的nexus repository。
该算子运行框架可以包括但不限于以下至少一项:spark、flink、horovod、dask、mars、mindspore。图7所示的为运行框架kubernetes。
在任意的一个实施例中,该其他***除包括算子仓库和算子运行框架之外,还可以进一步包括服务框架、管理框架。该服务框架为算子运行所需要依赖的数据框架,该服务框架可以包括但不限于图7所示的elasticsearch、MySQL。该管理框架被配置为管理任务,该管理框架可以是图7所示的分布式版本控制***git。
本实施例中,状态***(watcher)620,被配置为跟踪任务状态并记录。参照图7,状态***620可以监控kubernetes中的任务,并记录至elasticsearch、MySQL,根据具体情况状态追踪器620中包含日志监控采集组件,或者对接第三方。
任务配置器630(addmission webhook),被配置为监听任务并作任务配置。参照图7,任务配置器630可以监听提交至kubernetes中的任务,做集群级别、namespace级别的统一配置。
可以理解的,以上代理器610、状态***620和任务配置器630提供暴露数据的能力,具体数据的存储可依据现场依赖灵活选择。
结合图8所示,示出一个例子的算子运行数据流程示意图,图8中,先提交算子执行任务,并提交任务至算子运行框架platform,并在算子运行框架中执行任务,在算子执行任务的过程中,可以通过状态追踪器620中的meta/status、log、trace在算子运行时进行信息
收集,并查看算子运行时信息。
<装置实施例一>
本公开实施例还提供一种算子开发装置900,如图9所示,该算子开发装置900包括第一确定模块910、第二确定模块920、填充模块930、打包模块940。
第一确定模块910,被配置为基于用户在开发界面上的第一操作信息,确定目标算子包的描述文件,所述描述文件被配置为描述算子基础能力与运行能力,所述运行能力至少包括框架信息与开发语言,所述框架信息被配置为指示算子所支持的运行框架。
第二确定模块920,被配置为确定与所述描述文件相匹配的算子标准模板;其中,所述算子标准模板至少包括:算子的描述文件与算子源代码。
填充模块930,被配置为基于所述用户在所述开发界面上的第二操作信息,填充所述算子标准模板,得到所述目标算子。
打包模块940,被配置为将所述目标算子打包并存入标准算子库。
在一个实施例中,所述第二确定模块920,具体被配置为:
在预设的算子标准模板中,选择与所述框架信息与所述开发语言相匹配的算子标准模板;
或者,基于所述框架信息与所述开发语言,生成与之匹配的算子标准模板。
在一个实施例中,所述算子标准模板还包括以下至少一项:算子的说明文档、算子的测试文件。
在一个实施例中,所述算子基础能力包括:输入、输出、运行参数。
在一个实施例中,所述算子运行能力还包括以下至少一项:运行类型、规模、负载模式、底层平台、管理模式、依赖、输出数据数据集、状态、架构。
在一个实施例中,所述框架信息所支持的运行框架包括以下至少一项:spark、flink、horovod、dask、mars、mindspore。
在一个实施例中,所述打包模块940,具体被配置为:验证所述目标算子是否满足算子准入机制;当所述目标算子满足所述算子准入机制时,将所述目标算子打包并存入标准算子库。
根据本公开实施例,其是通过目标算子包的描述文件和算子标准模板进行目标算子的开发,其中,目标算子的描述文件被配置为描述算子基础能力与运行能力,运行能力至少包括框架信息与开发语言,框架信息被配置为指示算子所支持的运行框架,其中,算子标准模板包括算子的描述文件和算子源代码。具体进行目标算子开发时,需要确定出目标算子包的描述文件,然后确定与该描述文件匹配的算子标准模板,并填充该算子标准模板以得到目标算子。即,其是通过配置算子标准模板的方式进行目标算子的开发,能够降低开发算子时的重复工作,提升了算子的适配性,大幅降低机器学习落地难度,降低了开发成本和时间成本。
<装置实施例二>
本公开实施例还提供一种算子处理装置1000,如图10所示,该算子处理装置1000包括获取模块1010、转换模块1020、发送模块1030。
获取模块1010,被配置为从标准算子库获取目标算子;其中,所述目标算子至少包括算子的描述文件与算子源代码;所述描述文件被配置为描述算子基础能力与运行能力,所述运行能力至少包括框架信息与开发语言,所述框架信息被配置为指示算子所支持的运行框架。
转换模块1020,被配置为根据目标运行框架所支持的协议,将所述目标算子的描述文件进行转换,获得与所述目标运行框架对应的处理后的目标算子。
发送模块1030,被配置为将与所述目标运行框架对应的处理后的目标算子发送至所述目标运行框架,以在所述目标运行框架中运行所述目标算子。
在一个实施例中,所述装置还包括编排模块(图中未示出),具体被配置为:通过与编
排框架对应的适配器,将所述目标算子的描述文件解析为所述编排框架可读的第一文件;所述编排框架基于根据所述目标算子对应的所述第一文件,对所述目标算子进行编排,得到编排后的目标算子。
在一个实施例中,所述转换模块1020,具体被配置为:根据所述目标运行框架所支持的协议,将所述目标标准的描述文件解析为所述协议可读的第二文件;根据所述第二文件,得到与所述目标运行框架对应的处理后的目标算子。
在一个实施例中,所述算子标准模板还包括以下至少一项:算子的说明文档、算子的测试文件。
在一个实施例中,所述算子基础能力包括:输入、输出、运行参数。
在一个实施例中,所述算子运行能力还包括以下至少一项:运行类型、规模、负载模式、底层平台、管理模式、依赖。
在一个实施例中,所述框架信息所支持的运行框架包括以下至少一项:spark、flink、horovod、dask、mars、mindspore。
根据本公开实施例,其所获取的目标算子是从标准算子库中获取的,该目标算子包括算子的描述文件,描述文件被配置为描述算子运行能力,且运行能力至少包括框架信息与开发语言,框架信息被配置为指示算子所支持的运行框架。即,其所获取的目标算子可以根据需求进行框架信息的配置,提升了算子的适配性,并且,在描述文件中定义有不同运行框架的情况下,可以使得目标算子具备跨运行框架的能力,即该目标算子可以与不同的运行框架适配,进而运行在不同的运行框架中,实现跨运行框架。
<设备实施例>
本公开实施例提供了一种电子设备1100,该电子设备1100可以是服务器,也可以是终端设备。其中,服务器可以是整体式服务器或是跨多计算机或计算机数据中心的分散式服务器。终端设备可以是手机、平板电脑、笔记本电脑等。
如图11所示,电子设备1100包括处理器1110和存储器1120,该存储器1120被配置为存储可执行的指令;该处理器1110被配置为根据指令的控制运行电子设备1100执行根据本公开任意实施例的算子开发方法或者算子处理方法。
该电子设备可以是具有处理器和存储器的电子产品。例如可以是智能手机、便携式电脑、台式计算机、平板电脑等。
<介质实施例>
本公开实施例提供了一种计算机可读存储介质,其上存储有计算机程序,所述计算机程序在被处理器执行时,实现前述任一实施例提供的算子开发方法或者算子处理方法。
本公开可以是***、方法和/或计算机程序产品。计算机程序产品可以包括计算机可读存储介质,其上载有用于使处理器实现本公开的各个方面的计算机可读程序指令。
计算机可读存储介质可以是可以保持和存储由指令执行设备使用的指令的有形设备。计算机可读存储介质例如可以是――但不限于――电存储设备、磁存储设备、光存储设备、电磁存储设备、半导体存储设备或者上述的任意合适的组合。计算机可读存储介质的更具体的例子(非穷举的列表)包括:便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、静态随机存取存储器(SRAM)、便携式压缩盘只读存储器(CD-ROM)、数字多功能盘(DVD)、记忆棒、软盘、机械编码设备、例如其上存储有指令的打孔卡或凹槽内凸起结构、以及上述的任意合适的组合。这里所使用的计算机可读存储介质不被解释为瞬时信号本身,诸如无线电波或者其他自由传播的电磁波、通过波导或其他传输媒介传播的电磁波(例如,通过光纤电缆的光脉冲)、或者通过电线传输的电信号。
这里所描述的计算机可读程序指令可以从计算机可读存储介质下载到各个计算/处理设备,或者通过网络、例如因特网、局域网、广域网和/或无线网下载到外部计算机或外部存储设备。网络可以包括铜传输电缆、光纤传输、无线传输、路由器、防火墙、交换机、
网关计算机和/或边缘服务器。每个计算/处理设备中的网络适配卡或者网络接口从网络接收计算机可读程序指令,并转发该计算机可读程序指令,以供存储在各个计算/处理设备中的计算机可读存储介质中。
用于执行本公开操作的计算机程序指令可以是汇编指令、指令集架构(ISA)指令、机器指令、机器相关指令、微代码、固件指令、状态设置数据、或者以一种或多种编程语言的任意组合编写的源代码或目标代码,所述编程语言包括面向对象的编程语言—诸如Smalltalk、C++等,以及常规的过程式编程语言—诸如“C”语言或类似的编程语言。计算机可读程序指令可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中,远程计算机可以通过任意种类的网络—包括局域网(LAN)或广域网(WAN)—连接到用户计算机,或者,可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。在一些实施例中,通过利用计算机可读程序指令的状态信息来个性化定制电子电路,例如可编程逻辑电路、现场可编程门阵列(FPGA)或可编程逻辑阵列(PLA),该电子电路可以执行计算机可读程序指令,从而实现本公开的各个方面。
这里参照根据本公开实施例的方法、装置(***)和计算机程序产品的流程图和/或框图描述了本公开的各个方面。应当理解,流程图和/或框图的每个方框以及流程图和/或框图中各方框的组合,都可以由计算机可读程序指令实现。
这些计算机可读程序指令可以提供给通用计算机、专用计算机或其它可编程数据处理装置的处理器,从而生产出一种机器,使得这些指令在通过计算机或其它可编程数据处理装置的处理器执行时,产生了实现流程图和/或框图中的一个或多个方框中规定的功能/动作的装置。也可以把这些计算机可读程序指令存储在计算机可读存储介质中,这些指令使得计算机、可编程数据处理装置和/或其他设备以特定方式工作,从而,存储有指令的计算机可读介质则包括一个制造品,其包括实现流程图和/或框图中的一个或多个方框中规定的功能/动作的各个方面的指令。
也可以把计算机可读程序指令加载到计算机、其它可编程数据处理装置、或其它设备上,使得在计算机、其它可编程数据处理装置或其它设备上执行一系列操作步骤,以产生计算机实现的过程,从而使得在计算机、其它可编程数据处理装置、或其它设备上执行的指令实现流程图和/或框图中的一个或多个方框中规定的功能/动作。
附图中的流程图和框图显示了根据本公开的多个实施例的***、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上,流程图或框图中的每个方框可以代表一个模块、程序段或指令的一部分,所述模块、程序段或指令的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。在有些作为替换的实现中,方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如,两个连续的方框实际上可以基本并行地执行,它们有时也可以按相反的顺序执行,这依所涉及的功能而定。也要注意的是,框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合,可以用执行规定的功能或动作的专用的基于硬件的***来实现,或者可以用专用硬件与计算机指令的组合来实现。对于本领域技术人员来说公知的是,通过硬件方式实现、通过软件方式实现以及通过软件和硬件结合的方式实现都是等价的。
以上已经描述了本公开的各实施例,上述说明是示例性的,并非穷尽性的,并且也不限于所披露的各实施例。在不偏离所说明的各实施例的范围和精神的情况下,对于本技术领域的普通技术人员来说许多修改和变更都是显而易见的。本文中所用术语的选择,旨在最好地解释各实施例的原理、实际应用或对市场中的技术改进,或者使本技术领域的其它普通技术人员能理解本文披露的各实施例。本公开的范围由所附权利要求来限定。
通过本公开实施例,通过目标算子包的描述文件和算子标准模板进行目标算子的开发,
其中,目标算子的描述文件被配置为描述算子基础能力与运行能力,运行能力至少包括框架信息与开发语言,框架信息被配置为指示算子所支持的运行框架,其中,算子标准模板包括算子的描述文件和算子源代码。具体进行目标算子开发时,需要确定出目标算子包的描述文件,然后确定与该描述文件匹配的算子标准模板,并填充该算子标准模板以得到目标算子。即,其是通过配置算子标准模板的方式进行目标算子的开发,能够降低开发算子时的重复工作,提升了算子的适配性,大幅降低机器学习落地难度,降低了开发成本和时间成本。
Claims (31)
- 一种算子开发方法,所述方法包括:基于用户在开发界面上的第一操作信息,确定目标算子包的描述文件,所述描述文件被配置为描述算子基础能力与运行能力,所述运行能力至少包括框架信息与开发语言,所述框架信息被配置为指示算子所支持的运行框架;确定与所述描述文件相匹配的算子标准模板;其中,所述算子标准模板至少包括:算子的描述文件与算子源代码;基于所述用户在所述开发界面上的第二操作信息,填充所述算子标准模板,得到所述目标算子;将所述目标算子打包并存入标准算子库。
- 根据权利要求1所述的方法,其中,所述确定与所述描述文件相匹配的算子标准模板,包括:在预设的算子标准模板中,选择与所述框架信息与所述开发语言相匹配的算子标准模板;或者,基于所述框架信息与所述开发语言,生成与之匹配的算子标准模板。
- 根据权利要求2所述的方法,其中,所述算子标准模板还包括以下至少一项:算子的说明文档、算子的测试文件。
- 根据权利要求1-3任一项所述的方法,其中,所述算子基础能力包括:输入、输出、运行参数。
- 根据权利要求1-3任一项所述的方法,其中,所述算子运行能力还包括以下至少一项:运行类型、规模、负载模式、底层平台、管理模式、依赖、输出数据数据集、状态、架构。
- 根据权利要求1-3任一项所述的方法,其中,所述框架信息所支持的运行框架包括以下至少一项:spark、flink、horovod、dask、mars、mindspore、local、k8s、hadoop、SAAS。
- 根据权利要求1-3任一项所述的方法,其中,所述将所述目标算子打包并存入标准算子库,包括:验证所述目标算子是否满足算子准入机制;当所述目标算子满足所述算子准入机制时,将所述目标算子打包并存入标准算子库。
- 一种算子处理方法,所述方法包括:从标准算子库获取目标算子;其中,所述目标算子至少包括算子的描述文件与算子源代码;所述描述文件被配置为描述算子基础能力与运行能力,所述运行能力至少包括框架信息与开发语言,所述框架信息被配置为指示算子所支持的运行框架;根据目标运行框架所支持的协议,将所述目标算子的描述文件进行转换,获得与所述目标运行框架对应的处理后的目标算子;将与所述目标运行框架对应的处理后的目标算子发送至所述目标运行框架,以在所述目标运行框架中运行所述目标算子。
- 根据权利要求8所述的方法,其中,在所述根据目标运行框架所支持的协议,将所述目标算子的描述文件进行转换,获得与所述目标运行框架对应的处理后的目标算子之前,还包括:通过与编排框架对应的适配器,将所述目标算子的描述文件解析为所述编排框架可读的第一文件;通过所述编排框架,基于所述目标算子对应的所述第一文件,对所述目标算子进行编排,得到编排后的目标算子。
- 根据权利要求9所述的方法,其中,所述根据目标运行框架所支持的协议,将所述目标算子的描述文件进行转换,获得与所述目标运行框架对应的处理后的目标算子,包 括:根据所述目标运行框架所支持的协议,将所述目标标准的描述文件解析为所述协议可读的第二文件;根据所述第二文件,获取与所述目标运行框架对应的处理后的目标算子。
- 根据权利要求8-10任一项所述的方法,其中,所述算子标准模板还包括以下至少一项:算子的说明文档、算子的测试文件。
- 根据权利要求8-10任一项所述的方法,其中,所述算子基础能力包括:输入、输出、运行参数。
- 根据权利要求8-10任一项所述的方法,其中,所述算子运行能力还包括以下至少一项:运行类型、规模、负载模式、底层平台、管理模式、依赖、输出数据数据集、状态、架构。
- 根据权利要求8-10任一项所述的方法,其中,所述框架信息所支持的运行框架包括以下至少一项:spark、flink、horovod、dask、mars、mindspore、local、k8s、hadoop、SAAS。
- 一种算子开发装置,所述装置包括:第一确定模块,被配置为基于用户在开发界面上的第一操作信息,确定目标算子包的描述文件,所述描述文件被配置为描述算子基础能力与运行能力,所述运行能力至少包括框架信息与开发语言,所述框架信息被配置为指示算子所支持的运行框架;第二确定模块,被配置为确定与所述描述文件相匹配的算子标准模板;其中,所述算子标准模板至少包括:算子的描述文件与算子源代码;填充模块,被配置为基于所述用户在所述开发界面上的第二操作信息,填充所述算子标准模板,得到所述目标算子;打包模块,被配置为将所述目标算子打包并存入标准算子库。
- 根据权利要求15所述的装置,其中,所述第二确定模块,具体被配置为:在预设的算子标准模板中,选择与所述框架信息与所述开发语言相匹配的算子标准模板;或者,基于所述框架信息与所述开发语言,生成与之匹配的算子标准模板。
- 根据权利要求16所述的装置,其中,所述算子标准模板还包括以下至少一项:算子的说明文档、算子的测试文件。
- 根据权利要求15-17任一项所述的装置,其中,所述算子基础能力包括:输入、输出、运行参数。
- 根据权利要求15-17任一项所述的装置,其中,所述算子运行能力还包括以下至少一项:运行类型、规模、负载模式、底层平台、管理模式、依赖、输出数据数据集、状态、架构。
- 根据权利要求15-17任一项所述的装置,其中,所述框架信息所支持的运行框架包括以下至少一项:spark、flink、horovod、dask、mars、mindspore、local、k8s、hadoop、SAAS。
- 根据权利要求15-17任一项所述的装置,其中,所述打包模块,具体被配置为:验证所述目标算子是否满足算子准入机制;当所述目标算子满足所述算子准入机制时,将所述目标算子打包并存入标准算子库。
- 一种算子处理装置,所述装置包括:获取模块,被配置为从标准算子库获取目标算子;其中,所述目标算子至少包括算子的描述文件与算子源代码;所述描述文件被配置为描述算子基础能力与运行能力,所述运行能力至少包括框架信息与开发语言,所述框架信息被配置为指示算子所支持的运行框架;转换模块,被配置为根据目标运行框架所支持的协议,将所述目标算子的描述文件进行转换,获得与所述目标运行框架对应的处理后的目标算子;发送模块,被配置为将与所述目标运行框架对应的处理后的目标算子发送至所述目标运行框架,以在所述目标运行框架中运行所述目标算子。
- 根据权利要求22所述的装置,其中,所述装置还包括编排模块,具体被配置为:通过与编排框架对应的适配器,将所述目标算子的描述文件解析为所述编排框架可读的第一文件;通过所述编排框架,基于所述目标算子对应的所述第一文件,对所述目标算子进行编排,得到编排后的目标算子。
- 根据权利要求23所述的装置,其中,所述转换模块,具体被配置为:根据所述目标运行框架所支持的协议,将所述目标标准的描述文件解析为所述协议可读的第二文件;根据所述第二文件,得到与所述目标运行框架对应的处理后的目标算子。
- 根据权利要求22-24任一项所述的装置,其中,所述算子标准模板还包括以下至少一项:算子的说明文档、算子的测试文件。
- 根据权利要求22-24任一项所述的装置,其中,所述算子基础能力包括:输入、输出、运行参数。
- 根据权利要求22-24任一项所述的装置,其中,所述算子运行能力还包括以下至少一项:运行类型、规模、负载模式、底层平台、管理模式、依赖。
- 根据权利要求22-24任一项所述的装置,其中,所述框架信息所支持的运行框架包括以下至少一项:spark、flink、horovod、dask、mars、mindspore、local、k8s、hadoop、SAAS。
- 一种包括至少一个计算装置和至少一个存储装置的电子设备,所述至少一个存储装置被配置为存储指令,所述指令被配置为控制所述至少一个计算装置执行根据权利要求1至14中任一项所述的方法;或者,所述设备通过所述计算装置和所述存储装置实现根据权利要求15至28所述的装置。
- 一种计算机可读存储介质,其上存储有计算机程序,所述计算机程序在被处理器执行时实现如权利要求1至14中任一项所述的方法。
- 一种算子运行***,所述***包括:代理器,被配置为代理与其他***的交互,所述其他***包括:算子仓库、算子运行框架;状态***,被配置为跟踪任务状态并记录;任务配置器,被配置为监听任务并作任务配置。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202210762431.6A CN117348848A (zh) | 2022-06-29 | 2022-06-29 | 算子开发、处理方法、装置、电子设备、***及存储介质 |
CN202210762431.6 | 2022-06-29 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2024001594A1 true WO2024001594A1 (zh) | 2024-01-04 |
Family
ID=89367910
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2023/094913 WO2024001594A1 (zh) | 2022-06-29 | 2023-05-18 | 算子开发、处理方法、装置、电子设备、***及存储介质 |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN117348848A (zh) |
WO (1) | WO2024001594A1 (zh) |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110928529A (zh) * | 2019-11-06 | 2020-03-27 | 第四范式(北京)技术有限公司 | 辅助算子开发的方法和*** |
CN111459474A (zh) * | 2020-03-13 | 2020-07-28 | 北京百分点信息科技有限公司 | 一种模板化的数据处理方法及装置 |
US20210182036A1 (en) * | 2019-12-12 | 2021-06-17 | Huawei Technologies Co., Ltd. | Hardware platform specific operator fusion in machine learning |
CN114218929A (zh) * | 2022-02-22 | 2022-03-22 | 之江实验室 | 一种基于元算子融合的多平台算子智能化开发***及方法 |
CN114579331A (zh) * | 2022-03-01 | 2022-06-03 | 奇安信科技集团股份有限公司 | 一种数据处理方法、装置、电子设备和存储介质 |
-
2022
- 2022-06-29 CN CN202210762431.6A patent/CN117348848A/zh active Pending
-
2023
- 2023-05-18 WO PCT/CN2023/094913 patent/WO2024001594A1/zh unknown
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110928529A (zh) * | 2019-11-06 | 2020-03-27 | 第四范式(北京)技术有限公司 | 辅助算子开发的方法和*** |
US20210182036A1 (en) * | 2019-12-12 | 2021-06-17 | Huawei Technologies Co., Ltd. | Hardware platform specific operator fusion in machine learning |
CN111459474A (zh) * | 2020-03-13 | 2020-07-28 | 北京百分点信息科技有限公司 | 一种模板化的数据处理方法及装置 |
CN114218929A (zh) * | 2022-02-22 | 2022-03-22 | 之江实验室 | 一种基于元算子融合的多平台算子智能化开发***及方法 |
CN114579331A (zh) * | 2022-03-01 | 2022-06-03 | 奇安信科技集团股份有限公司 | 一种数据处理方法、装置、电子设备和存储介质 |
Also Published As
Publication number | Publication date |
---|---|
CN117348848A (zh) | 2024-01-05 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP7371141B2 (ja) | リアルタイムデータフロープログラミング言語のためのツールおよび方法 | |
CN110928529B (zh) | 辅助算子开发的方法和*** | |
US10705942B1 (en) | Simulated testing of API | |
US9846638B2 (en) | Exposing method related data calls during testing in an event driven, multichannel architecture | |
US9697108B2 (en) | System, method, and apparatus for automatic recording and replaying of application executions | |
CN107436844B (zh) | 一种接口用例合集的生成方法及装置 | |
US11126938B2 (en) | Targeted data element detection for crowd sourced projects with machine learning | |
US20130212435A1 (en) | Integrated Fuzzing | |
US9489277B2 (en) | Application component testing | |
US8918762B2 (en) | Generating test plans and test cases from service-oriented architecture and process models | |
CN115658529A (zh) | 用户页面的自动化测试方法以及相关设备 | |
US20180275957A1 (en) | Assistive technology for code generation using voice and virtual reality | |
US9588874B2 (en) | Remote device automation using a device services bridge | |
CN111767209A (zh) | 代码测试方法、装置、存储介质以及终端 | |
WO2021129812A1 (zh) | 运行人工智能应用的方法、***及设备 | |
US20210232494A1 (en) | Network application testing using domain-specific natural language | |
WO2024001594A1 (zh) | 算子开发、处理方法、装置、电子设备、***及存储介质 | |
Jayathilake | A mind map based framework for automated software log file analysis | |
CN111858302A (zh) | 小程序的测试方法、装置、电子设备及存储介质 | |
CN113504912B (zh) | 实时任务的处理方法和装置、存储介质及电子装置 | |
Lefterov et al. | Automated software testing framework “Stаssy” | |
CN115543807A (zh) | 一种自动化回归测试方法、装置、计算机设备及存储介质 | |
CN118260198A (zh) | 业务流程的执行方法、装置、计算机设备、软件机器人 | |
CN118012427A (zh) | 组件编辑方法、装置、电子设备及存储介质 | |
CN117940891A (zh) | 管理应用、尤其是开发包括事件工件的应用的方法和*** |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 23829780 Country of ref document: EP Kind code of ref document: A1 |