CN112199086B - Automatic programming control system, method, device, electronic equipment and storage medium - Google Patents
Automatic programming control system, method, device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN112199086B CN112199086B CN202010988772.6A CN202010988772A CN112199086B CN 112199086 B CN112199086 B CN 112199086B CN 202010988772 A CN202010988772 A CN 202010988772A CN 112199086 B CN112199086 B CN 112199086B
- Authority
- CN
- China
- Prior art keywords
- code
- target
- operator
- information
- environment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 72
- 238000003860 storage Methods 0.000 title claims abstract description 11
- 238000011144 upstream manufacturing Methods 0.000 claims abstract description 46
- 238000006243 chemical reaction Methods 0.000 claims abstract description 39
- 230000001419 dependent effect Effects 0.000 claims description 26
- 238000012545 processing Methods 0.000 claims description 24
- 230000003993 interaction Effects 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 5
- 230000001960 triggered effect Effects 0.000 claims description 4
- 238000013473 artificial intelligence Methods 0.000 description 26
- 238000004422 calculation algorithm Methods 0.000 description 12
- 238000010801 machine learning Methods 0.000 description 12
- 238000007726 management method Methods 0.000 description 12
- 208000025174 PANDAS Diseases 0.000 description 11
- 208000021155 Paediatric autoimmune neuropsychiatric disorders associated with streptococcal infection Diseases 0.000 description 11
- 238000011161 development Methods 0.000 description 10
- 238000005516 engineering process Methods 0.000 description 10
- 240000004718 Panda Species 0.000 description 9
- 235000016496 Panda oleosa Nutrition 0.000 description 9
- 238000010586 diagram Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 9
- 238000007781 pre-processing Methods 0.000 description 8
- 238000004364 calculation method Methods 0.000 description 7
- 238000012986 modification Methods 0.000 description 7
- 230000004048 modification Effects 0.000 description 7
- 238000012549 training Methods 0.000 description 7
- 238000000605 extraction Methods 0.000 description 6
- 230000008676 import Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 238000007405 data analysis Methods 0.000 description 4
- 238000012217 deletion Methods 0.000 description 4
- 230000037430 deletion Effects 0.000 description 4
- 238000003491 array Methods 0.000 description 3
- 230000002452 interceptive effect Effects 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 230000006978 adaptation Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000004140 cleaning Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000011176 pooling Methods 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- 101100328886 Caenorhabditis elegans col-2 gene Proteins 0.000 description 1
- 101100328884 Caenorhabditis elegans sqt-3 gene Proteins 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 238000013499 data model Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000006698 induction Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000002787 reinforcement Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000007711 solidification Methods 0.000 description 1
- 230000008023 solidification Effects 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 238000013526 transfer learning Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/34—Graphical or visual programming
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/35—Creation or generation of source code model driven
- G06F8/355—Round-trip engineering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/40—Transformation of program code
- G06F8/41—Compilation
- G06F8/44—Encoding
- G06F8/447—Target code generation
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Stored Programmes (AREA)
Abstract
The disclosure relates to an automatic programming control system, an automatic programming control method, an automatic programming control device, an electronic device and a storage medium, relates to the technical field of computers, and aims to solve the problem of insufficient flexibility of a modeling method in the related art. The system comprises: the operator management component is used for selecting target operators required for realizing target service according to the operator selection instruction and displaying configuration description information corresponding to each target operator; receiving input parameter configuration information of each target operator, and upstream and downstream connection information among each target operator; the automatic programming component is used for generating code segments corresponding to all target operators according to the parameter configuration information of all target operators and a pre-configured code conversion template; based on the upstream and downstream connection information between the respective target operators, the respective code segments are spliced to generate the target code. The method and the device can flexibly configure the target operator according to the requirements of the target service, and achieve compatibility of efficiency and flexibility by generating the target code capable of being directly operated.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to an automatic programming control system, an automatic programming control method, an automatic programming control device, an electronic device, and a storage medium.
Background
With the wide application of AI (ARTIFICIAL INTELLIGENCE ) in the industrial field, how to apply an algorithm model to the engineering field quickly, and at the same time, provide a fast generation of an AI model algorithm code, and provide efficient system performance, so as to meet the fast iteration and wide application of the service.
The modeling operator component and the execution method in the related technology can partially solve the modeling task in the big data analysis field, but the method mainly emphasizes the execution calculation deployment under specific scenes and clusters, and solidified operators are adopted when the modeling task is carried out. However, due to the very personalized requirements of the real scene in the AI field on data processing, the modeling of the operator based on solidification is difficult to widely apply, and the flexibility is insufficient.
Disclosure of Invention
The present disclosure provides an automatic programming control system, method, apparatus, electronic device, and storage medium to at least solve the problem of insufficient flexibility of modeling methods in related art. The technical scheme of the present disclosure is as follows:
according to a first aspect of embodiments of the present disclosure, there is provided an automatic programming control system, the system comprising:
an operator management component and an automatic programming component;
The operator management component is used for providing a human-computer interaction interface, selecting target operators required for realizing target service according to operator selection instructions triggered by the human-computer interaction interface, and displaying configuration description information corresponding to each target operator; receiving parameter configuration information of each target operator and upstream and downstream connection information among each target operator, wherein the parameter configuration information and the upstream and downstream connection information are determined based on the configuration description information and processing logic of the target service;
The automatic programming component is used for calling an API (Application Programming Interface ), and generating code segments corresponding to all target operators according to parameter configuration information of all target operators acquired by the operator management component based on the man-machine interaction interface and a pre-configured code conversion template; and splicing the code segments based on the upstream and downstream connection information between the target operators to generate target codes for realizing the target service.
Optionally, the operator management component is further configured to:
Before the configuration description information corresponding to each target operator is displayed, a json (JavaScript Object Notation, script language object numbered musical notation) file of each target operator which is configured in advance is analyzed, and the configuration description information corresponding to each target operator is obtained; or (b)
Based on the corresponding relation between the pre-established operators and the configuration description information, the configuration description information corresponding to each target operator is obtained.
Optionally, the automatic programming component is specifically configured to:
And respectively carrying out data conversion processing on the parameter configuration information of each target operator based on the code conversion templates corresponding to each target operator, and generating code segments corresponding to each target operator.
Optionally, the automatic programming component is further configured to:
after the splicing of the code segments to generate an object code for realizing the object service, detecting a code running environment of the object code; the object code is executed based on the detected code execution environment.
Optionally, the code running environment is a distributed cluster; the automatic programming component is specifically configured to:
constructing a distributed environment based on the detected code running environment, and running the object code in the constructed distributed environment.
Optionally, the code running environment is a single machine; the automatic programming component is specifically configured to:
and operating the target code on the stand-alone equipment corresponding to the detected code operating environment.
Optionally, the object code is in a Notebook (Notebook, essentially a Web application) document format or a python (boa, a computer programming language) format.
Optionally, the object code is in a Notebook document format; the automatic programming component is further configured to:
The object code is debugged based on a Notebook editor before the object code is run.
Optionally, the object code further includes environment dependency information for introducing an operator execution environment required to execute each object operator, and/or code base dependency information for introducing a code base required to execute each object operator.
Optionally, the object code includes the environment dependent information; the automatic programming component is further configured to:
detecting operator operation environments required by operating each target operator according to parameter configuration information of each target operator;
Merging and de-duplicating the detected operator operation environments, and generating environment dependency information for introducing operator operation environments required by operating each target operator;
the environment dependency information is added to the top of the object code.
Optionally, the object code includes the code base dependency information; the automatic programming component is further configured to:
Detecting a code library required by running each target operator according to the parameter configuration information of each target operator;
Merging and de-duplicating the detected code libraries, and generating code library dependent information for introducing the code libraries required by operating each target operator;
the code base dependency information is added to the top of the object code.
According to a second aspect of embodiments of the present disclosure, there is provided an automatic programming control method, including:
Selecting target operators required for realizing target business, and displaying configuration description information corresponding to each target operator;
Receiving input parameter configuration information of each target operator and upstream and downstream connection information between each target operator, wherein the parameter configuration information and the upstream and downstream connection information are determined based on the configuration description information and processing logic of the target service;
Generating code segments corresponding to each target operator based on parameter configuration information of each target operator and a pre-configured code conversion template; and
Based on the upstream and downstream connection information between the respective target operators, the respective code segments are spliced to generate target codes for implementing the target service.
In an alternative embodiment, before the displaying of the configuration description information corresponding to each target operator, the method further includes:
analyzing json files of each preset target operator to obtain configuration description information corresponding to each target operator; or (b)
Based on the corresponding relation between the pre-established operators and the configuration description information, the configuration description information corresponding to each target operator is obtained.
In an optional implementation manner, the generating a code segment corresponding to each target operator based on parameter configuration information of each target operator and a pre-configured code conversion template specifically includes:
And respectively carrying out data conversion processing on the parameter configuration information of each target operator based on the code conversion templates corresponding to each target operator, and generating code segments corresponding to each target operator.
In an alternative embodiment, after the splicing the code segments to generate the target code for implementing the target service, the method further includes:
detecting a code running environment of the target code;
the object code is executed based on the detected code execution environment.
In an alternative embodiment, the code execution environment is a distributed cluster; the running the target code based on the detected code running environment specifically comprises:
constructing a distributed environment based on the detected code running environment, and running the object code in the constructed distributed environment.
In an alternative embodiment, the code execution environment is a stand-alone machine; the running the target code based on the detected code running environment specifically comprises:
and operating the target code on the stand-alone equipment corresponding to the detected code operating environment.
In an alternative embodiment, the object code is in a Notebook document format or a python format.
In an alternative embodiment, the object code is in a Notebook document format; the method further comprises the steps of:
The object code is debugged based on a Notebook editor before the object code is run.
In an alternative embodiment, the object code further comprises environment dependency information for introducing an operator runtime environment required to run the respective object operator, and/or code base dependency information for introducing a code base required to run the respective object operator.
In an alternative embodiment, the object code includes the environment dependent information; the splicing the code segments to generate the target code for realizing the target service further comprises:
detecting operator operation environments required by operating each target operator according to parameter configuration information of each target operator;
Merging and de-duplicating the detected operator operation environments, and generating environment dependency information for introducing operator operation environments required by operating each target operator;
the environment dependency information is added to the top of the object code.
In an alternative embodiment, the object code includes the code base dependency information; the splicing the code segments to generate the target code for realizing the target service further comprises:
Detecting a code library required by running each target operator according to the parameter configuration information of each target operator;
Merging and de-duplicating the detected code libraries, and generating code library dependent information for introducing the code libraries required by operating each target operator;
the code base dependency information is added to the top of the object code.
According to a third aspect of embodiments of the present disclosure, there is provided an automatic programming control apparatus, including:
the operator selection unit is configured to execute selection of target operators required for realizing target business and display configuration description information corresponding to each target operator;
a parameter configuration unit configured to perform receiving input parameter configuration information of each target operator and upstream and downstream connection information between each target operator, wherein the parameter configuration information and the upstream and downstream connection information are determined based on the configuration description information and processing logic of the target service;
A code conversion unit configured to execute a code segment corresponding to each target operator based on parameter configuration information of each target operator and a pre-configured code conversion template; and
Based on the upstream and downstream connection information between the respective target operators, the respective code segments are spliced to generate target codes for implementing the target service.
Optionally, the operator selecting unit is further configured to perform:
before the configuration description information corresponding to each target operator is displayed, analyzing a json file of each target operator which is configured in advance to obtain the configuration description information corresponding to each target operator; or (b)
Based on the corresponding relation between the pre-established operators and the configuration description information, the configuration description information corresponding to each target operator is obtained.
Optionally, the transcoding unit is specifically configured to:
And respectively carrying out data conversion processing on the parameter configuration information of each target operator based on the code conversion templates corresponding to each target operator, and generating code segments corresponding to each target operator.
Optionally, the apparatus further includes:
A code execution unit configured to execute a code execution environment that detects the object code;
the object code is executed based on the detected code execution environment.
Optionally, the code running environment is a distributed cluster; the code execution unit is configured to execute:
constructing a distributed environment based on the detected code running environment, and running the object code in the constructed distributed environment.
Optionally, the code running environment is a single machine; the code execution unit is configured to execute:
and operating the target code on the stand-alone equipment corresponding to the detected code operating environment.
Optionally, the object code is in a Notebook document format or a python format.
Optionally, the object code is in a Notebook document format; the apparatus further comprises:
And the debugging unit is configured to execute debugging of the target code based on a Notebook editor before the target code is executed.
Optionally, the object code further includes environment dependency information for introducing an operator execution environment required to execute each object operator, and/or code base dependency information for introducing a code base required to execute each object operator.
Optionally, the object code includes the environment dependent information; the transcoding unit is further configured to perform:
detecting operator operation environments required by operating each target operator according to parameter configuration information of each target operator;
Merging and de-duplicating the detected operator operation environments, and generating environment dependency information for introducing operator operation environments required by operating each target operator;
the environment dependency information is added to the top of the object code.
Optionally, the object code includes the code base dependency information; the transcoding unit is further configured to perform:
Detecting a code library required by running each target operator according to the parameter configuration information of each target operator;
Merging and de-duplicating the detected code libraries, and generating code library dependent information for introducing the code libraries required by operating each target operator;
the code base dependency information is added to the top of the object code.
According to a fourth aspect of embodiments of the present disclosure, there is provided an electronic device, comprising:
A processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the automatic programming control method of any of the second aspects of the embodiments of the present disclosure.
According to a fifth aspect of embodiments of the present disclosure, there is provided a non-transitory readable storage medium, which when executed by a processor of an electronic device, causes the electronic device to perform the automatic programming control method of any one of the second aspects of embodiments of the present disclosure.
According to a sixth aspect of embodiments of the present disclosure, there is provided a computer program product which, when run on an electronic device, causes the electronic device to perform a method of implementing the above-described second aspect of embodiments of the present disclosure and any one of the possible concerns of the second aspect.
The technical scheme provided by the embodiment of the disclosure at least brings the following beneficial effects:
Because the embodiment of the disclosure provides the AI automatic programming control method and system based on the visualization technology, after selecting the target operator by means of dragging and the like, the user can set parameter configuration information of the target operator according to the requirements of the target service, compared with the solidified operator adopted in the related technology, the target operator in the embodiment of the disclosure is transparent, and the user can realize flexible parameter configuration based on the configuration description information of the target operator; and moreover, by configuring upstream and downstream connection information among all target operators, pipeline operation can be realized, and further, high-quality target codes capable of directly running can be generated by calling an API (application program interface), so that the efficiency of the coding development process of an AI task is greatly improved, and the compatibility of the efficiency and the flexibility is realized.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure and do not constitute an undue limitation on the disclosure.
FIG. 1 is a block diagram of the constituent architecture of an automatic programming control system, according to an example embodiment;
FIG. 2 is a schematic diagram of a target operator upstream-downstream join relationship, according to an example embodiment;
FIG. 3 is a schematic diagram illustrating the architecture of an automatic programming control system in accordance with an exemplary embodiment;
FIG. 4 is a flowchart illustrating an automatic programming control method, according to an example embodiment;
FIG. 5 is a flowchart illustrating a complete method of automatic programming control, according to an exemplary embodiment;
FIG. 6 is a block diagram of an automatic programming control device, according to an exemplary embodiment;
Fig. 7 is a block diagram of an electronic device, according to an example embodiment.
Detailed Description
In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.
It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the foregoing figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the disclosure described herein may be capable of operation in sequences other than those illustrated or described herein. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as detailed in the accompanying claims.
Some words appearing hereinafter are explained:
1. The term "and/or" in the embodiments of the present disclosure describes an association relationship of association objects, which indicates that three relationships may exist, for example, a and/or B may indicate: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship.
2. The term "electronic device" in embodiments of the present disclosure may be a mobile phone, computer, digital broadcast terminal, messaging device, game console, tablet device, medical device, exercise device, personal digital assistant, or the like.
3. The term "OP (Operator) Operator" in the embodiments of the present disclosure is a calculation unit of an algorithm, belongs to a target Operator in the embodiments of the present disclosure, and the OP Operator may refer to a basic unit of neural network calculation, including conv (convolution), pooling (pooling), activation, normalization (standardization), and the like.
4. The term "configuration description information" in the embodiments of the present disclosure is description information related to each configuration of an operator, and is related rules or definitions for describing each configuration of an operator, for example, what type a certain operator is, what a name is, what an alias is, which parameters are included, and which properties are supported by these parameters.
5. The term "upstream and downstream join information" in embodiments of the present disclosure is used for operators and join relationships between operators. In the embodiment of the disclosure, a user can configure upstream and downstream connection information by setting a type attribute in an input and output interface of an operator through upstream and downstream connection lines.
6. The term "parameter configuration information" in the embodiments of the present disclosure is specific configuration information of parameters related to an operator, for example, a name of a table read corresponding to a read operator, a number of columns read, and the like. For example, the algorithm operator corresponds to a task type of an algorithm, various parameters related to the algorithm, and the like.
7. The term "Notebook editor" in the embodiments of the present disclosure is a very popular code editor suitable for AI development and data analysis and code presentation. Specifically, jupyterNotebook (Jupyter notebook), which is an interactive notebook, supports running more than 40 programming languages. JupyterNotebook is essentially a Web application that facilitates creation and sharing of a literacy document, supports real-time code, mathematical equations, visualization, and markdown (a lightweight markup language). The application comprises the following steps: data cleaning and conversion, numerical simulation, statistical modeling, machine learning, and the like. In the embodiment of the disclosure, a user can perform personalized modification on the generated target code through a Notebook editor, so that secondary development of the target code is realized.
8. The term "operator runtime environment" in embodiments of the present disclosure refers to an environment in which semi-compiled runtime code is run on a target machine. The operator operation environment is an operation mode which is in charge of a compiler and an interpreter. Such as the "TensorFlow operator runtime environment" listed in the embodiments of the present disclosure, wherein TensorFlow is an end-to-end open source machine learning platform. It has a comprehensive and flexible ecosystem that contains various tools, libraries and community resources that can assist researchers in the development of advanced machine learning techniques and enable developers to easily build and deploy applications supported by machine learning.
9. The term "code running environment" in the embodiments of the present disclosure refers to an environment in which a device is run when an object code is run, and in the embodiments of the present disclosure, the code running environment is mainly divided into two code running environments, i.e., a single machine and a distributed cluster.
10. The term Numpy in the presently disclosed embodiments is an open-source numerical calculation extension of Python. Such tools can be used to store and process large matrices, much more efficient than the Python's own nested list (NESTED LIST structure) structure (which can also be used to represent a matrix). Numpy provides a base library for scientific calculations in Python, and is used for numerical calculations, mainly for libraries for processing multidimensional arrays (matrices). For storing and processing large matrices, is much more efficient than the nested list structure of Python itself. The method is developed by the C language, is a very basic extension, and most of the rest scientific calculation extensions of Python are based on the development.
11. The term "Pandas" in the presently disclosed embodiments is based on NumPy, a tool that is created to address the task of data analysis. Pandas incorporate a large library and some standard data models, providing the tools needed to efficiently operate large datasets. pandas provide a number of functions and methods that enable us to quickly and easily process data. In Python pandas is built based on NumPy arrays, making the data preprocessing, cleaning, analysis work faster and simpler. pandas are specifically designed for handling tables and miscellaneous data, while NumPy is more suited for handling unified numerical array data. The following format convention may be used, introducing pandas packets: import PANDAS AS PD. pandas, which can import, clean, process, count and export data. pandas is based on Numpy library, so to speak, pandas library is generated for data analysis.
12. The term "Hive" in this embodiment of the disclosure is a data warehouse tool based on Hadoop (sea Du Pu, a distributed system infrastructure), which can map a structured data file into a database table, provide a complete sql (Structured Query Language ) query function, and convert sql statements into MapReduce (a programming model) tasks for running. The method has the advantages of low learning cost, capability of rapidly realizing simple MapReduce statistics through SQL-like sentences, no need of developing special MapReduce application, and suitability for statistical analysis of a data warehouse. The other is a Windows registry file.
13. The term "json" in embodiments at the time of this disclosure is a lightweight data exchange format that stores and represents data in a text format that is completely independent of the programming language. The compact and clear hierarchical structure makes json an ideal data exchange language. Is easy to read and write by people, is easy to analyze and generate by machines, and effectively improves the network transmission efficiency.
Embodiments of the present disclosure relate to artificial intelligence and machine learning (MACHINELEARNING, ML) techniques, designed based on computer vision techniques and machine learning in artificial intelligence.
Artificial intelligence is the theory, method, technique and application system that uses a digital computer or a digital computer-controlled machine to simulate, extend and expand human intelligence, sense the environment, acquire knowledge and use the knowledge to obtain optimal results. In other words, artificial intelligence is an integrated technology of computer science that attempts to understand the essence of intelligence and to produce a new intelligent machine that can react in a similar way to human intelligence.
Artificial intelligence, i.e. research on design principles and implementation methods of various intelligent machines, enables the machines to have functions of sensing, reasoning and decision. Artificial intelligence techniques mainly include computer vision techniques, natural language processing techniques, machine learning/deep learning, and other major directions. With research and progress of artificial intelligence technology, artificial intelligence is developed in various fields such as common smart home, intelligent customer service, virtual assistant, smart speaker, smart marketing, unmanned, automatic driving, robot, smart medical, etc., and it is believed that with the development of technology, artificial intelligence will be applied in more fields and become more and more important value.
Machine learning is a multi-domain interdisciplinary, involving multiple disciplines such as probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and the like. It is specially studied how a computer simulates or implements learning behavior of a human to acquire new knowledge or skills, and reorganizes existing knowledge structures to continuously improve own performance.
Machine learning is the core of artificial intelligence, a fundamental approach to letting computers have intelligence, which is applied throughout various areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, induction learning, and the like.
Based on the automatic programming control method enumerated in the embodiments of the present disclosure, high quality AI model training, predictive codes may be automatically generated. A model, such as a machine learning model, can be produced after the code is run, after which the model of production can be stored in model db (model database).
The application scenario described in the embodiments of the present disclosure is for more clearly describing the technical solution of the embodiments of the present disclosure, and does not constitute a limitation on the technical solution provided by the embodiments of the present disclosure, and as a person of ordinary skill in the art can know that, with the appearance of a new application scenario, the technical solution provided by the embodiments of the present disclosure is equally applicable to similar technical problems. Wherein in the description of the present disclosure, unless otherwise indicated, the meaning of "plurality" is used.
FIG. 1 is a schematic diagram of the composition and structure of an automatic programming control system 100, shown in FIG. 1, including an operator management component 101 and an automatic programming component 102, according to an example embodiment;
The operator management component 101 is configured to provide a human-computer interaction interface, select a target operator required for implementing a target service according to an operator selection instruction triggered by the human-computer interaction interface, and display configuration description information corresponding to each target operator; receiving parameter configuration information of each target operator and upstream and downstream connection information among the target operators, wherein the parameter configuration information and the upstream and downstream connection information are determined based on configuration description information and processing logic of a target service;
The automatic programming component 102 is used for calling an API, and generating code segments corresponding to each target operator according to parameter configuration information of each target operator obtained by the operator management component based on the human-computer interaction interface and a pre-configured code conversion template; and splicing the code segments based on the upstream and downstream connection information between the target operators to generate target codes for realizing the target service.
In the embodiment of the present disclosure, a user may select, by dragging or the like, a target operator required for executing the target business process from a pre-constructed OP operator library, which is exemplified by the OP operator.
It should be noted that, the OP operator library in the embodiments of the present disclosure is pre-built, where the OP operator is not a cured operator, but is transparent. When a user selects an operator from the OP operator library to drag through the UI (User Interface) interface, an operator selection instruction can be triggered, and then the configuration description information of the OP operator can be displayed in the UI interface. The UI interface hereinafter is a man-machine interaction interface in the embodiments of the present disclosure.
Then, the user can refer to the configuration description information displayed in the UI interface, configure the parameters of each OP operator in combination with the requirements of the target service, and input parameter configuration information; in addition, the configuration of the upstream and downstream connection relation between the respective OP operators is also required, and the input of the upstream and downstream connection information is also required.
The configuration description information of the OP operator displayed in the UI interface may be obtained by the operator management component by:
The first acquisition mode is used for analyzing the json file of each preset target operator and acquiring configuration description information corresponding to each target operator.
In this way, each target operator in the embodiment of the disclosure is defined by a corresponding json file, and json description conditions, that is, configuration description information of the target operator, can be obtained by parsing json.
And acquiring the configuration description information corresponding to each target operator based on the corresponding relation between the pre-established operator and the configuration description information in the second acquisition mode.
In the embodiment of the disclosure, the json file of the operator can be analyzed in advance to obtain the configuration description information of each operator, the configuration description information of each operator can be stored in the form of a document and the like, and then the corresponding relation between each target operator and the configuration description information corresponding to the operator is pre-established, so that after the user drags the target operator, the corresponding configuration description information can be determined directly according to the pre-established corresponding relation, the json file of the target operator is not required to be analyzed, and the working efficiency can be effectively improved.
It should be noted that, the OP operator in the embodiments of the present disclosure may be defined by json files, and of course, may be defined by other files, which is not limited herein specifically. Taking json file definition as an example, the two ways of acquiring the configuration description information of the target operator listed in the above embodiment are both exemplified, and other acquiring ways besides the way of acquiring the configuration description information of the target operator listed in the above embodiment are also applicable to the embodiments of the present disclosure, and are not limited in detail herein.
In the above embodiment, the configuration description information of the target operator is displayed to the user through the UI interface, so as to prompt the user to configure corresponding parameters, upstream-downstream connection relationships, and the like according to the configuration description information and the service requirement of the target service, so as to implement flexible configuration and pipeline of the target operator.
Specifically, the configuration description information of the OP operator in the embodiment of the present disclosure specifically includes: operator attribute information, operator parameter list, operator input/output interface and other parts.
Wherein the attribute information is used to describe the attributes of the OP operator, including but not limited to part or all of the following:
op_id (operator sequence number), op_name, op_alias (operator alias), op_type (operator type).
Wherein, when the operator is defined by type, the operator comprises the following types:
read_data (read data source), write_data (write data source), preprocessing (data preprocessing), feature_extraction (feature extraction), trace (training), predict (prediction), custom (custom).
It should be noted that, part or all of the attribute information of the operator may be preconfigured when the OP operator library is constructed, and the configuration description information of the OP operator may be directly displayed to the user when the configuration description information of the OP operator is displayed in the UI interface, without requiring the user to reconfigure. Whereas the parameter list and the input-output interface in the embodiments of the present disclosure need to be configured with specific values and the like by the user according to the description information.
The following first describes a parameter list, and a user may refer to specific parameter configuration information of the parameter list configuration parameters.
In the disclosed embodiment, a parameter list is used to introduce the individual parameter items to which the OP operator relates. Specifically, parameter list uses parameters key, each supporting some or all of the following:
Name (parameter Name), description (parameter Description), label (parameter prompt), type (parameter type), required (must fill item (Y), optional (N)), default (default value, default is generally null), depend (dependent items such as depend = "jobtype = 'XGBRegressor'"), options (alternative list).
Wherein the parameter types include, but are not limited to, some or all of the following:
string (string), such as table name (da. Table_name_demo), hdfs address (/ home/reco/test_path);
bool (boolean value), values are: yes (True), no (False);
list, such as a field list ([ col1, col2, col3 ]).
The user may configure the parameters of the OP operator by referring to the configuration description information, examples are as follows:
-parameters:[
{ "type": "string", "description": "hive table name", "name": "table_name", label= "hive table name", request= "Y", default= ", },
{ "Type": "list", "description": "task type", "name": "jobtype", label = "task type", options = [ 'XGBClassifier': 'classification', "XGBRegressor": 'regression' ] },
......]
The above examples show specific configurations of a parameter, i.e. parameter configuration information, as known from the above examples:
The parameter types of the first parameter are: the character string type, the parameters are described as: hive table names, parameter names: table_name (table name), parameter prompt information is: hive table name, the default value of this parameter is null. Furthermore, the requirement that the parameter is a mandatory term, that is, when the user refers to the configuration description information in the UI interface to configure the input parameter, one OP operator may involve multiple parameters, where the parameter is a mandatory term.
The following parameter types of the other parameters are: list type, parameters are described as: task type, parameter name: jobtype (task type), parameter prompt information is: the task type and the parameter alternative list are as follows: 'XGBClassifier' category ',' XGBRegressor 'regression'.
The parameter alternative list is an alternative task type provided for the user and is preconfigured, for example, the example indicates that the alternative task type can be XGB classification or XGB regression, and the user can select one of the two task types for configuration in a selective mode.
Note that XGBoost is one of boosting algorithms. The idea of Boosting algorithm is to integrate many weak classifiers together to form one strong classifier. Since XGBoost is a lifting tree model, it integrates many tree models together to form a strong classifier. In addition to the above listed parameters for configuring task types, the user may configure XGBoost that the tree integration model includes several trees, and the depth of each tree is not listed here.
In the embodiment of the disclosure, parameters of the operator need to be configured by a user on the UI side, and parameters are present in the json file of the operator, and after the parameters are resolved, the user can configure the parameters in the manner listed above.
In the embodiments of the present disclosure, a user may refer to description information of the input/output interfaces, and define the input/output interfaces of the OP operators to establish an upstream-downstream connection relationship between the OP operators.
Wherein, the input interfaces and the output interfaces are list arrays, supporting the following properties:
alias (interface name, which may be input or output in particular), types (OP operator type that defines the OP that can support the upstream and downstream links), name (interface name for obtaining a value by changing the name), value (value expression).
The following lists one configuration example regarding the input interface and the output interface of the OP operator:
example 1: input interfaces:
[ { "alias": "input", "types": [ "read_data", "processing", "feature_extraction" ], "name": "track" }, ]
In the above example, an input interface is configured, where the alias of the input interface is "input", and the following types of OP operators may be connected to the input interface, where the OP operators are respectively: read_data (read data source), preprocessing (feature_extraction), the output representing these types of OP operators can be used as the input of the OP operator, and these types of OP operators can be connected with the OP operator. The name of the input interface is: train (training).
Example 2: output interfaces:
[ { "alias": "output ","types":['preprocessing','feature_extraction','train','predict','custom'],"name":"df","value":"df_{{node_id}}"}]
In the above example, an output interface is configured, where the alias of the output interface is "output", and the types of OP operators that can support the connection are: preprocessing (data preprocessing), feature_extraction, train, predict (prediction), custom, represent that the output of the OP operator can be used as input to the several types of OP operators to which the OP operator connection can be connected. The name of the output interface is: df, the value expression is: df_ { { node_id }.
As can be seen from the foregoing embodiments, in the embodiments of the present disclosure, a user may establish an upstream-downstream connection relationship between OP operators by configuring an attribute of type in an input interface or an output interface of the OP operators, and may determine upstream-downstream connection information between OP operators input by the user based on the configuration of the attribute of type.
Referring to fig. 2, an upstream-downstream connection relationship between OP operators is shown in the embodiment of the present disclosure, where a first OP operator is a read data source operator for reading hive table data, for the OP operator, only a type attribute of an output interface of the OP operator needs to be configured, specifically, an output connection deletion value of the OP operator for reading hive table data fills an input of the OP operator, an output connection deletion value of the OP operator fills an input of a standardized OP operator, an output connection XGBoost of the standardized OP operator trains an input of the OP operator, and an output connection XGBoost trains an input of the OP operator for writing hdfs (Hadoop Distributed FILE SYSTEM ) data.
In the embodiment of the disclosure, after parameter configuration information of each OP operator and upstream and downstream connection information between OP operators are acquired based on the operator management component, an API interface can be called by the automatic programming component to generate object code.
It should be noted that in the embodiment of the present disclosure, transcoding templates corresponding to different OP operators are also preconfigured. Specifically, when generating the target code, the code conversion is required to be performed according to the code conversion template corresponding to the OP operators and by combining parameter configuration information of the OP operators, so as to obtain a code segment corresponding to the OP operators, and splicing is performed according to upstream and downstream connection information between the OP operators so as to generate the target code corresponding to the target service.
Optionally, the automatic programming component is specifically configured to perform data conversion processing on parameter configuration information of the target operator based on a code conversion template corresponding to the target operator, so as to generate a code segment corresponding to the target operator. After obtaining a code segment corresponding to a target operator, the upstream and downstream connection information between the operators can be based. And outputting the code segment to a downstream node, continuing to perform data conversion processing on the parameter configuration information of the next target operator, obtaining a corresponding code segment, and outputting the code segment to the next downstream node. And finally, splicing the code segments corresponding to the target operators based on the upstream and downstream connection relations among the target operators to obtain target codes.
The following lists a transcoding template (templates) corresponding to the OP operator for reading hive table data:
Wherein isinstance () function determines whether an object is of a known type, like type (). The syntax of the isinstance () method is: isinstance (object, classinfo).
Where parameter one object represents an instance object, parameter two classinfo may be a direct or indirect class name, a base type, or a tuple made up of them, a return value. If the type of the object is the same as the type of parameter two (classinfo), then True is returned, otherwise False is returned.
The function realized by the template is to read the Hive table, and whether the Hive link is an effective Hive link needs to be judged first; if not, the database is concatenated to generate a hive active link. Then, the user selects which columns (partition_columns) are read in the data table on the UI side, the partition of the column selected by the user is selected based on the column name selected by the user, and the table name (table_name) of which hive table is required to be read is loaded; then, the data can be read according to the partition_cols, and limit_num indicates that all output quantities are required to be obtained during reading; cached indicates whether or not the read data is cached; op_id represents the ID (operator number) of the operator, and the operator is uniquely identified through the operator number, so that data between the same type of OP operators is prevented from being covered. For example, an operator may be used twice in pipeline, once in training and once in using, in order to prevent the variable generated by the operator from collision, the operator is covered, and the uniqueness of the variable can be ensured through op_id.
The variables in the above-listed transcoding templates are variables that need to be filled in when transcoding, and when a code is generated by one key, a user can fill in the transcoding templates based on the parameter configuration information configured on the UI side, and generate corresponding code segments.
The post-conversion code is as follows:
the column name of the user selection is 'p_date' = 20200710, the table name table_name is db. The column names, table names, and the like described above are parameter configuration information set by the user.
In the embodiment, the code segments are automatically generated according to the flexible configuration of the user, and the code input and output between operators can be automatically spliced to form the target code, and the target code is directly converted based on the code conversion template, so that the efficiency is extremely high.
Considering that in the related art, a user does not know the process of the task execution bottom layer, the research and exploration stage and the execution stage of the AI algorithm model are split, and consistency is difficult to ensure. In the embodiment of the disclosure, the generated target code is directly provided for the user, so that the user can clearly know the process of the task execution bottom layer. Furthermore, the automatic programming component is also for: after splicing the code segments to generate an object code for realizing the object service, detecting a code running environment of the object code; the object code is executed based on the detected code execution environment. Wherein the code execution environment comprises: distributed clusters or stand-alone.
Specifically, the automatic programming component is specifically configured to: and constructing a distributed environment based on the detected code running environment, and running target codes in the constructed distributed environment. And when the code running environment is a single machine, running the target code on the single machine equipment corresponding to the detected code running environment.
That is, the AI task code generated by the interactive modeling in the embodiment of the disclosure may be run in a single machine and a distributed scenario at the same time, for example, a small data set may be first explored and applied on the single machine, and after the verification is feasible, the AI task code is put under a large distributed task, so that consistency of an exploration stage and distributed industrial deployment is maintained, no modification of the code is required, a distributed task code development threshold is shielded, a consistent algorithm debugging and engineering application method is provided, and engineering time from AI model development to online is greatly shortened.
Optionally, the object code is in a Notebook document format or a python format. Wherein, the python format can be expressed as a run.py script, which is a python script that can be directly run; the Notebook document format may be represented as a run.ipynb script, which is an interactive exploration format, and the user may execute these code blocks in a single line in the Notebook to explore. Thus, when the object code is in the Notebook document format, the automatic programming component is also operative to debug the object code based on the Notebook editor prior to running the object code.
In the above implementation manner, the output of the embodiment of the disclosure supports jupyter notebook document format output in addition to python code, which is convenient for data scientists to modify and debug, and to perform personalized modification of the object code, thereby improving flexibility.
In addition, it should be noted that operators depend on operator operation environments, but the system generally provides a general operator operation environment and supports the operator library of a specific version uniformly. If some specific operator running environments or code libraries need to be introduced, the object code in the embodiments of the disclosure may further include operator dependent information, specifically including environment dependent information for introducing operator running environments required to run each object operator, and/or code library dependent information for introducing code libraries required to run each object operator.
The following describes the manner of acquiring the environment-dependent information and the code base-dependent information, respectively:
When the object code includes environment dependent information, the automatic programming component is further operable to: detecting operator operation environments required by operating each target operator according to parameter configuration information of each target operator; merging and de-duplicating the detected operator operation environments, and generating environment dependency information for introducing operator operation environments required by operating each target operator; the environment dependency information is added to the top of the object code.
Similarly, when the object code includes code base dependent information, the automatic programming component is further configured to detect a code base required for running each object operator according to parameter configuration information of each object operator; merging and de-duplicating the detected code libraries, and generating code library dependent information for introducing the code libraries required by operating each target operator; code base dependency information is added to the top of the object code.
In the embodiment of the disclosure, whether the task is a stand-alone task or a distributed task, whether the operator operation environment exists or not needs to be detected before operation, and when the operator operation environment does not exist, the corresponding operator operation environment needs to be introduced based on the operator operation environment dependency information at the top of the target code. When numpy is available in the object operator, the code package is imported first and added to the top of the object code, and when the object code is operated, the corresponding code package is imported according to the code base dependency information sequence of the top.
The following lists one example of operator runtime environment dependency information and code library dependency information as follows:
Operator runtime environment dependency information-requires [ "numpy = 1.18.1", "tenesorflow = 1.14.1" ];
code base dependency information-imports [ "import numpy as np", "import PANDAS AS PD" ].
In the disclosed embodiment, requires is used to detect and record the operator runtime environment, after which the operator runtime environment may be installed via pip or conda, etc., 1.18.1 refers to the version number of numpy and 1.14.1 refers to the version number of tenesorflow. Wherein Pip represents PIP INSTALLS PACKAGES, which is a common manager of Python packets; conda is a language independent cross-platform environment manager.
In addition, the code library (also referred to as a code package) needs to be imported in advance before imports is executed, and in the embodiment of the disclosure, if the detected code library has repetition, all imports may be merged and de-duplicated at the time of code generation and then imported into the top of the target code. The same is true for operator runtime environment dependency information.
In the embodiment of the disclosure, when operators are defined through json files, the environmental problem on which the operators depend can be settled, in addition, the operator codes can be automatically spliced by calling the attribute and the interface value, the code input and output between operators can greatly improve the coding development process of the AI task, the code output of classical AI application can be completed within a few minutes, and algorithm personnel can secondarily develop the codes, so that the compatibility of efficiency and flexibility is achieved.
Referring to fig. 3, an architecture diagram of an automatic programming control system according to an embodiment of the disclosure specifically includes a UI and an API, where the UI corresponds to the operator management component 101 shown in fig. 1, the component may provide a man-machine interaction interface, i.e., a UI interface, and the API corresponds to the automatic programming component 102 shown in fig. 1.
Specifically, a user drags OP operators required for realizing the target service from an OP operator library through a UI interface, wherein the OP operators are defined based on json files, configuration description information of the OP operators after the json files are analyzed and displayed, the operators are added and deleted through interaction with the user, parameters are configured, upstream and downstream connection relation is configured, and the like, and pipeline is completed through interaction. And then, the parameter configuration information, the upstream and downstream connection information and the like can be requested to an automatic programming component side through an API, and the operator is updated to a database, so that the addition, deletion and modification of the operator are realized. Namely, the management of the operator library is realized through adding, deleting and checking through the UI call API. Further, run.py or run.ipynb is generated by calling the API.
For example, when a data scientist uses an operator, dragging an operator for reading a data source, an operator for preprocessing data, an operator for filling a value (for example, filling a value is null), an operator for training, an operator for writing, and the like on a UI interface, and then connecting the operators by configuring upstream and downstream connection information between the operators to generate and store a pipeline. And then, calling an API to perform operator management, specifically comprising deletion and examination and the like, managing pipeline, manually or automatically starting target codes run.py or run.ipynb, scheduling execution jobs, generating single-machine jobs (jobs) or distributed jobs, and executing the target codes. Further, when executing the object code, the user may select to execute immediately, or to execute at a timing, etc. through the UI interface.
For example, after the user clicks the run button in the UI, run. Py is generated through the API, and then the code base creates a training job to run. Py, and after the run is completed, the model is automatically placed in the model database. When the operator does not necessarily meet the model index or the user has some unique understanding on the data, the button can be operated at first, a run.ipynb file is generated through one-key conversion, and an object code file is opened in a Notebook service by default to debug codes, such as code modification, exploration and the like. After the exploration is finished, the points are saved, and personalized target codes are generated; and finally, running the debugged new target code through newly built training operation.
Wherein, the Util toolkit can be also called when the object code is run. The Util toolkit in the embodiment at the time of this disclosure refers to a tool kit containing a collection framework, an event model, date and time facilities, internationalization and various utility classes (string tag generator, random number generator and bit array, date class, stack class, vector class, etc.). Collection class, time processing mode, date and time tool and other common tool kits.
Through the scheme, compared with the cured operator in the related technology, the OP operator in the embodiment of the disclosure is not a black box for a user, is visible to the user, can be converted into codes, can be seen by the user to be processed by the specific operator, and can be subjected to personalized modification.
Based on the same inventive concept, the embodiment of the present disclosure also provides an automatic programming control method. As shown in fig. 4, which is a flowchart of an automatic programming control method according to an exemplary embodiment of the present disclosure, includes the following steps.
In step S41, selecting a target operator required for realizing a target service, and displaying configuration description information corresponding to each target operator;
In step S42, receiving input parameter configuration information of each target operator and upstream and downstream connection information between each target operator, wherein the parameter configuration information and the upstream and downstream connection information are determined based on the configuration description information and processing logic of the target service;
In step S43, code segments corresponding to the respective target operators are generated based on the parameter configuration information of the respective target operators and the pre-configured transcoding template; and splicing the code segments based on the upstream and downstream connection information between the target operators to generate target codes for realizing the target service.
Through the scheme, after the target operator is selected in a dragging mode and the like, the user can set parameter configuration information of the target operator according to the requirements of the target service, and compared with a solidified operator adopted in the related technology, the target operator in the embodiment of the disclosure is transparent, and the user can realize flexible parameter configuration based on configuration description information of the target operator; and moreover, by configuring the upstream and downstream connection information among all target operators, pipeline can be realized, and further, high-quality target codes capable of directly running can be generated by calling an API (application program interface), so that the efficiency of the coding development process of an AI (advanced technology) task is greatly improved, and the compatibility of the efficiency and the flexibility is realized.
In an alternative embodiment, before displaying the configuration description information corresponding to each target operator, the method further includes:
analyzing json files of each preset target operator to obtain configuration description information corresponding to each target operator; or (b)
Based on the corresponding relation between the pre-established operators and the configuration description information, the configuration description information corresponding to each target operator is obtained.
In an alternative embodiment, based on parameter configuration information of each target operator and a pre-configured transcoding template, a code segment corresponding to each target operator is generated, which specifically includes:
And respectively carrying out data conversion processing on the parameter configuration information of each target operator based on the code conversion templates corresponding to each target operator, and generating code segments corresponding to each target operator.
In an alternative embodiment, after splicing the code segments to generate the target code for implementing the target service, the method further includes:
detecting a code running environment of the target code;
the object code is executed based on the detected code execution environment.
In an alternative embodiment, the code execution environment is a distributed cluster; running object code based on the detected code running environment, specifically comprising:
A distributed environment is built based on the detected code running environment, and object code is run in the built distributed environment.
In an alternative embodiment, the code execution environment is a stand-alone machine; running object code based on the detected code running environment, specifically comprising:
And running the target code on the stand-alone equipment corresponding to the detected code running environment.
In an alternative embodiment, the object code is in a Notebook document format or a python format.
In an alternative embodiment, the object code is in a Notebook document format; the method further comprises the steps of:
The object code is debugged based on a Notebook editor before it is run.
In an alternative embodiment, the object code further comprises environment dependency information for introducing an operator runtime environment required to run the respective object operator, and/or code base dependency information for introducing a code base required to run the respective object operator.
In an alternative embodiment, the object code includes environment dependent information; splicing the code segments to generate an object code for implementing the object service, further comprising:
detecting operator operation environments required by operating each target operator according to parameter configuration information of each target operator;
Merging and de-duplicating the detected operator operation environments, and generating environment dependency information for introducing operator operation environments required by operating each target operator;
the environment dependency information is added to the top of the object code.
In an alternative embodiment, the object code includes code base dependency information; splicing the code segments to generate an object code for implementing the object service, further comprising:
Detecting a code library required by running each target operator according to the parameter configuration information of each target operator;
Merging and de-duplicating the detected code libraries, and generating code library dependent information for introducing the code libraries required by operating each target operator;
code base dependency information is added to the top of the object code.
FIG. 5 is a flowchart illustrating a complete method of automatic programming control, according to an exemplary embodiment, including the steps of:
S51: selecting a target operator required for realizing the target service, analyzing a json file of each pre-configured target operator, acquiring configuration description information corresponding to each target operator, and displaying the configuration description information corresponding to each target operator;
S52: receiving input parameter configuration information of each target operator and upstream and downstream connection information among each target operator;
S53: detecting operator operation environments and code libraries required by operating each target operator according to parameter configuration information of each target operator;
s54: merging and deduplicating the detected operator operation environments to generate environment dependency information for introducing operator operation environments required by operating each target operator, merging and deduplicating the detected code libraries to generate code library dependency information for introducing code libraries required by operating each target operator;
S55: generating code segments corresponding to all target operators based on parameter configuration information of all target operators and a pre-configured code conversion template, and splicing all the code segments based on upstream and downstream connection information among all the target operators to generate target codes for realizing target services;
S56: adding the environment dependency information and the code base dependency information to the top of the object code;
S57: a code execution environment of the object code is detected, and the object code is executed based on the detected code execution environment.
Based on the same inventive concept, the embodiment of the disclosure also provides an automatic programming control device. Fig. 6 is a block diagram illustrating an automatic programming control device 600, according to an example embodiment. Referring to fig. 6, the apparatus includes:
an operator selecting unit 601 configured to perform selecting a target operator required for implementing a target service, and display configuration description information corresponding to each target operator;
A parameter configuration unit 602 configured to perform receiving input parameter configuration information of each target operator and upstream and downstream connection information between each target operator, wherein the parameter configuration information and the upstream and downstream connection information are determined based on the configuration description information and processing logic of the target service;
a code conversion unit 603 configured to execute a code segment corresponding to each target operator based on parameter configuration information of each target operator and a pre-configured code conversion template; and
Based on the upstream and downstream connection information between the respective target operators, the respective code segments are spliced to generate target codes for implementing the target service.
Optionally, the operator selection unit 601 is further configured to perform:
before the configuration description information corresponding to each target operator is displayed, analyzing a json file of each target operator which is configured in advance to obtain the configuration description information corresponding to each target operator; or (b)
Based on the corresponding relation between the pre-established operators and the configuration description information, the configuration description information corresponding to each target operator is obtained.
Optionally, the transcoding unit 603 is specifically configured to:
And respectively carrying out data conversion processing on the parameter configuration information of each target operator based on the code conversion templates corresponding to each target operator, and generating code segments corresponding to each target operator.
Optionally, the apparatus further comprises:
A code execution unit 604 configured to execute a code execution environment of the detection target code;
the object code is executed based on the detected code execution environment.
Optionally, the code running environment is a distributed cluster; the code execution unit 604 is configured to execute:
A distributed environment is built based on the detected code running environment, and object code is run in the built distributed environment.
Optionally, the code running environment is a single machine; the code execution unit 604 is configured to execute:
And running the target code on the stand-alone equipment corresponding to the detected code running environment.
Optionally, the object code is in a Notebook document format or a python format.
Optionally, the object code is in a Notebook document format; the apparatus further comprises:
The debug unit 605 is configured to execute debugging of the object code based on the notbook editor before running the object code.
Optionally, the object code further comprises environment dependency information for introducing an operator execution environment required to execute the respective object operator, and/or code base dependency information for introducing a code base required to execute the respective object operator.
Optionally, the object code includes environment dependent information; the transcoding unit 603 is further configured to perform:
detecting operator operation environments required by operating each target operator according to parameter configuration information of each target operator;
Merging and de-duplicating the detected operator operation environments, and generating environment dependency information for introducing operator operation environments required by operating each target operator;
the environment dependency information is added to the top of the object code.
Optionally, the object code includes code base dependency information; the transcoding unit 603 is further configured to perform:
Detecting a code library required by running each target operator according to the parameter configuration information of each target operator;
Merging and de-duplicating the detected code libraries, and generating code library dependent information for introducing the code libraries required by operating each target operator;
code base dependency information is added to the top of the object code.
The specific manner in which the respective units execute the requests in the apparatus of the above embodiment has been described in detail in the embodiment concerning the method, and will not be described in detail here.
Fig. 7 is a block diagram of an electronic device 700, according to an example embodiment, the apparatus comprising:
A processor 710;
a memory 720 for storing instructions executable by the processor 710;
Wherein the processor 710 is configured to execute instructions to implement the automatic programming control method in embodiments of the present disclosure.
In an exemplary embodiment, a storage medium is also provided, such as a memory 720 including instructions executable by the processor 710 of the electronic device 700 to perform the above-described method. Alternatively, the storage medium may be a non-transitory computer readable storage medium, for example, a ROM, random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.
The disclosed embodiments also provide a computer program product that, when run on an electronic device, causes the electronic device to perform a method that implements any of the above-described automatic programming control methods or any of the possible ways of automatic programming control methods of the disclosed embodiments.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This disclosure is intended to cover any adaptations, uses, or adaptations of the disclosure following the general principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It is to be understood that the present disclosure is not limited to the precise arrangements and instrumentalities shown in the drawings, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims (33)
1. An automatic programming control system, comprising: an operator management component and an automatic programming component;
the operator management component is used for providing a human-computer interaction interface, selecting target operators required for realizing target service according to operator selection instructions triggered by the human-computer interaction interface, and displaying configuration description information corresponding to each target operator; receiving parameter configuration information of each target operator and upstream and downstream connection information among the target operators, wherein the parameter configuration information and the upstream and downstream connection information are determined by processing logic of a target service based on the configuration description information;
The automatic programming component is used for calling an application programming interface API, and generating code segments corresponding to all target operators according to parameter configuration information of all target operators obtained by the operator management component based on the man-machine interaction interface and a pre-configured code conversion template; and splicing the code segments based on the upstream and downstream connection information between the target operators to generate target codes for realizing the target service, wherein the target codes further comprise environment dependency information for introducing operator running environments required for running the target operators and/or code base dependency information for introducing code bases required for running the target operators.
2. The system of claim 1, the operator management component further to:
before the configuration description information corresponding to each target operator is displayed, analyzing a json file of each target operator which is configured in advance to obtain the configuration description information corresponding to each target operator; or (b)
Based on the corresponding relation between the pre-established operators and the configuration description information, the configuration description information corresponding to each target operator is obtained.
3. The system of claim 1, wherein the automatic programming component is specifically configured to:
And respectively carrying out data conversion processing on the parameter configuration information of each target operator based on the code conversion templates corresponding to each target operator, and generating code segments corresponding to each target operator.
4. The system of claim 1, wherein the automatic programming component is further to:
after the splicing of the code segments to generate an object code for realizing the object service, detecting a code running environment of the object code; the object code is executed based on the detected code execution environment.
5. The system of claim 4, wherein the code execution environment is a distributed cluster; the automatic programming component is specifically configured to:
constructing a distributed environment based on the detected code running environment, and running the object code in the constructed distributed environment.
6. The system of claim 4, wherein the code execution environment is a stand-alone machine; the automatic programming component is specifically configured to:
and operating the target code on the stand-alone equipment corresponding to the detected code operating environment.
7. The system of claim 4, wherein the object code is in a notbook document format or a python format.
8. The system of claim 7, wherein the object code is in a notbook document format; the automatic programming component is further configured to:
The object code is debugged based on a Notebook editor before the object code is run.
9. The system of claim 1, wherein the object code includes the environment-dependent information; the automatic programming component is further configured to:
detecting operator operation environments required by operating each target operator according to parameter configuration information of each target operator;
Merging and de-duplicating the detected operator operation environments, and generating environment dependency information for introducing operator operation environments required by operating each target operator;
the environment dependency information is added to the top of the object code.
10. The system of claim 1, wherein the object code includes the code base dependency information; the automatic programming component is further configured to:
Detecting a code library required by running each target operator according to the parameter configuration information of each target operator;
Merging and de-duplicating the detected code libraries, and generating code library dependent information for introducing the code libraries required by operating each target operator;
the code base dependency information is added to the top of the object code.
11. An automatic programming control method, characterized in that it is applied to an automatic programming control system, comprising:
Selecting target operators required for realizing target business, and displaying configuration description information corresponding to each target operator;
Receiving input parameter configuration information of each target operator and upstream and downstream connection information between each target operator, wherein the parameter configuration information and the upstream and downstream connection information are determined based on the configuration description information and processing logic of the target service;
Generating code segments corresponding to each target operator based on parameter configuration information of each target operator and a pre-configured code conversion template; and
Based on the upstream and downstream connection information between the respective target operators, the respective code segments are spliced to generate target codes for realizing the target service, and the target codes further comprise environment dependency information for introducing operator running environments required for running the respective target operators and/or code base dependency information for introducing code bases required for running the respective target operators.
12. The method of claim 11, further comprising, prior to said displaying configuration description information corresponding to each target operator:
analyzing json files of each preset target operator to obtain configuration description information corresponding to each target operator; or (b)
Based on the corresponding relation between the pre-established operators and the configuration description information, the configuration description information corresponding to each target operator is obtained.
13. The method of claim 11, wherein the generating the code segment corresponding to each target operator based on parameter configuration information of each target operator and a pre-configured transcoding template specifically comprises:
And respectively carrying out data conversion processing on the parameter configuration information of each target operator based on the code conversion templates corresponding to each target operator, and generating code segments corresponding to each target operator.
14. The method of claim 11, further comprising, after said concatenating the respective code segments to generate an object code for implementing the target service:
detecting a code running environment of the target code;
the object code is executed based on the detected code execution environment.
15. The method of claim 14, wherein the code execution environment is a distributed cluster; the running the target code based on the detected code running environment specifically comprises:
constructing a distributed environment based on the detected code running environment, and running the object code in the constructed distributed environment.
16. The method of claim 14, wherein the code execution environment is a stand-alone machine; the running the target code based on the detected code running environment specifically comprises:
and operating the target code on the stand-alone equipment corresponding to the detected code operating environment.
17. The method of claim 14, wherein the object code is in a notbook document format or a python format.
18. The method of claim 17, wherein the object code is in a notbook document format; the method further comprises the steps of:
The object code is debugged based on a Notebook editor before the object code is run.
19. The method of claim 11, wherein the object code includes the environment dependent information; the splicing the code segments to generate the target code for realizing the target service further comprises:
detecting operator operation environments required by operating each target operator according to parameter configuration information of each target operator;
Merging and de-duplicating the detected operator operation environments, and generating environment dependency information for introducing operator operation environments required by operating each target operator;
the environment dependency information is added to the top of the object code.
20. The method of claim 11, wherein the object code includes the code base dependency information; the splicing the code segments to generate the target code for realizing the target service further comprises:
Detecting a code library required by running each target operator according to the parameter configuration information of each target operator;
Merging and de-duplicating the detected code libraries, and generating code library dependent information for introducing the code libraries required by operating each target operator;
the code base dependency information is added to the top of the object code.
21. An automatic programming control device, comprising:
the operator selection unit is configured to execute selection of target operators required for realizing target business and display configuration description information corresponding to each target operator;
a parameter configuration unit configured to perform receiving input parameter configuration information of each target operator and upstream and downstream connection information between each target operator, wherein the parameter configuration information and the upstream and downstream connection information are determined based on the configuration description information and processing logic of the target service;
A code conversion unit configured to execute a code segment corresponding to each target operator based on parameter configuration information of each target operator and a pre-configured code conversion template; and
Based on the upstream and downstream connection information between the respective target operators, the respective code segments are spliced to generate target codes for realizing the target service, and the target codes further comprise environment dependency information for introducing operator running environments required for running the respective target operators and/or code base dependency information for introducing code bases required for running the respective target operators.
22. The automatic programming control device of claim 21, wherein the operator selection unit is further configured to perform:
before the configuration description information corresponding to each target operator is displayed, analyzing a json file of each target operator which is configured in advance to obtain the configuration description information corresponding to each target operator; or (b)
Based on the corresponding relation between the pre-established operators and the configuration description information, the configuration description information corresponding to each target operator is obtained.
23. The apparatus of claim 21, wherein the transcoding unit is specifically configured to:
And respectively carrying out data conversion processing on the parameter configuration information of each target operator based on the code conversion templates corresponding to each target operator, and generating code segments corresponding to each target operator.
24. The apparatus of claim 21, wherein the apparatus further comprises:
A code execution unit configured to execute a code execution environment that detects the object code;
the object code is executed based on the detected code execution environment.
25. The apparatus of claim 24, wherein the code execution environment is a distributed cluster; the code execution unit is configured to execute:
constructing a distributed environment based on the detected code running environment, and running the object code in the constructed distributed environment.
26. The apparatus of claim 24, wherein the code execution environment is a stand-alone machine; the code execution unit is configured to execute:
and operating the target code on the stand-alone equipment corresponding to the detected code operating environment.
27. The apparatus of claim 24, wherein the object code is in a notbook document format or a python format.
28. The apparatus of claim 27, wherein the object code is in a notbook document format; the apparatus further comprises:
And the debugging unit is configured to execute debugging of the target code based on a Notebook editor before the target code is executed.
29. The apparatus of claim 21, wherein the object code comprises the environment dependent information; the transcoding unit is further configured to perform:
detecting operator operation environments required by operating each target operator according to parameter configuration information of each target operator;
Merging and de-duplicating the detected operator operation environments, and generating environment dependency information for introducing operator operation environments required by operating each target operator;
the environment dependency information is added to the top of the object code.
30. The apparatus of claim 21, wherein the object code includes the code base dependency information; the transcoding unit is further configured to perform:
Detecting a code library required by running each target operator according to the parameter configuration information of each target operator;
Merging and de-duplicating the detected code libraries, and generating code library dependent information for introducing the code libraries required by operating each target operator;
the code base dependency information is added to the top of the object code.
31. An electronic device, comprising:
A processor;
a memory for storing the processor-executable instructions;
Wherein the processor is configured to execute the instructions to implement the automatic programming control method of any one of claims 11 to 20.
32. A storage medium, wherein instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the automatic programming control method of any one of claims 11 to 20.
33. A computer program product comprising a computer program, characterized in that the computer program, when executed by a processor, implements the automatic programming control method of any one of claims 11 to 20.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010988772.6A CN112199086B (en) | 2020-09-18 | 2020-09-18 | Automatic programming control system, method, device, electronic equipment and storage medium |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010988772.6A CN112199086B (en) | 2020-09-18 | 2020-09-18 | Automatic programming control system, method, device, electronic equipment and storage medium |
Publications (2)
Publication Number | Publication Date |
---|---|
CN112199086A CN112199086A (en) | 2021-01-08 |
CN112199086B true CN112199086B (en) | 2024-07-09 |
Family
ID=74015659
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202010988772.6A Active CN112199086B (en) | 2020-09-18 | 2020-09-18 | Automatic programming control system, method, device, electronic equipment and storage medium |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN112199086B (en) |
Families Citing this family (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112965794B (en) * | 2021-02-19 | 2024-03-22 | 西安智周深鉴信息科技集团有限公司 | Algorithm calling method, electronic equipment and storage medium |
CN113434134B (en) * | 2021-06-21 | 2024-04-16 | 北京达佳互联信息技术有限公司 | Component processing method and device, electronic equipment and storage medium |
CN114358136B (en) * | 2021-12-10 | 2024-05-17 | 鹏城实验室 | Image data processing method and device, intelligent terminal and storage medium |
CN114115835B (en) * | 2022-01-27 | 2022-05-17 | 广州庚亿信息科技有限公司 | Automatic coding execution flow construction method and device, storage medium and intelligent terminal |
CN114218929B (en) * | 2022-02-22 | 2022-05-17 | 之江实验室 | Multi-platform operator intelligent development system and method based on meta-operator fusion |
CN114826899B (en) * | 2022-03-16 | 2024-01-23 | 青岛海尔科技有限公司 | Debugging method and device for equipment control service, storage medium and electronic device |
CN114879943A (en) * | 2022-06-30 | 2022-08-09 | 浙江大华技术股份有限公司 | Algorithm scheme generation method and device and computer readable storage medium |
CN116112479A (en) * | 2022-12-08 | 2023-05-12 | 联通(浙江)产业互联网有限公司 | http interface implementation method, device, equipment and storage medium |
CN117555533B (en) * | 2024-01-12 | 2024-04-12 | 北京天元特通科技有限公司 | Code generation method, electronic device and storage medium |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103309650A (en) * | 2012-03-14 | 2013-09-18 | 阿里巴巴集团控股有限公司 | Generating method and device of persistence layer codes |
CN109783084A (en) * | 2019-01-18 | 2019-05-21 | 广州思坦德计算机信息科技有限公司 | A kind of computer code automatic generation method, system, device and storage medium |
CN110442329A (en) * | 2019-06-26 | 2019-11-12 | 平安科技(深圳)有限公司 | Generation method, device, storage medium and the computer equipment of code segment |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104679511A (en) * | 2015-02-10 | 2015-06-03 | 北京***工程研究所 | Method for generating MapReduce codes based on MDE (Model-Driven Engineering) model transformation |
US20160357519A1 (en) * | 2015-06-05 | 2016-12-08 | Microsoft Technology Licensing, Llc | Natural Language Engine for Coding and Debugging |
CN106919434B (en) * | 2017-03-22 | 2021-02-12 | 恒生电子股份有限公司 | Code generation method and device |
US10678522B1 (en) * | 2018-12-14 | 2020-06-09 | Mouri Tech Llc | Compiler and method for compiling business rules for a serverless runtime environment |
CN110297632A (en) * | 2019-06-12 | 2019-10-01 | 百度在线网络技术(北京)有限公司 | Code generating method and device |
CN110928529B (en) * | 2019-11-06 | 2021-10-26 | 第四范式(北京)技术有限公司 | Method and system for assisting operator development |
CN111427561A (en) * | 2020-03-26 | 2020-07-17 | 中国建设银行股份有限公司 | Service code generation method and device, computer equipment and storage medium |
-
2020
- 2020-09-18 CN CN202010988772.6A patent/CN112199086B/en active Active
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103309650A (en) * | 2012-03-14 | 2013-09-18 | 阿里巴巴集团控股有限公司 | Generating method and device of persistence layer codes |
CN109783084A (en) * | 2019-01-18 | 2019-05-21 | 广州思坦德计算机信息科技有限公司 | A kind of computer code automatic generation method, system, device and storage medium |
CN110442329A (en) * | 2019-06-26 | 2019-11-12 | 平安科技(深圳)有限公司 | Generation method, device, storage medium and the computer equipment of code segment |
Also Published As
Publication number | Publication date |
---|---|
CN112199086A (en) | 2021-01-08 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN112199086B (en) | Automatic programming control system, method, device, electronic equipment and storage medium | |
CN107450902B (en) | Method and system for visual modeling | |
US20200356350A1 (en) | Methods and Systems for Visual Programming using Polymorphic, Dynamic Multi-Dimensional Structures | |
CN110825362A (en) | Low-code application software development system and method | |
US9087296B2 (en) | Navigable semantic network that processes a specification to and uses a set of declaritive statements to produce a semantic network model | |
CN115617327A (en) | Low code page building system, method and computer readable storage medium | |
US10114619B2 (en) | Integrated development environment with multiple editors | |
CN114625353A (en) | Model framework code generation system and method | |
US20170139685A1 (en) | Visual software modeling method to construct software views based on a software meta view | |
CN110941427B (en) | Code generation method and code generator | |
CN112860260B (en) | Web-based cross-platform application construction tool and method in college scene | |
US11861469B2 (en) | Code generation for Auto-AI | |
US20120166977A1 (en) | User programming access to data model in user interface design | |
US10776351B2 (en) | Automatic core data service view generator | |
CN117389541A (en) | Configuration system and device for generating template based on dialogue retrieval | |
CN114117645B (en) | Ship overall performance forecasting integrated application system | |
US20220308841A1 (en) | Enabling custom software development by domain experts | |
CN106096159A (en) | Distributed system behavior simulation under a kind of cloud platform analyzes the implementation method of system | |
US20240086165A1 (en) | Systems and methods for building and deploying machine learning applications | |
US20240037325A1 (en) | Ability to add non-direct ancestor columns in child spreadsheets | |
CN115469860A (en) | Method and system for automatically generating demand-to-software field model based on instruction set | |
CN114238072A (en) | HIL requirement design method, HIL requirement design device, computer equipment and storage medium | |
Buck Woody et al. | Data Science with Microsoft SQL Server 2016 | |
CN110019177A (en) | The method and apparatus of rule storage | |
Okamoto et al. | SUNPRISM: An approach and software tools for collaborative climate change research |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |