CN116450382A - Data processing method and system based on function definition - Google Patents
Data processing method and system based on function definition Download PDFInfo
- Publication number
- CN116450382A CN116450382A CN202310722721.2A CN202310722721A CN116450382A CN 116450382 A CN116450382 A CN 116450382A CN 202310722721 A CN202310722721 A CN 202310722721A CN 116450382 A CN116450382 A CN 116450382A
- Authority
- CN
- China
- Prior art keywords
- algorithm
- function definition
- data
- data processing
- parameters
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 26
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 266
- 238000000034 method Methods 0.000 claims abstract description 34
- 230000008569 process Effects 0.000 claims abstract description 14
- 230000006870 function Effects 0.000 claims description 99
- 238000012545 processing Methods 0.000 claims description 28
- 238000004458 analytical method Methods 0.000 claims description 18
- 230000003993 interaction Effects 0.000 claims description 10
- 230000007613 environmental effect Effects 0.000 abstract description 4
- 238000005538 encapsulation Methods 0.000 abstract description 3
- 238000002955 isolation Methods 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 10
- 238000004590 computer program Methods 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000009133 cooperative interaction Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 238000012856 packing Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Stored Programmes (AREA)
Abstract
The invention discloses a data processing method and a system based on function definition, wherein the method comprises the following steps: obtaining a function definition instruction, wherein the function definition instruction comprises: algorithm version information, algorithm input parameters, algorithm output parameters and environment variables; the method comprises the steps of defining basic algorithm information, algorithm input parameters, algorithm output parameters and environment variables in a unified format in an algorithm template mode; determining an algorithm package corresponding to the algorithm version information based on the function definition instruction, and running the algorithm package in a containerized or independent process or service mode; and acquiring data from the message queue or the network channel or an external input source based on the algorithm packet, analyzing the data, and inputting the analyzed data into a new message queue or the network channel. The invention can realize the isolation of the running environment, and realizes the unification of formats of input and output parameters, environmental parameters and the like of different algorithms of each manufacturer through the encapsulation definition of the algorithm template, and the multiple algorithms of multiple devices can cooperatively interact.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data processing method and system based on function definition.
Background
The final core of the development of the Internet of things is the comparison of cloud technologies, and with the rapid increase of the number of terminal devices of the Internet of things, the traditional centralized data processing mode based on a cloud computing model cannot effectively process mass data generated by network edge devices due to the problems of limited network bandwidth, high transmission cost, high response delay and the like.
Methods for distributing software to a terminal device for analysis are already known in the prior art. For example, the platform transmits the software package to the terminal device through the network, and updates and upgrades the specific software of the specific terminal device, which has the disadvantages of being limited by the device model and the operation environment, and the function definition depends on the device chip customization and the algorithm customization. Different equipment can only run algorithms customized by home platforms or manufacturers, so that the equipment of each manufacturer are mutually independent, the algorithms cannot be used across the equipment of different manufacturers, a data collaboration mechanism is not provided, the high-efficiency utilization of computing resources cannot be realized, the service realization cost is high, and the period is long; the function modules of the equipment are all preset and do not support customization.
Accordingly, there is a need for improvement and advancement in the art.
Disclosure of Invention
Aiming at the defects of the prior art, the invention provides a data processing method and system based on function definition, which aims to solve the problems that different equipment in the prior art can only run algorithms customized by a home platform or manufacturer, so that the equipment of each manufacturer is mutually independent, the algorithms cannot be used across the equipment of different manufacturers, no data collaboration mechanism exists, the high-efficiency utilization of computing resources cannot be realized, the service realization cost is high, and the period is long; the function modules of the equipment are all preset and do not support the problem of self definition.
In a first aspect, the present invention provides a data processing method based on a function definition, wherein the method includes:
obtaining a function definition instruction, wherein the function definition instruction comprises: algorithm version information, algorithm basic information, algorithm input parameters, algorithm output parameters and environment variables;
the method comprises the steps of defining basic algorithm information, input algorithm parameters, output algorithm parameters and environment variables in a unified format in an algorithm template mode, wherein the basic algorithm information comprises the following components: algorithm authors, creation time, algorithm description, model framework, model storage format, hardware type, resource requirements;
determining an algorithm package corresponding to the algorithm version information based on the function definition instruction, and running and analyzing the algorithm package in a containerized or independent process or service mode;
and acquiring data from the message queue or the designated network channel or an external input source based on the algorithm packet, analyzing the data, inputting the analyzed data into a new message queue or network channel, and importing the analyzed data into a result distribution service.
In one implementation manner, the determining, based on the function definition instruction, an algorithm package corresponding to the algorithm version information includes:
analyzing the function definition instruction and determining the algorithm version information in the function definition instruction;
and pulling the algorithm package corresponding to the algorithm version information from the algorithm bin based on the algorithm version information.
In one implementation, before the acquiring the function definition instruction, the method includes:
acquiring an issued analysis task, selecting an algorithm flow based on the analysis task, and starting to acquire an issued function definition instruction.
In one implementation, the acquiring the issued analysis task includes:
and if the analysis tasks are multiple, uniformly distributing the analysis tasks to each equipment cluster according to the available resource conditions, and communicating the equipment in each equipment cluster by a network.
In one implementation, the device with the most abundant resources is selected as the central node when the function definition instruction is issued.
In one implementation, each algorithm loads own input and output queue names through environment variables, and performs data interaction through a message queue; or, inputting and outputting network channel protocol, IP address or URL information and port information, and carrying out data interaction through the network channel.
In one implementation, the method further comprises:
and summarizing the analyzed data based on the result distribution service, and reporting the generated result stream and the generated characteristic stream to a cloud/side system.
In a second aspect, an embodiment of the present invention further provides a data processing system based on a function definition, where the system includes:
the function definition module is used for acquiring function definition instructions, and the function definition instructions comprise: algorithm version information, algorithm basic information, algorithm input parameters, algorithm output parameters and environment variables;
the format unifying module is used for defining the unified format of the basic algorithm information, the input algorithm parameters, the output algorithm parameters and the environment variables in a mode of an algorithm template, wherein the basic algorithm information comprises the following components: algorithm authors, creation time, algorithm description, model framework, model storage format, hardware type, resource requirements;
the algorithm module is used for determining an algorithm package corresponding to the algorithm version information based on the function definition instruction and running and analyzing the algorithm package in a containerized or independent process or service mode;
the data processing module is used for acquiring data from the message queue or the appointed network channel or the external input source based on the algorithm packet, analyzing the data, inputting the analyzed data to a new message queue or network channel, and converging the analyzed data into a result distribution service.
In a third aspect, an embodiment of the present invention further provides an apparatus, where the apparatus includes a memory, a processor, and a data processing program stored in the memory and capable of running a function definition based data processing program on the processor, and when the processor executes the function definition based data processing program, the processor implements the steps of the function definition based data processing method according to any one of the above schemes.
In a fourth aspect, an embodiment of the present invention further provides a computer readable storage medium, where a data processing program based on a function definition is stored on the computer readable storage medium, where the data processing program based on the function definition implements the steps of the data processing method based on the function definition according to any one of the above aspects when the data processing program based on the function definition is executed by a processor.
The beneficial effects are that: compared with the prior art, the invention provides a data processing method based on function definition, which comprises the steps of firstly obtaining a function definition instruction, wherein the function definition instruction comprises the following components: algorithm version information, algorithm basic information, algorithm input parameters, algorithm output parameters and environment variables; the method comprises the steps of defining basic algorithm information, algorithm input parameters, algorithm output parameters and environment variables in a unified format in an algorithm template mode; determining an algorithm package corresponding to the algorithm version information based on the function definition instruction, and running and analyzing the algorithm package in a containerized or independent process and service mode; and acquiring data from the message queue or the designated network channel or an external input source based on the algorithm packet, analyzing the data, inputting the analyzed data into a new message queue or network channel, and importing the analyzed data into a result distribution service. The invention realizes the isolation of the running environment by running the algorithm model in the equipment in a containerized or independent process or service mode, realizes the unification of different formats of the input and output parameters of the algorithms, the environmental parameters and the like of each manufacturer by the encapsulation definition of the algorithm template, and can cooperatively interact with a plurality of algorithms of a plurality of equipment.
Drawings
Fig. 1 is a flowchart of a specific implementation of a data processing method based on a function definition according to an embodiment of the present invention.
Fig. 2 is a schematic operation diagram of an algorithm model container in a data processing method based on function definition according to an embodiment of the present invention.
Fig. 3 is an algorithm template diagram in a data processing method based on function definition according to an embodiment of the present invention.
Fig. 4 is an algorithm description file diagram in a data processing method based on function definition according to an embodiment of the present invention.
Fig. 5 is a main processing function structure diagram in the data processing method based on function definition according to the embodiment of the present invention.
Fig. 6 is a diagram illustrating a main processing function code in a data processing method based on a function definition according to an embodiment of the present invention.
Fig. 7 is a diagram illustrating an application of the data processing method based on function definition in the single device function definition instruction issue.
Fig. 8 is a multi-algorithm collaborative graph when a data processing method based on function definition provided in an embodiment of the present invention is applied to multiple devices.
FIG. 9 is a functional schematic of a data processing system based on functional definitions provided by an embodiment of the present invention.
Fig. 10 is a schematic block diagram of an apparatus according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and effects of the present invention clearer and more specific, the present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
The embodiment provides a data processing method based on function definition, and the method firstly obtains a function definition instruction, where the function definition instruction includes: algorithm version information, algorithm basic information, algorithm input parameters, algorithm output parameters and environment variables; the method comprises the steps of defining basic algorithm information, algorithm input parameters, algorithm output parameters and environment variables in a unified format in an algorithm template mode; determining an algorithm package corresponding to the algorithm version information based on the function definition instruction, and running and analyzing the algorithm package in a containerized or independent process or service mode; and acquiring data from the message queue or the designated network channel or an external input source based on the algorithm packet, analyzing the data, inputting the analyzed data into a new message queue or network channel, and importing the analyzed data into a result distribution service. In the embodiment, the algorithm model is containerized or an independent process or service mode is operated on the equipment, so that the isolation of the runtime environment is realized; supporting an algorithm manufacturer to upload an algorithm, defining input and output parameter specifications of the algorithm by providing an algorithm template, and unifying different algorithm parameter format definitions; the whole intelligent terminal resource is pooled, the algorithm can be scheduled to the optimal equipment operation, and multiple algorithms of multiple equipment can cooperatively interact; the full flow of function definition is automatic, one-key model uploading, algorithm input/output updating, algorithm packing, algorithm issuing updating and algorithm result table automatic updating are completed, and the function definition cost is reduced.
Exemplary method
The data processing method based on the function definition of the embodiment can be applied to equipment, wherein the equipment comprises computers, intelligent televisions and other intelligent product terminals. As shown in fig. 1, the data processing method based on the function definition of the present embodiment includes the following:
step S100, obtaining a function definition instruction, where the function definition instruction includes: algorithm version information, algorithm basic information, algorithm input parameters, algorithm output parameters and environment variables;
step 200, defining the basic information of the algorithm, the input parameters of the algorithm, the output parameters of the algorithm and the environment variables in a unified format in a mode of an algorithm template;
step S300, determining an algorithm package corresponding to the algorithm version information based on the function definition instruction, and running and analyzing the algorithm package in a containerized or independent process or service mode;
step 400, acquiring data from a message queue or a designated network channel or an external input source based on the algorithm packet, analyzing the data, inputting the analyzed data into a new message queue or a new network channel, and converging the analyzed data into a result distribution service.
In this embodiment, as shown in fig. 2, both the algorithm and the message queue operate in a containerized or stand-alone process or service, and the algorithm and the message queue may operate on different devices or on the same device. The message queues are used as data interactions between devices or algorithms. Algorithm inputs take the output of other algorithms from external input sources (video, images, etc.) or from queues or from network channels, and then output the algorithm results to new queues or network channels.
And, each algorithm program is based on the algorithm template to realize internal processing and interaction with the outside. As shown in FIG. 3, a default algorithm template is built in, and the directory structure of the algorithm template is agreed according to Bin, script, data, META-INF:
bin: program files for storing algorithm programs, main processing functions (used for packaging user algorithms as program entries), algorithm models and other files;
script: the algorithm program starts, stops and queries the script;
data: storing sample data such as picture files and video files for demonstrating the functional effects of the algorithm;
META-INF: the algorithm information describes a file.
The algorithm information description file is a core for realizing interaction among algorithms, is provided in a yaml format, and is divided into the following four parts as shown in fig. 4:
algorithm basic information: including algorithm author, creation time, algorithm description, version number, model framework, model storage format, hardware type, resource requirements, etc.;
input format definition: including data type, basic parameters (including task id, binary data length, algorithm information, etc.), service parameters (defined according to a specific algorithm, such as frame rate of a frame extraction algorithm), and custom parameters;
output format definition: including data type, basic parameters (including task id, binary data length, algorithm information, etc.), business parameters (defined according to a specific algorithm, such as the format of an output image frame), custom parameters;
environmental variable definition: message queue connection configuration information, a queue name input by an algorithm and a queue name output by the algorithm; or the message queue is connected with the network channel protocol, IP address or URL information, port information, video source information, custom environment variables and the like which are input and output.
For determinable static information, the data (such as input and output parameters) to be dynamically transferred can be configured into $ { xxx } format, and when the algorithm program runs, the data is dynamically transferred into the algorithm. The algorithm framework generating tool automatically generates an algorithm main processing function framework according to the definition of the algorithm information description file in the algorithm template.
As shown in fig. 5, the main processing function of the algorithm of the present embodiment is responsible for the internal processing of the algorithm and interaction with the outside, and mainly consists of the following three parts:
the algorithm inputs parameters: the format is JSON+ binary data, wherein the JSON data comprises basic parameters, service parameters and user-defined parameters, and the binary data (not necessary) comprises picture data and characteristic data;
algorithm output parameters: the format is JSON+ binary data, wherein the JSON data comprises basic parameters, service parameters and user-defined parameters, and the binary data (not necessary) comprises picture data and characteristic data;
environmental variable: message queue connection configuration information, a queue name input by an algorithm and a queue name output by the algorithm; or the message queue is connected with the network channel protocol, IP address or URL information, port information, video source information, custom environment variables and the like which are input and output.
For convenience of expansion, the JSON data may be defined in JSON Schema format.
The main processing function of the algorithm shields the acquisition and the transmission of the output of the input of the algorithm, and a user only needs to pay attention to the development of the core code of the algorithm, and then the main processing function is nested to interact with other algorithms. The main processing function code of this embodiment is shown in fig. 6.
In a specific application, as shown in fig. 7, an issued analysis task is acquired, an algorithm flow is selected based on the analysis task, and an issued function definition instruction is started to be acquired. Then, issuing a function definition instruction to the device, wherein the function definition instruction comprises algorithm version information and algorithm template information, and the algorithm template information comprises: algorithm basic information, algorithm input parameters (external input source parameters or names of input message queues or protocols of input network channels, IP address information, ports and the like), algorithm output parameters (names of output message queues or protocols of output network channels, IP address information, ports and the like) and environment variables. The parameters described above may also be sent via a function definition instruction + algorithm description information file. After receiving the instruction, the device analyzes the function definition instruction and determines the algorithm version information in the function definition instruction. And then pulling the algorithm package corresponding to the algorithm version information from the algorithm bin (i.e., the algorithm model bin in fig. 7) based on the algorithm version information. The device then obtains data from the message queue or designated network channel or external input source based on the algorithm package, analyzes the data, and inputs the analyzed data into the new message queue or network channel. For example, the algorithm A acquires data from the message queue 1 or an external input source (video, image and the like), analyzes the data and outputs the data to the new message queue 2, and realizes data interaction among the algorithms through the queues; the data of the message queue 2 is imported into the result distribution service and can also be continuously input as data of other algorithms. And finally, the result distribution service is responsible for final data summarization, and the generated result stream and the generated feature stream are reported to the cloud/side system.
In another implementation, as shown in fig. 8, when there are multiple devices, the user selects an algorithm, multiple devices, a scheduling policy, and the like, issues batch analysis tasks, and the scheduling module formulates an optimal scheduling scheme according to the scheduling policy. And if the analysis tasks are multiple, uniformly distributing the analysis tasks to each equipment cluster according to the available resource conditions, and communicating the equipment in each equipment cluster by a network. For a single analysis task, the analysis task is further split into a plurality of subtasks, and the subtasks are distributed to a plurality of devices in the same cluster. The function definition instruction of this embodiment selects the most abundant device as the central node when issuing, deploys the message queue and the result distribution service (if there is still a surplus, the algorithm can also be run, and the data analysis is participated). The devices in the same cluster can directly interact data through the message queue or the network channel, so that the device data is prevented from being transferred by a cloud/side system, the network delay is reduced, and the overall data processing performance is effectively improved. Each device independently runs different algorithms, each algorithm loads own input and output queue names through environment variables, and data interaction is carried out through the message queues.
Specifically, as shown in fig. 8, algorithm a (i.e., algorithm mirror image a in fig. 8) runs on device 1 (i.e., terminal device 1 in fig. 8), algorithm B and algorithm C are different algorithms, and run on device 2 and device 3 respectively; algorithms D1 and D2 are the same algorithm running on device 4 and device 5, respectively. And outputting the result of the algorithm A to a message queue 1, acquiring data from the message queue 1 by the algorithm B, and monitoring the output queue of the algorithm A by the algorithm B to realize algorithm concatenation. And outputting the result of the algorithm A to a message queue 1, and acquiring data from the message queue 1 by using an algorithm B and an algorithm C, wherein the algorithm B and the algorithm C are in a parallel connection relationship, and the algorithm B and the algorithm C consume the complete result of the algorithm A independently and output the result of the algorithm to different queues. And outputting the result of the algorithm C to a message queue 3, acquiring data from the message queue 3 by both the algorithm D1 and the algorithm D2, wherein the algorithms D1 and D2 are multiple copies of the same algorithm, and sharing the output result of the consumption algorithm C by the algorithms D1 and D2 together, and outputting the algorithm result to the same queue.
Therefore, the embodiment unifies format definitions such as algorithm input and output based on containerization or independent process/service and template encapsulation, solves the problem that different devices can only run algorithms customized by home platforms or manufacturers, and realizes the intercommunication and sharing of the algorithms. In addition, the embodiment realizes the cooperative interaction of multiple algorithms of multiple devices through the issuing of the function definition instruction.
Exemplary System
Based on the above embodiment, the present invention further provides a data processing system based on function definition, as shown in fig. 9, where the system includes: a function definition module 10, a format unification module 20, an algorithm module 30, and a data processing module 40. Specifically, the function definition module 10 is configured to obtain function definition instructions, where the function definition instructions include: algorithm version information, algorithm basic information, algorithm input parameters, algorithm output parameters, and environment variables. The format unifying module 20 is configured to perform definition of a unified format on the basic algorithm information, the input algorithm parameters, the output algorithm parameters and the environment variables by using an algorithm template. The algorithm module 30 is configured to determine an algorithm package corresponding to the algorithm version information based on the function definition instruction, and run and analyze the algorithm package in a manner of containerization or an independent process or service. The data processing module 40 is configured to obtain data from a message queue or a designated network channel or an external input source based on the algorithm packet, analyze the data, input the analyzed data into a new message queue or a network channel, and collect the analyzed data into a result distribution service.
The working principle of each module in the data processing system based on functional definition in this embodiment is the same as that of each step in the above method embodiment, and will not be described here again.
Based on the above embodiments, the present invention also provides an apparatus, and a schematic block diagram of the apparatus may be shown in fig. 10. The device may comprise one or more processors 100 (only one shown in fig. 10), a memory 101 and a computer program 102 stored in the memory 101 and executable on the one or more processors 100, e.g. a data processing program based on a functional definition. The functions of the various modules/units in the data processing system embodiments based on the functional definition may be implemented by one or more processors 100 when executing computer program 102, and are not limited in this regard.
In one embodiment, the processor 100 may be a central processing unit (Central Processing Unit, CPU), but may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
In one embodiment, the memory 101 may be an internal storage unit of the electronic device, such as a hard disk or a memory of the electronic device. The memory 101 may also be an external storage device of the electronic device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash card (flash card) or the like, which are provided on the electronic device. Further, the memory 101 may also include both an internal storage unit and an external storage device of the electronic device. The memory 101 is used to store computer programs and other programs and data required by the device. The memory 101 may also be used to temporarily store data that has been output or is to be output.
It will be appreciated by persons skilled in the art that the functional block diagram shown in FIG. 10 is merely a block diagram of some of the structures associated with the present inventive arrangements and is not limiting of the apparatus to which the present inventive arrangements may be applied, as specific apparatus may include more or fewer components than shown, or may be combined with certain components, or may have different arrangements of components.
Those skilled in the art will appreciate that implementing all or part of the above-described methods may be accomplished by way of a computer program, which may be stored on a non-transitory computer readable storage medium and which, when executed, may comprise the steps of the above-described embodiments of the methods. Any reference to memory, storage, operational databases, or other media used in the various embodiments provided herein may include non-volatile and volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual operation data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. A method of data processing based on a function definition, the method comprising:
obtaining a function definition instruction, wherein the function definition instruction comprises: algorithm version information, algorithm basic information, algorithm input parameters, algorithm output parameters and environment variables;
the method comprises the steps of defining basic algorithm information, input algorithm parameters, output algorithm parameters and environment variables in a unified format in an algorithm template mode, wherein the basic algorithm information comprises the following components: algorithm authors, creation time, algorithm description, model framework, model storage format, hardware type, resource requirements;
determining an algorithm package corresponding to the algorithm version information based on the function definition instruction, and running the algorithm package in a containerized or independent process or service mode;
and acquiring data from the message queue or the network channel or an external input source based on the algorithm packet, analyzing the data, inputting the analyzed data into a new message queue or the network channel, and converging the analyzed data into a result distribution service.
2. The function definition based data processing method according to claim 1, wherein the determining an algorithm package corresponding to the algorithm version information based on the function definition instruction includes:
analyzing the function definition instruction and determining the algorithm version information in the function definition instruction;
and pulling the algorithm package corresponding to the algorithm version information from the algorithm bin based on the algorithm version information.
3. The function definition based data processing method according to claim 1, wherein before the function definition instruction is acquired, comprising:
acquiring an issued analysis task, selecting an algorithm flow based on the analysis task, and starting to acquire an issued function definition instruction.
4. A method of data processing based on functional definitions according to claim 3, wherein said obtaining an issued analysis task comprises:
and if the analysis tasks are multiple, uniformly distributing the analysis tasks to each equipment cluster according to the available resource conditions, and communicating the equipment in each equipment cluster by a network.
5. The method for data processing based on function definition according to claim 4, wherein the device with the most abundant resources is selected as a central node when the function definition command is issued.
6. The data processing method based on function definition according to claim 4, wherein the devices in each device cluster independently run different algorithms, each algorithm loads own input and output queue names through environment variables, and performs data interaction through message queues; or, the input and output network channel protocol, IP address or URL information and port information are used for data interaction through the network channel.
7. The function definition based data processing method according to claim 1, wherein the method further comprises:
and summarizing the analyzed data based on the result distribution service, and reporting the generated result stream and the generated characteristic stream to a cloud/side system.
8. A data processing system based on a functional definition, the system comprising:
the function definition module is used for acquiring function definition instructions, and the function definition instructions comprise: algorithm version information, algorithm basic information, algorithm input parameters, algorithm output parameters and environment variables;
the format unifying module is used for defining the unified format of the basic algorithm information, the input algorithm parameters, the output algorithm parameters and the environment variables in a mode of an algorithm template, wherein the basic algorithm information comprises the following components: algorithm authors, creation time, algorithm description, model framework, model storage format, hardware type, resource requirements;
the algorithm module is used for determining an algorithm package corresponding to the algorithm version information based on the function definition instruction and running and analyzing the algorithm package in a containerized or independent process or service mode;
the data processing module is used for acquiring data from the message queue or the designated network channel or an external input source based on the algorithm packet, analyzing the data, inputting the analyzed data into a new message queue or network channel, and converging the analyzed data into a result distribution service.
9. An apparatus comprising a memory, a processor and a function definition based data processing program stored in the memory and executable on the processor, the processor implementing the steps of the function definition based data processing method according to any one of claims 1-7 when executing the function definition based data processing program.
10. A computer-readable storage medium, on which a data processing program based on a function definition is stored, which, when being executed by a processor, implements the steps of the data processing method based on a function definition as claimed in any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310722721.2A CN116450382A (en) | 2023-06-19 | 2023-06-19 | Data processing method and system based on function definition |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310722721.2A CN116450382A (en) | 2023-06-19 | 2023-06-19 | Data processing method and system based on function definition |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116450382A true CN116450382A (en) | 2023-07-18 |
Family
ID=87130628
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310722721.2A Pending CN116450382A (en) | 2023-06-19 | 2023-06-19 | Data processing method and system based on function definition |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116450382A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110311973A (en) * | 2019-06-28 | 2019-10-08 | 东北大学 | A kind of image cloud service system and method towards multi-tenant |
| CN114489925A (en) * | 2021-12-09 | 2022-05-13 | 广东电网有限责任公司 | Containerized service scheduling framework and flexible scheduling algorithm |
| CN114896038A (en) * | 2022-04-28 | 2022-08-12 | 北京和利时***工程有限公司 | Method and system for managing and scheduling data analysis algorithms |
-
2023
- 2023-06-19 CN CN202310722721.2A patent/CN116450382A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110311973A (en) * | 2019-06-28 | 2019-10-08 | 东北大学 | A kind of image cloud service system and method towards multi-tenant |
| CN114489925A (en) * | 2021-12-09 | 2022-05-13 | 广东电网有限责任公司 | Containerized service scheduling framework and flexible scheduling algorithm |
| CN114896038A (en) * | 2022-04-28 | 2022-08-12 | 北京和利时***工程有限公司 | Method and system for managing and scheduling data analysis algorithms |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11243704B2 (en) | Data pipeline architecture for analytics processing stack | |
| CN110888722B (en) | Task processing method and device, electronic equipment and computer readable storage medium | |
| US10831562B2 (en) | Method and system for operating a data center by reducing an amount of data to be processed | |
| CN108510082B (en) | Method and device for processing machine learning model | |
| US10996878B2 (en) | Data pipeline architecture for cloud processing of structured and unstructured data | |
| CN110764752A (en) | System and method for realizing graphical service arrangement of Restful service based on micro-service architecture | |
| US20220014584A1 (en) | Distributed pipeline configuration in a distributed computing system | |
| WO2022083512A1 (en) | Information sharing method, apparatus and server, and storage medium | |
| AU2017254506A1 (en) | Method, apparatus, computing device and storage medium for data analyzing and processing | |
| CN111858727A (en) | Multi-data-source data export system and method based on template configuration | |
| CN112052011A (en) | Method and device for combining small programs, electronic equipment and medium | |
| CN116450382A (en) | Data processing method and system based on function definition | |
| CN110636073A (en) | Distributed asynchronous remote service arranging and calling technology | |
| CN116226045A (en) | File data aggregation method, file data aggregation device and query system | |
| WO2022111466A1 (en) | Task scheduling method, control method, electronic device and computer-readable medium | |
| CN111190607B (en) | Task plugin processing method and device, task scheduling server and storage medium | |
| CN113742385A (en) | Data query method and device | |
| CN112637288A (en) | Streaming data distribution method and system | |
| CN112445811A (en) | Data service method, device, storage medium and component based on SQL configuration | |
| CN117435367B (en) | User behavior processing method, device, equipment, storage medium and program product | |
| WO2023035147A1 (en) | Data processing method of industry edge product and distributed computing protocol engine thereof | |
| CN112380023B (en) | Distributed model calculation control method and device | |
| US20240146800A1 (en) | Data plane reduction for eventing components | |
| CN117707602A (en) | Data request processing method, device, server and storage medium | |
| CN113778392A (en) | Method and device for realizing streaming development based on configuration |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20230718 |