CN114217885A - Data processing method, device and storage medium - Google Patents
Data processing method, device and storage medium Download PDFInfo
- Publication number
- CN114217885A CN114217885A CN202111553718.XA CN202111553718A CN114217885A CN 114217885 A CN114217885 A CN 114217885A CN 202111553718 A CN202111553718 A CN 202111553718A CN 114217885 A CN114217885 A CN 114217885A
- Authority
- CN
- China
- Prior art keywords
- development
- data processing
- components
- component
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 12
- 238000011161 development Methods 0.000 claims abstract description 310
- 238000012545 processing Methods 0.000 claims abstract description 98
- 238000013515 script Methods 0.000 claims abstract description 66
- 238000000034 method Methods 0.000 claims abstract description 47
- 230000015654 memory Effects 0.000 claims description 29
- 238000004590 computer program Methods 0.000 claims description 26
- 238000007405 data analysis Methods 0.000 abstract description 2
- 238000013461 design Methods 0.000 description 13
- 238000010586 diagram Methods 0.000 description 10
- 230000001360 synchronised effect Effects 0.000 description 7
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000003068 static effect Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000007667 floating Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000005096 rolling process Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/448—Execution paradigms, e.g. implementations of programming paradigms
- G06F9/4482—Procedural
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Stored Programmes (AREA)
Abstract
The application provides a data processing method, data processing equipment and a storage medium, which are applicable to the field of data analysis. The method is applied to a big data development platform and comprises the following steps: acquiring n target development components, wherein the n target development components comprise m first development components, n is larger than or equal to m and larger than 0, the first development components do not belong to the big data development platform, generating a script of a data processing task by running the n target development components, and completing the data processing task by running the script of the data processing task. The convenience of big data development and the efficiency of data processing are improved.
Description
Technical Field
The present application relates to the field of data analysis, and in particular, to a data processing method, device, and storage medium.
Background
Big data development platforms are often used to provide a development environment for executable scripts that address the processing tasks of big data. The big data development platform can be packaged with a plurality of development components, each development component is used for developing a functional script, and when developers need to develop programs aiming at data processing tasks, the development components in the big data development platform can be used for developing the scripts corresponding to the data processing tasks.
In some development scenarios, development components packaged in a big data development platform may not meet development requirements of data processing tasks, and need to be combined with external development components to develop scripts corresponding to the data processing tasks. However, since the development components packaged by different data development platforms are incompatible, the big data development platform cannot run the development components outside the big data development platform, and in this case, a developer needs to run the development components in a plurality of big data development platforms respectively to obtain scripts of data processing tasks, which results in low data processing efficiency.
Disclosure of Invention
The application provides a data processing method, a device and a storage medium, which aim to improve the efficiency of data processing.
In a first aspect, an embodiment of the present application provides a data processing method, which is applied to a big data development platform, and includes: acquiring n target development components, wherein the n target development components comprise m first development components, n is more than or equal to m and is more than 0, and the first development components do not belong to the big data development platform; generating a script of a data processing task by running n target development components; and completing the data processing task by running the script of the data processing task.
In a second aspect, an embodiment of the present application provides an electronic device, where a big data development platform is deployed in the electronic device, and the electronic device includes: the acquisition unit is used for acquiring n target development components, wherein the n target development components comprise m first development components, n is more than or equal to m and is more than 0, and the first development components do not belong to the big data development platform; the processing unit is used for generating a script of a data processing task by running the n target development components; the processing unit is also used for completing the data processing task by running the script of the data processing task.
In a third aspect, an embodiment of the present application provides an electronic device, including: a processor and a memory, the memory being configured to store a computer program, the processor being configured to invoke and execute the computer program stored in the memory to perform a method as in the first aspect or in each possible implementation.
In a fourth aspect, an embodiment of the present application provides a chip, including: a processor, configured to call and execute the computer instructions from the memory, so that the device on which the chip is installed performs the method according to the first aspect or each possible implementation manner.
In a fifth aspect, embodiments of the present application provide a computer-readable storage medium for storing computer program instructions, where the computer program instructions cause a computer to execute the method according to the first aspect or each possible implementation manner.
In a sixth aspect, the present application provides a computer program product, which includes computer program instructions, and the computer program instructions make a computer execute the method in the first aspect or each possible implementation manner.
According to the data processing method, the data processing equipment and the storage medium, the big data development platform generates the script of the data processing task by running the external first development component, and then the data processing task can be completed by running the script of the data processing task. The convenience of big data development and the efficiency of data processing are improved.
Drawings
Fig. 1 is a schematic structural diagram of a big data processing platform 100 according to an embodiment of the present disclosure;
fig. 2 is a schematic diagram of a data processing method 200 according to an embodiment of the present application;
FIG. 3a is a schematic diagram of a data processing task development page provided by an embodiment of the present application;
FIG. 3b is a schematic diagram of another data processing task development page provided by an embodiment of the present application;
FIG. 4 shows a schematic block diagram of an electronic device 400 according to an embodiment of the application;
fig. 5 is a schematic structural diagram of an electronic device 500 according to an embodiment of the present application.
Detailed Description
The technical solution in the present application will be described below with reference to the accompanying drawings.
Big data development platforms are often used to provide a development environment for executable scripts for data processing tasks. The big data development platform can be a complete program or a collection of programs (also referred to as development components), each of which can be used, for example, to develop a functional script. When a developer needs to develop a program for a data processing task, the developer can use a development component in the big data development platform to develop the program corresponding to the data processing task (i.e., the executable script is generated). However, in some development scenarios, the development component packaged in the big data development platform may not meet the development requirement of the data processing task, and an executable script corresponding to the data processing task needs to be developed together with an external development component, however, the external development component is often written based on a Structured Query Language (SQL) script because the writing script is different from the writing script of the development component in the big data development platform (for example, the development component in the big data development platform is written based on an Extract-Transform-Load (ETL) database mapping technology, and the external component is written based on the SQL script), so that the external development component cannot be compatible in the big data development platform, in this case, a developer needs to develop a data processing together with the external development component, and only the external development component can be independently run outside the big data development platform, and realizing link connection of scripts of the data processing task by a mode of sharing a database with a big data development platform. On one hand, developers need to perform big data development on a plurality of development platforms, which affects the convenience of big data development, and on the other hand, the developers need to respectively generate partial scripts of data processing tasks on the plurality of development platforms, and then respectively run the partial scripts of the data processing tasks to complete the data processing tasks, which results in low efficiency of data processing.
In view of the above technical problems, embodiments of the present application provide a data processing scheme. The external first development component is operated through the big data development platform to generate a script of the data processing task, and then the data processing task can be completed through the script of the data processing task. The convenience of big data development and the efficiency of data processing are improved.
Fig. 1 is a schematic structural diagram of a big data processing platform 100 according to an embodiment of the present disclosure. As shown in FIG. 1, the big data development platform 100 includes at least a development design service 110 and a scheduling engine 120. The development design service 110 is used to provide a service of development design for a user (e.g., a data developer), such as providing an optional set of development components, or completing development design in response to user operations, etc.; scheduler engine 120 is used to schedule execution and management of all data task related services (e.g., development component services).
In some embodiments, the big data development platform 100 also includes at least one development component service 140. It is to be appreciated that the development component service 140 can be employed to execute development components to obtain corresponding scripts. Wherein the development component services 140 include at least one first development component service (e.g., 141 and 142). The first development component services (e.g., 141 and 142) may be used to execute a first development component that is not part of the big data development platform 100, such as development components that may be packaged by other big data development platforms, or development components that a developer writes outside of the big data development platform 100.
Optionally, development component services 140 may also include at least one second development component service (e.g., 143). The second development component service 143 may be configured to execute a second development component belonging to the big data development platform 100, for example, the development component may be a development component packaged by the big data development platform 100, or a development component written by a developer on the big data development platform 100.
The big data development platform 100 may be deployed in an electronic device, which may be, for example, a terminal device, such as a Mobile Phone (Mobile Phone), a tablet computer (Pad), a computer, a Television (TV), a Virtual Reality (VR) terminal device, an Augmented Reality (AR) terminal device, a terminal device in industrial control (industrial control), a terminal device in unmanned driving (self driving), a terminal device in remote medical (remote medical), a terminal device in smart city (smart city), or a terminal device in smart home (smart home). The terminal equipment in this application embodiment can also be wearable equipment, and wearable equipment also can be called as wearing formula smart machine, is the general term of using wearing formula technique to carry out intelligent design, develop the equipment that can dress to daily wearing, like glasses, gloves, wrist-watch, dress and shoes etc.. A wearable device is a portable device that is worn directly on the body or integrated into the clothing or accessories of the user.
Or the electronic device may be a server, for example a cloud server.
Optionally, the big data development platform 100 may further include a shared storage 130, where the shared storage 130 is used for storing (subtask) scripts of data tasks generated after each development component service 140 executes a development component. Generally, the shared storage 130 can store and read data according to the schedule of the big data development platform 100. For example, scheduler engine 120 may, upon receiving an execution instruction for a data processing task, read a script for the data processing task from shared storage 130 and execute the script to complete the data processing task.
Alternatively, the shared storage 130 may be a storage system external to the big data development platform 100. Further, shared storage 130 may be deployed in a different electronic device than big data development platform 100, e.g., shared storage 130 may be deployed in a storage server.
Alternatively, shared storage 130 may be a distributed storage system, such as a distributed storage system implemented based on a Remote Dictionary service (redis).
The data processing method provided by the embodiment of the present application will be described below with reference to the accompanying drawings.
The execution subject of the embodiment of the present application may be a big data development platform, such as the big data development platform 100 shown in fig. 1, or an electronic device deployed with the big data development platform.
Fig. 2 is a schematic diagram of a data processing method 200 according to an embodiment of the present application. As shown in fig. 2, the method may include at least some of the following steps.
S210, acquiring n target development components, wherein the n target development components comprise m first development components, n is more than or equal to m and is more than 0, and the first development components do not belong to the big data development platform;
s220, generating a script of a data processing task by running n target development components;
and S230, completing the data processing task by running the script of the data processing task.
For the above S210, it should be noted that:
the n target development components are used for developing the data processing task, for example, the big data development platform can produce a script corresponding to the data processing task by running the n target development components, and then can complete the data processing task by running the script corresponding to the data processing task.
The data processing tasks are not limited in any way by the present application. For example, a data processing task may generally include at least one data processing manner of preprocessing, unloading, loading, checking, moving, processing, copying and the like. Generally, a data processing task for big data is usually a combination of the above listed data processing manners, and each included data processing manner can be understood as a sub-task of the data processing task.
The n targeted development components may include at least one first development component, and when n is greater than 1, the n targeted development components may further include a second development component. That is, the big data development platform may run only the first development component or both the first and second development components during one big data development process in S220. Of course, the embodiment of the present application does not exclude that the big data development platform only runs the second development component in one big data development process.
For example, the big data development platform acquires n target development components, which may include: the big data development platform acquires a development component set, the development component set comprises n target development components, the development component set is displayed, and then first operation of a user on the development components in the development component set is acquired to obtain the n target development components.
Alternatively, the first operation may be a selection operation of a development component in the development component set, and the selection operation may be, for example, a single-click operation, a double-click operation, a drag operation, and the like of a desired development component.
Generally, a development component set includes a plurality of development components, including, for example, at least one first development component and at least one second development component.
As shown in connection with FIG. 3a, in the data processing task development page 300, the big data development platform may display a set of development components in a development component interface 310. The user may perform a first operation on the desired development component in the development component interface 310, e.g., the user may drag the desired development component into the design development interface 320 in the middle of the development component interface 310. For example, as shown in fig. 3a, the development component interface 310 includes a first development component a, a first development component b, a second development component c, a second development component d, and so on, and the user may drag the first development component a into the design development interface 320 twice to obtain two development components named M1 and M2, respectively, and for example, the user may drag the first development component b into the design development interface 320 to obtain a development component named M3, and for example, the user may drag the second development component d into the design development interface 320 to obtain a development component named M4. M1, M2, M3 and M4 are n target development components.
The big data development platform can load an external first development component to obtain a development component set containing the first development component. For example, the big data development platform may obtain registration information of the first development component, and load the at least one first development component into the development component set according to the registration information of the at least one first development component.
Optionally, the registration information of the first development component includes: the property, name (including Chinese and/or English name), edit page address (such as Uniform Resource Locator (URL)), issue interface, execution result viewing page address (such as URL), icon address (such as URL), job category, operation type, trigger mode, start interface, start command of the first development component. For example, the registration information of the first development component may include the contents as shown in table 1 below, for example.
TABLE 1
For example, the big data development platform may display a development component registration page, which may include, for example, at least one registration information template, which may include at least one item as shown in table 1, at which the user completes the completion of the registration information of the first development component. Accordingly, the big data development platform receives the registration information of the first development component input by the user in the registration page of the development component.
In S220, the big data development platform may generate a script of the data processing task by running the n target development components. The script of the data processing task may include, for example, a subtask script corresponding to each target development component.
For example, the big data development platform can respectively generate n subtask scripts by running n target development components, and the n subtask scripts form a script packet to obtain a script of the data processing task.
In some embodiments, the big data development platform may run the n target development components after obtaining the user's selected operation on the publishing control.
With reference to fig. 3a, a user may perform operations such as single-click and double-click on the publishing control 330 in the data processing task development page 300, and have already implemented the selection of the publishing control 330, after the big data development platform obtains the operation performed by the user on the selection of the developing control 330, the big data development platform may call, through a scheduling engine (e.g., the scheduling engine 120 in fig. 1), development component services corresponding to n development components respectively according to a publishing interface of the development components, and run the n development components to obtain a script of the data processing task.
Before the above S220, the data processing method may further include an editing operation on the first development component and/or the second development component to enable the development component to obtain a script required for the data processing task by running. The following description takes editing the first development component as an example:
for example, the big data development platform may obtain a second operation of the user on the first development component, where the second operation may be, for example, a selected operation (for example, operations including a click operation and a double click operation) on the first development component, and then the big data development platform may call an edit page address of the first development component according to the second operation to open an edit page of the first development component.
Further, the big data development platform can receive task parameters input by a user in an editing page of the first development component.
Referring to FIG. 3b, the data processing task development page 300 includes: a development component interface 310, a design development interface 320, and an editing page 340. The development component interface 310 and the design development interface 320 are similar to those in fig. 3a, and are not described herein again.
The edit page 340 may include a plurality of editable task parameters of the development component, such as at least one of a task name, a file to be collected, whether to open a local storage, a file generation policy (rolling daily or monthly), a cache configuration, and a transport configuration.
Optionally, the editing page 340 may be a floating window displayed above the development component interface 310 and/or the design development interface 320.
Optionally, after the user edits the n development components, the release control 330 may be selected, and the big data development platform runs the n development components to generate the script of the data processing task according to the operation of the user selecting the release control 330.
Optionally, the big data development platform may call the edit page of the development component and the corresponding saving logic of the edit page in a frame (iframe) embedding manner.
Optionally, the big data development platform may package n scripts obtained by running n target development components to obtain a script packet of the data processing task, and store the script packet in a shared storage (e.g., shared storage 130 in fig. 1).
It should be understood that, at present, the operation types of each development component include an interface, a script, a command, and a dockerfile, and when the large data development platform issues the operation types, each development component is converted into an executed script command. The scheduling engine need only call this command to start the job. The inherent operational details of each development component need not be known.
In the embodiment of the application, the big data development platform generates the script of the data processing task by running the external first development component, and then the data processing task can be completed by running the script of the data processing task. The convenience of big data development and the efficiency of data processing are improved.
While method embodiments of the present application are described in detail above in conjunction with fig. 2 and 3, and apparatus embodiments of the present application are described in detail below in conjunction with fig. 4 and 5, it is to be understood that apparatus embodiments correspond to method embodiments and similar descriptions may be had with reference to method embodiments.
Fig. 4 shows a schematic block diagram of an electronic device 400 according to an embodiment of the application. As shown in fig. 4, the electronic device 400 includes:
an obtaining unit 410, configured to obtain n target development components, where the n target development components include m first development components, n is greater than or equal to m and is greater than 0, and the first development components do not belong to the big data development platform;
a processing unit 420 for generating a script of a data processing task by running the n target development components
The processing unit 420 is also configured to complete the data processing task by running a script of the data processing task.
In some embodiments, the obtaining unit 410 is specifically configured to: acquiring a development component set, wherein the development component set comprises the n target development components; displaying the set of development components; and acquiring a first operation of the user on the development components in the development component set to obtain the n target development components.
In some embodiments, the obtaining unit 410 is specifically configured to: acquiring registration information of the first development component; and loading the at least one first development component to the development component set according to the registration information of the at least one first development component.
In some embodiments, the obtaining unit 410 is specifically configured to: displaying a development component registration page; and receiving registration information of the first development component input by a user in the development component registration page.
In some embodiments, the registration information of the first development component includes: the first development component comprises at least one of attribute, name, editing page address, issuing interface, execution result viewing page address, icon address, job category, operation type, triggering mode, starting interface and starting command.
In some embodiments, the obtaining unit 410 is further configured to: acquiring a second operation of the user on the first development component; and calling the editing page address of the first development component according to the second operation, and opening the editing page of the first development component.
In some embodiments, the obtaining unit 410 is further configured to: and receiving the task parameters input by the user in the editing page of the first development component.
In some embodiments, the processing unit 420 is specifically configured to: respectively generating n subtask scripts by operating the n target development components; and forming the n subtask scripts into a script packet to obtain the script of the data processing task.
In some embodiments, the obtaining unit 410 is further configured to: acquiring the selection operation of a user on a release control; and running the n target development components according to the selected operation.
In some embodiments, the electronic device 400 may specifically implement corresponding processes in the methods of the embodiments of the present application, and for brevity, details are not described here again.
Fig. 5 is a schematic structural diagram of an electronic device 500 according to an embodiment of the present application. The electronic device 500 shown in fig. 5 includes a processor 510, and the processor 510 can call and run a computer program from a memory to implement the method in the embodiment of the present application.
In some embodiments, as shown in FIG. 5, the electronic device 500 may also include memory 520. From the memory 520, the processor 510 can call and run a computer program to implement the method in the embodiment of the present application.
The memory 520 may be a separate device from the processor 510, or may be integrated into the processor 510.
In some embodiments, the electronic device 500 may also include an input interface 530. The processor 510 may control the input interface 530 to communicate with other devices or chips, and in particular, may obtain information or data transmitted by other devices or chips.
In some embodiments, the electronic device 500 may also include an output interface 540. The processor 510 may control the output interface 540 to communicate with other devices or chips, and may particularly output information or data to the other devices or chips.
In some embodiments, the electronic device 500 may be applied to a sending end or a receiving end in this embodiment, and the electronic device 500 may implement a corresponding procedure implemented by the sending end or the receiving end in each method in this embodiment, which is not described herein again for brevity.
In some embodiments, the apparatuses mentioned in the embodiments of the present application may also be chips. For example, it may be a system-on-chip, a system-on-chip or a system-on-chip, etc.
It should be understood that the processor of the embodiments of the present application may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method embodiments may be performed by integrated logic circuits of hardware in a processor or instructions in the form of software. The Processor may be a general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, or discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in a memory, and a processor reads information in the memory and completes the steps of the method in combination with hardware of the processor.
It will be appreciated that the memory in the embodiments of the subject application can be either volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory. The non-volatile Memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable PROM (EEPROM), or a flash Memory. Volatile Memory can be Random Access Memory (RAM), which acts as external cache Memory. By way of example, but not limitation, many forms of RAM are available, such as Static random access memory (Static RAM, SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic random access memory (Synchronous DRAM, SDRAM), Double Data Rate Synchronous Dynamic random access memory (DDR SDRAM), Enhanced Synchronous SDRAM (ESDRAM), Synchronous link SDRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to comprise, without being limited to, these and any other suitable types of memory.
It should be understood that the above memories are exemplary but not limiting illustrations, for example, the memories in the embodiments of the present application may also be Static Random Access Memory (SRAM), dynamic random access memory (dynamic RAM, DRAM), Synchronous Dynamic Random Access Memory (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (enhanced SDRAM, ESDRAM), Synchronous Link DRAM (SLDRAM), Direct Rambus RAM (DR RAM), and the like. That is, the memory in the embodiments of the present application is intended to comprise, without being limited to, these and any other suitable types of memory.
The embodiment of the application also provides a computer readable storage medium for storing the computer program.
In some embodiments, the computer-readable storage medium may be applied to an electronic device (e.g., a receiving end or a transmitting end) in the embodiments of the present application, and the computer program enables a computer to execute corresponding processes implemented by the receiving end or the transmitting end in the methods in the embodiments of the present application, which are not described herein again for brevity.
Embodiments of the present application also provide a computer program product comprising computer program instructions.
In some embodiments, the computer program product may be applied to a receiving end or a transmitting end in the embodiments of the present application, and the computer program instructions enable a computer to execute corresponding processes implemented by the receiving end or the transmitting end in the methods in the embodiments of the present application, which are not described herein again for brevity.
The embodiment of the application also provides a computer program.
In some embodiments, the computer program may be applied to the receiving end or the transmitting end in the embodiments of the present application, and when the computer program runs on a computer, the computer executes a corresponding process implemented by the receiving end or the transmitting end in each method in the embodiments of the present application, which is not described herein again for brevity.
In some embodiments, the computer program may be applied to the receiving end or the transmitting end in the embodiments of the present application, and when the computer program runs on a computer, the computer executes a corresponding process implemented by the receiving end or the transmitting end in each method in the embodiments of the present application, which is not described herein again for brevity.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. With regard to such understanding, the technical solutions of the present application may be essentially implemented or contributed to by the prior art, or may be implemented in a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to perform all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (13)
1. A data processing method is applied to a big data development platform and comprises the following steps:
acquiring n target development components, wherein the n target development components comprise m first development components, n is more than or equal to m and is more than 0, and the first development components do not belong to the big data development platform;
generating a script of a data processing task by running n target development components;
and completing the data processing task by running the script of the data processing task.
2. The method of claim 1, wherein obtaining n target development components comprises:
obtaining a development component set, wherein the development component set comprises the n target development components;
displaying the set of development components;
and acquiring a first operation of a user on the development components in the development component set to obtain the n target development components.
3. The method of claim 2, wherein obtaining the set of development components comprises:
acquiring registration information of the first development component;
loading the at least one first development component to the set of development components according to the registration information of the at least one first development component.
4. The method of claim 3, wherein obtaining registration information for the first development component comprises:
displaying a development component registration page;
and receiving registration information of the first development component, which is input by a user in the development component registration page.
5. The method of claim 3 or 4, wherein the registration information of the first development component comprises: the first development component comprises at least one of attribute, name, editing page address, issuing interface, execution result viewing page address, icon address, job category, operation type, triggering mode, starting interface and starting command.
6. The method according to any one of claims 2 to 4, further comprising:
acquiring a second operation of the user on the first development component;
and calling the editing page address of the first development assembly according to the second operation, and opening the editing page of the first development assembly.
7. The method of claim 6, further comprising:
and receiving task parameters input by a user in an editing page of the first development component.
8. The method of any of claims 1 to 4, wherein generating a script of a data processing task by running n target development components comprises:
respectively generating n subtask scripts by operating the n target development components;
and forming a script packet by the n subtask scripts to obtain the script of the data processing task.
9. The method according to any one of claims 1 to 4, further comprising:
acquiring the selection operation of a user on a release control;
and running the n target development components according to the selected operation.
10. An electronic device, wherein a big data development platform is deployed in the electronic device, and the electronic device comprises:
the system comprises an acquisition unit, a storage unit and a processing unit, wherein the acquisition unit is used for acquiring n target development components, the n target development components comprise m first development components, n is more than or equal to m and more than 0, and the first development components do not belong to a big data development platform;
the processing unit is used for generating a script of a data processing task by running the n target development components;
the processing unit is also used for completing the data processing task by running the script of the data processing task.
11. An electronic device, characterized by a processor and a memory for storing a computer program, the processor being adapted to invoke and execute the computer program stored in the memory, performing the method according to any of claims 1 to 9.
12. A computer-readable storage medium for storing computer program instructions, the computer program causing a computer to perform the method of any one of claims 1 to 9.
13. A computer program product comprising computer program instructions for causing a computer to perform the method of any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111553718.XA CN114217885A (en) | 2021-12-17 | 2021-12-17 | Data processing method, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111553718.XA CN114217885A (en) | 2021-12-17 | 2021-12-17 | Data processing method, device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114217885A true CN114217885A (en) | 2022-03-22 |
Family
ID=80703846
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111553718.XA Pending CN114217885A (en) | 2021-12-17 | 2021-12-17 | Data processing method, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114217885A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111324332A (en) * | 2018-12-17 | 2020-06-23 | 北京京东尚科信息技术有限公司 | Big data task processing method and system, electronic device and storage medium |
| CN112988298A (en) * | 2021-03-12 | 2021-06-18 | 土巴兔集团股份有限公司 | Visual interface configuration method and device, computer equipment and storage medium |
| CN113568614A (en) * | 2021-08-04 | 2021-10-29 | 掌阅科技股份有限公司 | Task issuing method, electronic device and storage medium |
| CN113760262A (en) * | 2021-01-21 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Task processing method, device, computer system and computer readable storage medium |
-
2021
- 2021-12-17 CN CN202111553718.XA patent/CN114217885A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111324332A (en) * | 2018-12-17 | 2020-06-23 | 北京京东尚科信息技术有限公司 | Big data task processing method and system, electronic device and storage medium |
| CN113760262A (en) * | 2021-01-21 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Task processing method, device, computer system and computer readable storage medium |
| CN112988298A (en) * | 2021-03-12 | 2021-06-18 | 土巴兔集团股份有限公司 | Visual interface configuration method and device, computer equipment and storage medium |
| CN113568614A (en) * | 2021-08-04 | 2021-10-29 | 掌阅科技股份有限公司 | Task issuing method, electronic device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109542399B (en) | Software development method and device, terminal equipment and computer readable storage medium | |
| US11816492B2 (en) | Third party integration of plugins and widgets | |
| CN107644286B (en) | Workflow processing method and device | |
| US10564988B1 (en) | Deploying cross-platform applications on mobile devices with native and web components | |
| CN105718313A (en) | Application operation method and device | |
| CN112433727A (en) | User interface configuration method and device, electronic equipment and readable storage medium | |
| CN111459541B (en) | Application program packaging method and device, computer equipment and storage medium | |
| CN106250104A (en) | A kind of remote operating system for server, method and device | |
| CN103853535A (en) | Method and device for modifying middleware | |
| CN111061475A (en) | Software code generation method and device, computer equipment and storage medium | |
| CN109445845A (en) | Interface call method, device, computer equipment and storage medium | |
| CN111739136B (en) | Rendering method, computer device, and storage medium | |
| CN116541336B (en) | Software running method of multi-core chip and coprocessor | |
| CN112486807A (en) | Pressure testing method and device, electronic equipment and readable storage medium | |
| CN116226921A (en) | Desensitization processing method and device | |
| CN114217885A (en) | Data processing method, device and storage medium | |
| CN115964331A (en) | Data access method, device and equipment | |
| CN115237467A (en) | Page view configuration method and device, computer equipment and storage medium | |
| CN107092601B (en) | Resource file construction method, resource file application method and device | |
| US11726818B1 (en) | System for executing tasks in different programming languages | |
| CN112214213A (en) | Linux kernel development and management method and device, computer equipment and storage medium | |
| CN114070892A (en) | Data transmission method and device | |
| CN113885996A (en) | User interface generation method and device | |
| CN115604331A (en) | Data processing system, method and device | |
| CN113656606B (en) | Code query method, device and equipment of application program and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |