CN116991607B - Adaptation method, system and equipment for communication between big data middle station and platform - Google Patents
Adaptation method, system and equipment for communication between big data middle station and platform Download PDFInfo
- Publication number
- CN116991607B CN116991607B CN202311244494.3A CN202311244494A CN116991607B CN 116991607 B CN116991607 B CN 116991607B CN 202311244494 A CN202311244494 A CN 202311244494A CN 116991607 B CN116991607 B CN 116991607B
- Authority
- CN
- China
- Prior art keywords
- platform
- big data
- task
- data
- uploading
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000006978 adaptation Effects 0.000 title claims abstract description 35
- 238000000034 method Methods 0.000 title claims abstract description 29
- 238000004891 communication Methods 0.000 title claims abstract description 20
- 230000001419 dependent effect Effects 0.000 claims abstract description 13
- 238000013515 script Methods 0.000 claims description 17
- 238000004590 computer program Methods 0.000 claims description 8
- 230000015654 memory Effects 0.000 claims description 8
- 230000009471 action Effects 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 6
- 230000003993 interaction Effects 0.000 claims description 6
- 238000007405 data analysis Methods 0.000 abstract description 2
- 210000001503 joint Anatomy 0.000 abstract description 2
- 238000007726 management method Methods 0.000 description 18
- 239000008186 active pharmaceutical agent Substances 0.000 description 5
- 238000010276 construction Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003032 molecular docking Methods 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/545—Interprogram communication where tasks reside in different layers, e.g. user- and kernel-space
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of big data analysis, in particular to an adaptation method, a system and equipment for communication between a big data center platform and a platform, wherein the method comprises the following steps: uploading a program: classifying and managing the existing program files and dependent files of the big data center, and uploading the classified and managed files to a big data platform in batches; library table establishment: determining a library table required by a large data middle platform, applying resources for the corresponding library table on a data bus, and establishing a single Zhang Kubiao or batch library table in a large data platform; task uploading: uploading real-time tasks and/or offline tasks required by a big data center to the big data platform in batches; task execution: and adapting the asynchronous task required to be executed by the big data middle platform to an interface provided by the big data platform, and driving the big data platform to execute the corresponding task. The invention can realize rapid adaptation and automatic butt joint between the big data center platform and the big data platform.
Description
Technical Field
The invention relates to the technical field of big data analysis, in particular to an adaptation method, a system and equipment for communication between a big data center station and a platform.
Background
At present, in big data processing, a big data platform and a data center platform running on the big data platform generally have respective characteristics and function positioning, wherein the big data platform mainly provides various basic big data processing components for running big data processing tasks, and the big data center platform provides data processing flow procedures per se, corresponding information of processing library tables, scheduling information of processing tasks and the like. For the traditional hadoop-based big data platform, the deployment can be completed only by copying the modules, the submitted tasks and the library table information to a certain client node in the big data platform, but for the big data platform with stronger security and management function, the deployment of the middle platform becomes very complex due to the fact that the platform cannot directly contact the client node environment of the big data platform.
Therefore, how to implement adaptation and automatic docking of a big data center platform and a big data platform is a problem that needs to be solved by those skilled in the art.
Disclosure of Invention
In view of the above, the invention provides an adaptation method, a system and equipment for communication between a big data center and a platform, which can realize rapid adaptation and automatic butt joint between the big data center and the big data platform.
In a first aspect, the present invention provides an adaptation method for communication between a large data center and a platform, comprising:
uploading a program: classifying and managing the existing program files and dependent files of the big data center, and uploading the classified and managed files to a big data platform in batches;
library table establishment: determining a library table required by a large data middle platform, applying resources for the corresponding library table on a data bus, and establishing a single Zhang Kubiao or batch library table in a large data platform;
task uploading: uploading real-time tasks and/or offline tasks required by a big data center to the big data platform in batches;
task execution: and adapting the asynchronous task required to be executed by the big data middle platform to an interface provided by the big data platform, and driving the big data platform to execute the corresponding task.
Further, in the program uploading step, the existing program files and the dependent files of the big data middle platform are uploaded to the designated interface of the big data platform in batches according to the specific directory structure.
Further, in the step of creating the library table, metadata required for the whole flow of data generation is managed together with metadata required to be provided in the data bus, and the metadata is used for representing a data structure; and according to a series of table building sentences required by the large data center table, the corresponding interface provided by the large data platform is called to realize a series of table building operations.
Further, when metadata required by the whole flow of data generation is managed, the structure information of the metadata has corresponding relations but different forms in the offline data and the online data; in the offline data, the structure information of the metadata is the structure information of a stored library table; in the online data, the structure information of the metadata is field information required for serialization and deserialization.
When the metadata required to be provided in the data bus is managed, different data are distinguished by themes in the data bus, so that the data in the data bus correspond to the structure information of the data in the database table, and the data generated in real time are stored in the corresponding database table to be offline data.
Further, in the task uploading step, starting with the existing task submitting script of the big data center, uploading the execution information of the task to a corresponding interface provided by the big data platform; when uploading tasks, the method comprises the following steps: code management, configuration and parameter management;
the code management is as follows: when executing each real-time task or offline task, acquiring the acquisition position of the program codes required by the current task;
the configuration and parameter management is as follows: before each task needs to be started, the action sequence and standard of interaction with the big data platform under the current task are known.
Further, in the task execution step, if the current computing task needs to be executed through a service driver, the computing task is driven through an API interface provided by the big data platform, where the API interface includes parameters required by the program during running.
In a second aspect, the present invention provides an adaptation system for communication between a large data center and a platform, comprising:
the program adaptation interface module is used for carrying out classified management on the existing program files and the dependent files of the big data middle platform and uploading the classified management to the big data platform in batches;
the library table adaptation interface module is used for determining a library table required by a large data middle platform, applying resources for the corresponding library table on a data bus, and establishing a single Zhang Kubiao or batch library table in the large data platform;
the computing task adaptation interface module is used for uploading real-time tasks and/or offline tasks required by the big data middle platform to the big data platform in batches;
and the task execution module is used for adapting the asynchronous task required to be executed by the big data middle platform and the interface provided by the big data platform and driving the big data platform to execute the corresponding task.
In a third aspect, the invention provides a computer device comprising a memory and a processor, the memory having stored thereon a computer program executable on the processor; the processor, when running the computer program, performs the steps of the adaptation method described above for communication between a big data center and a platform.
Compared with the prior art, the invention has the following beneficial effects:
according to the invention, on one hand, all program modules owned by the middle platform of the current big data are mastered, library table information to be established and information of calculation tasks to be submitted are acquired, and on the other hand, the big data platform is docked, and the middle platform can be deployed automatically by adapting to various interfaces of the big data platform, so that the workload of middle platform deployment can be reduced.
According to the invention, the large data platform is matched, only a plurality of standard middle platform deployment interfaces are required to be provided, so that the large data middle platform can be automatically deployed, meanwhile, the characteristics of safety, controllability and the like of the platform can be maintained, and for the platform needing to be approved, all links such as all library tables and calculation tasks of the middle platform can be uploaded to the platform for approval at one time through the interface of the invention, and all the deployments of the middle platform can be completed after the approval is passed once.
The invention can complete the description of all the services of the middle platform by defining the program modules needing to be uploaded, library table information needing to be generated, submitting scripts of calculation tasks needing to be executed and the like. For a large data platform, the large data platform can be provided with a large-scale processing module capable of batch uploading, a large number of intermediate library tables are established, and a large number of calculation tasks are saved, so that the large data platform has the capability of rapidly deploying the intermediate platform as a whole, and the capability does not require the large data platform to reduce the safety, the supervision and other standards of the large data platform.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of an adaptation method for communication between a large data center and a platform provided by the invention;
fig. 2 is a schematic structural diagram of an adaptation system for communication between a large data center and a platform according to the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
As shown in fig. 1, an embodiment of the present invention discloses an adaptation method for communication between a station and a platform in big data, including:
uploading a program: classifying and managing the existing program files and dependent files of the big data center, and uploading the classified and managed files to a big data platform in batches;
library table establishment: determining a library table required by a large data middle platform, applying resources for the corresponding library table on a data bus, and establishing a single Zhang Kubiao or batch library table in a large data platform;
task uploading: uploading real-time tasks and/or offline tasks required by a big data center to the big data platform in batches;
task execution: and adapting the asynchronous task required to be executed by the big data middle platform to an interface provided by the big data platform, and driving the big data platform to execute the corresponding task.
The steps described above are further described below.
(1) Program upload
The first step of the large data platform flow is usually to upload program files, if there are many modules such as program files, dependent files and the like required by the middle platform, the large data platform needs to provide other interfaces capable of uploading files in batches besides the conventional uploading file interfaces, for example:
the deployment files, including specifically the program jar package, shell script, python script, configuration file, code table data file, etc., are transmitted to the designated server or platform through the http interface or ssh port.
The effect of uploading a plurality of files in batches can be achieved by uploading the compressed package and then requesting the server to decompress.
Specifically, in the program uploading step, the method needs to adapt to the requirement of the big data platform, and the existing program files and the dependent files of the big data middle platform are uploaded to the designated interface of the big data platform in batches according to the specific directory structure.
Program upload management essentially manages a virtual folder under which all program modules and third party dependency packages required for computing tasks are stored in a tree structure. However, not all big data platforms allow the direct copying of the catalogue to the deployment node, and typically the big data platform would require the user to upload one program module (package and its dependent package) at a time, but the operation must be performed manually, and for a mature middle platform, uploading program code becomes a very workload consuming task, and if the big data platform provides interfaces, the middle platform may call these interfaces in batches to implement the function of uploading files in folders to such big data platforms with security restrictions, thereby ensuring security and meeting efficiency.
(2) Library table creation
Creating library tables in a large data platform may be performed in a manner that provides an interface, but this manner requires each field to be manually filled in, and is therefore inefficient. The middle platform usually prepares a series of table construction sentences of the library table required by the middle platform, so that when the large data platform builds the table, the middle platform calls the corresponding interface provided by the large data platform according to the series of table construction sentences required by the middle platform in advance to realize a series of table construction operations.
First, the platform can support input SQL script files and then build tables in batches by way of direct execution. In another mode, the operation of the graphical interface is interfaced, so that the middle platform can complete the list building action by calling the interface. The interface can only support to build a single table, but the middle platform can convert the batch information conversion mode of the self-stored table building sentences into a plurality of calls to the interface, so that the table building operation can be completed in a short time.
Meanwhile, when the library table is built, metadata required by the whole flow of data generation and metadata required to be provided in a data bus are managed together, and meanwhile, a large data platform is adapted, so that metadata related to the library table can be built on the large data platform in batches; the metadata is used to represent a data structure.
When metadata required by the whole flow of data generation is managed, the structure information of the metadata has corresponding relations but different forms in the offline data and the online data; in the offline data, the structure information of the metadata is the structure information of a stored library table; in online data, the structure information of metadata is field information required for serialization and deserialization, and can be essentially regarded as a table of a database, but the data in the table is generated in real time, so that the real-time and offline data can be managed simultaneously based on database table management.
When metadata required to be provided in a data bus is managed, different data are distinguished by themes in the data bus, so that the data in the data bus correspond to the structural information of the data in a database table, and the data generated in real time are stored in the corresponding database table to be offline data.
The data bus is used for real-time data production and consumption, so that large batches of data can be processed in real time. The data buses are subject-matter-based to distinguish between different data, which essentially correspond to a database table. The meta information management mechanism provided by the middle platform can straighten the corresponding relation between the data in the data bus and the structural information of the data in the database table, so that the data generated in real time can be saved in the database table through the persistence operation to be changed into offline data for subsequent offline analysis.
(3) Task upload
In a conventional supporting mode of a big data platform, a task submitting script can drive a flink or spark program by being directly executed. In the interface supporting mode, the user usually needs to convert the scripts into detail parameters so that the platform can complete supporting, for example, how many CPUs, how much memories are required to be applied by running tasks, how many nodes the tasks are started in a distributed mode, service parameters of program running and the like. The information which is originally intensively written in the running script needs to be respectively filled in the platform interfaces, so that the efficiency is reduced.
In order to improve the working efficiency of the part, a platform needs to provide uniform spark and flink task submitting interfaces, and a plurality of task information can be submitted through the interfaces at one time, and the fact that only the execution information of the computing task is submitted in the step is noted, and the computing task cannot be directly started. The middle platform can convert the submitting scripts of all the services into general execution information, including: task names, program modules and dependency module lists (jar package is used as a main component), applied resources (memory, CPU occupation, task node number and the like), configuration of a computing frame level (such as a flink or spark configuration parameter), service configuration parameters and the like. And then, calling a platform interface by using the information to finish the registration of the computing task and obtain the registration name of the computing task on the big data platform.
Some platforms may not allow for direct use of methods of executing a commit script to commit computing tasks, but instead require manual entry of information such as program module location, program configuration information, program resource requirements, etc., and then the platform restarts the computing program, which, while all of this information is already contained in the commit script, like program modules, a mature middle platform would contain very many computing tasks, and would be very labor intensive if each computing task were to be manually entered to accommodate the security requirements of the platform. If the platform can pack the process of submitting tasks into APIs, the possibility of automatic operation exists, the middle platform needs to adapt to the APIs, and then the computing tasks are submitted to a large data platform in batches, so that the efficiency is improved under the condition of meeting the safety requirement.
Therefore, in the task uploading process, the execution information of the task is uploaded to a corresponding interface provided by a big data platform by starting from the existing task submitting script of the big data center; by adapting to the interface provided by the big data platform, the computing tasks can be submitted on the big data platform, and the computing task information can be transmitted on the big data platform in batches.
Meanwhile, when uploading tasks, the invention needs to manage real-time tasks and/or offline tasks required by the intermediate platform, and the large data platform needs to meet the task uploading and management requirements of supporting batch.
The management of tasks includes: code management, configuration and parameter management;
the code management is as follows: when executing each real-time task or offline task, acquiring the acquisition position of the program codes required by the current task; i.e. where the program code needed to perform each real-time task or offline task is obtained.
The configuration and parameter management is as follows: before each task needs to be started, the action sequence and standard of interaction with the big data platform under the current task are known. I.e. knows what the standard sequence of actions and steps each task needs to interact with the big data platform if it needs to be started.
(4) Batch approval
If the big data platform has an approval mechanism, after the platform submits the content to the platform in batches, concentrated approval is carried out, and after the approval is passed, the batch effect can be achieved.
(5) Task execution
The step can carry out interface adaptation on asynchronous tasks to be executed by the middle platform, and drive the big data platform to execute corresponding tasks, and the big data platform needs to provide corresponding adaptation interfaces.
Usually, submitting a computing task on a big data platform needs to perform some interactions with the big data platform, writing the interaction processes into a script, and executing the script can be completely started. For conventional real-time tasks and timing tasks, the tasks can be started directly by executing relevant steps after the intermediate platform is deployed, but for asynchronous tasks, the asynchronous tasks are driven in terms of service, and the upper-layer application of the driving service is likely to be on other nodes, so that the intermediate platform needs to provide a standard interface service, usually an API interface in the form of Restful, and the intermediate platform starts a script to submit a computing task whenever the interface is called. The interface needs to contain parameters required by the program when running (the computing configuration is already done during the import of the task footsteps), which may be different for each execution and thus require dynamic delivery.
Through the five parts, each detail in the middle platform deployment process can be clearly defined, and an adaptation interface is provided for a big data platform. On the premise that the big data platform meets the interface, the deployment of the middle platform can be automatically carried out, and the big data platform does not need to reduce the security inspection standard.
In another embodiment, as shown in fig. 2, the present invention further provides an adaptation system for communication between a station and a platform in big data, comprising:
the program adaptation interface module is used for carrying out classified management on the existing program files and the dependent files of the big data middle platform and uploading the classified management to the big data platform in batches;
the library table adaptation interface module is used for determining a library table required by a large data middle platform, applying resources for the corresponding library table on a data bus, and establishing a single Zhang Kubiao or batch library table in the large data platform;
the computing task adaptation interface module is used for uploading real-time tasks and/or offline tasks required by the big data middle platform to the big data platform in batches;
and the task execution module is used for adapting the asynchronous task required to be executed by the big data middle platform and the interface provided by the big data platform and driving the big data platform to execute the corresponding task.
The program uploading between the middle platform and the big data platform can be realized through the program adapting interface module, the library table adapting interface module and the computing task adapting interface module, and the library table construction and the task uploading are automatically butted, so that the middle platform is very convenient to deploy.
In other embodiments, the invention also provides a computer device comprising a memory and a processor, the memory having stored thereon a computer program capable of running on the processor; the processor executes the steps of the adaptation method for communication between a big data center and a platform described above when running the computer program.
Meanwhile, the invention also provides a computer readable storage medium, on which a computer program is stored, which when executed, implements the steps of the adaptation method for communication between a big data center and a platform.
In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (6)
1. An adaptation method for communication between a large data center and a platform, comprising:
uploading a program: classifying and managing the existing program files and dependent files of the big data center, and uploading the classified and managed files to a big data platform in batches; uploading the existing program files and the dependent files of the big data middle platform to a designated interface of a big data platform in batches according to a specific directory structure;
library table establishment: determining a library table required by a large data middle platform, applying resources for the corresponding library table on a data bus, and establishing a single Zhang Kubiao or batch library table in a large data platform; metadata required by the whole flow of data generation is managed together with metadata required to be provided in a data bus, wherein the metadata is used for representing a data structure; according to a series of table building sentences required by a large data center table in advance, a corresponding interface provided by a large data platform is called to realize a series of table building operations;
task uploading: uploading real-time tasks and/or offline tasks required by a big data center to the big data platform in batches; starting with the existing task submitting script of the big data center, uploading the execution information of the task to a corresponding interface provided by a big data platform; when uploading tasks, the method comprises the following steps: code management, configuration and parameter management;
the code management is as follows: when executing each real-time task or offline task, acquiring the acquisition position of the program codes required by the current task;
the configuration and parameter management is as follows: before each task needs to be started, acquiring the action sequence and standard of interaction with a big data platform under the current task;
task execution: adapting an asynchronous task to be executed by a big data middle platform to an interface provided by a big data platform, and driving the big data platform to execute a corresponding task; if the current computing task needs to be executed through a service driver, the computing task is driven through an API interface provided by the big data platform, and the API interface contains parameters required by the running of the program.
2. The adaptation method for communication between a big data center and a platform according to claim 1, wherein when metadata required for a full flow of data generation is managed, structural information of the metadata has a correspondence but a different form in offline data and online data; in the offline data, the structure information of the metadata is the structure information of a stored library table; in the online data, the structure information of the metadata is field information required for serialization and deserialization.
3. The adaptation method for communication between a large data center and a platform according to claim 1, wherein when metadata required to be provided in a data bus is managed, different data are distinguished by subjects in the data bus, so that the data in the data bus and the structure information of the data in a database table correspond, and the data generated in real time are saved in the corresponding database table to be offline data.
4. An adaptation system for communication between a station and a platform in big data, comprising:
the program adaptation interface module is used for carrying out classified management on the existing program files and the dependent files of the big data middle platform and uploading the classified management to the big data platform in batches; uploading the existing program files and the dependent files of the big data middle platform to a designated interface of a big data platform in batches according to a specific directory structure;
the library table adaptation interface module is used for determining a library table required by a large data middle platform, applying resources for the corresponding library table on a data bus, and establishing a single Zhang Kubiao or batch library table in the large data platform; metadata required by the whole flow of data generation is managed together with metadata required to be provided in a data bus, wherein the metadata is used for representing a data structure; according to a series of table building sentences required by a large data center table in advance, a corresponding interface provided by a large data platform is called to realize a series of table building operations;
the computing task adaptation interface module is used for uploading real-time tasks and/or offline tasks required by the big data middle platform to the big data platform in batches; starting with the existing task submitting script of the big data center, uploading the execution information of the task to a corresponding interface provided by a big data platform; when uploading tasks, the method comprises the following steps: code management, configuration and parameter management;
the code management is as follows: when executing each real-time task or offline task, acquiring the acquisition position of the program codes required by the current task;
the configuration and parameter management is as follows: before each task needs to be started, acquiring the action sequence and standard of interaction with a big data platform under the current task;
and the task execution module is used for adapting the asynchronous task required to be executed by the big data middle platform and an interface provided by the big data platform, driving the big data platform to execute the corresponding task, and driving the calculation task through an API interface provided by the big data platform if the current calculation task is required to be executed through service driving, wherein the API interface comprises parameters required by program operation.
5. A computer device comprising a memory and a processor, the memory having stored thereon a computer program executable on the processor; the processor, when running the computer program, performs the steps of the adaptation method for communication between a big data center and a platform according to any of claims 1-3.
6. A computer readable storage medium, having stored thereon a computer program which, when executed, implements the steps of the adaptation method for communication between a large data center and a platform according to any of claims 1-3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311244494.3A CN116991607B (en) | 2023-09-26 | 2023-09-26 | Adaptation method, system and equipment for communication between big data middle station and platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311244494.3A CN116991607B (en) | 2023-09-26 | 2023-09-26 | Adaptation method, system and equipment for communication between big data middle station and platform |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116991607A CN116991607A (en) | 2023-11-03 |
| CN116991607B true CN116991607B (en) | 2023-12-22 |
Family
ID=88525133
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311244494.3A Active CN116991607B (en) | 2023-09-26 | 2023-09-26 | Adaptation method, system and equipment for communication between big data middle station and platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116991607B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112396404A (en) * | 2020-11-27 | 2021-02-23 | 广州光点信息科技有限公司 | Data center system |
| CN113129063A (en) * | 2021-04-20 | 2021-07-16 | 国网江西省电力有限公司信息通信分公司 | Electricity charge calculation and distribution method and system based on cloud platform and data center |
| CN113934408A (en) * | 2021-09-17 | 2022-01-14 | 华润置地控股有限公司 | Data service platform and method for generating application program interface |
| CN114266235A (en) * | 2021-12-10 | 2022-04-01 | 青岛海尔科技有限公司 | Data processing method and device, storage medium and electronic device |
| CN115080533A (en) * | 2021-03-12 | 2022-09-20 | 福州慧美丰科技有限公司 | Middle platform system for data exchange and sharing based on big data |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10235439B2 (en) * | 2010-07-09 | 2019-03-19 | State Street Corporation | Systems and methods for data warehousing in private cloud environment |
| US20230114277A1 (en) * | 2021-09-29 | 2023-04-13 | Gaurav Deshmukh | System and method for operations management |
-
2023
- 2023-09-26 CN CN202311244494.3A patent/CN116991607B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112396404A (en) * | 2020-11-27 | 2021-02-23 | 广州光点信息科技有限公司 | Data center system |
| CN115080533A (en) * | 2021-03-12 | 2022-09-20 | 福州慧美丰科技有限公司 | Middle platform system for data exchange and sharing based on big data |
| CN113129063A (en) * | 2021-04-20 | 2021-07-16 | 国网江西省电力有限公司信息通信分公司 | Electricity charge calculation and distribution method and system based on cloud platform and data center |
| CN113934408A (en) * | 2021-09-17 | 2022-01-14 | 华润置地控股有限公司 | Data service platform and method for generating application program interface |
| CN114266235A (en) * | 2021-12-10 | 2022-04-01 | 青岛海尔科技有限公司 | Data processing method and device, storage medium and electronic device |
Non-Patent Citations (1)
| Title |
|---|
| 基于数据中台的无线传感器网络数据共享***;宋宇翔 等;计算机工程与设计;第第43卷卷(第第11期期);3023-3028页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116991607A (en) | 2023-11-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR102493449B1 (en) | Edge computing test methods, devices, electronic devices and computer-readable media | |
| US20090077090A1 (en) | Method and apparatus for specifying an order for changing an operational state of software application components | |
| US8418074B2 (en) | Shared user interface services framework | |
| US8407713B2 (en) | Infrastructure of data summarization including light programs and helper steps | |
| CN111381940B (en) | Distributed data processing method and device | |
| CN112395736A (en) | Parallel simulation job scheduling method of distributed interactive simulation system | |
| CN113918637A (en) | BPMN2.0 specification-based process engine platform creation method and system | |
| CN115437808A (en) | Intercommunication method, device, equipment, medium and product between federal learning platforms | |
| CN112000734A (en) | Big data processing method and device | |
| CN117056240A (en) | Data element development and debugging method and system supporting offline Jar package | |
| CN101562622B (en) | Method for executing user request and corresponding server thereof | |
| CN116991607B (en) | Adaptation method, system and equipment for communication between big data middle station and platform | |
| CN117076096A (en) | Task flow execution method and device, computer readable medium and electronic equipment | |
| EP3611616A1 (en) | Software code optimizer and method | |
| Raj | A framework for migration of microservices based applications to serverless platform with efficient cold start latency | |
| CN113419814B (en) | Virtual machine creating method, device, equipment and storage medium in cloud platform | |
| CN114791826A (en) | Jenkins project operation method and device based on parameter configuration | |
| CN113641641A (en) | Switching method, switching system, equipment and storage medium of file storage service | |
| CN112905270A (en) | Workflow implementation method, device, platform, electronic equipment and storage medium | |
| CN115686483A (en) | Equipment manufacturing industry APP development framework and method | |
| CN117251251A (en) | Container starting sequence control method and system based on Kubernetes cloud primary cluster | |
| CN117667718A (en) | Automatic test method and system based on task scheduling | |
| CN116932147A (en) | Streaming job processing method and device, electronic equipment and medium | |
| CN116954814A (en) | Mirror image construction method and device based on Kubernetes | |
| CN115794295A (en) | Task processing method and device, system and computer readable storage medium thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |