CN109656682A - A kind of system and method for the exploration and development big data processing platform based on container technique - Google Patents
A kind of system and method for the exploration and development big data processing platform based on container technique Download PDFInfo
- Publication number
- CN109656682A CN109656682A CN201811467745.3A CN201811467745A CN109656682A CN 109656682 A CN109656682 A CN 109656682A CN 201811467745 A CN201811467745 A CN 201811467745A CN 109656682 A CN109656682 A CN 109656682A
- Authority
- CN
- China
- Prior art keywords
- big data
- module
- service
- exploration
- cluster
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45504—Abstract machines for programme code execution, e.g. Java virtual machine [JVM], interpreters, emulators
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45562—Creating, deleting, cloning virtual machine instances
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45591—Monitoring or debugging support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45595—Network integration; Enabling network access in virtual machine instances
Abstract
The system and method for the invention discloses a kind of exploration and development big data processing platform based on container technique, comprising: (1) presentation layer provides user items module, message center module, virtual desktop module, virtual server module, cloud storage module, resource management module, monitoring resource module, big data cluster module, big data model module, big data and submits module, big data display module etc.；(2) service layer includes that service management module, service request module, service approval module, service arrangement module mention, services and use module, service interface module etc.；(3) infrastructure layer includes authentication module, computing module, mirror module, block device module, network module, object storage module, virtual machine module, container module, resource management module, monitoring resource etc..The invention handles model template according to exploration and development software big data, and editor's setting big data handles model, to analyze the mass data of processing exploration and development software.
Description
Technical field
The present invention relates to a kind of exploration and development based on container technique of geological exploration and development technical field of data processing is big
The system and method for data processing platform (DPP).Specifically, being related to the exploration and development big data designed and developed with container technique
Processing platform, to create exploration and development big data analysis processing cluster；It is opened using big data analysis processing cluster to analyze exploration
Feel like jelly the service condition of part, realizes the fining management and running of professional software；The application demand of user is better met, is improved soft
The application efficiency of part.
Background technique
In exploration and development field, various exploration and development softwares are complicated using function, produce the processing data of magnanimity, such as
Approval software module uses the historical data of hot spot, the historical data of approval software module total evaluation, the history of exploratory area hot spot
It is soft that historical data that data, individual subscriber permissions module use, approval software module occupy the historical data of consumption resource, license
The historical data etc. of part module request refusal；The exploration and development software application behavioral data to these magnanimity is needed, big number is used
It is analyzed and processed according to platform, establishes the exploration and development software action analysis prediction model under cloud application, realize professional software
Refine management and running；The application demand for better meeting user improves the application efficiency of software；By to exploration and development software
Application characteristic, the analysis of user job habit, excavates hiding data value, provides preferably application experience for user.
Container technique is to install (SuSE) Linux OS on a physical server, creates multiple void by container technique
Quasi- server, these virtual servers and physical server share linux kernel；The file system of each virtual server uses
The file system of physical server, but be isolated, it appears that each virtual server has oneself independent file system；?
Virtual bridge equipment is established on physical server, each virtual server connects network by virtual bridge equipment；Virtual clothes
Business device directly uses the CPU, memory, hard disk of physical server.
Big data platform Hadoop is a distributed system infrastructure developed by apache foundation；User can
Without understanding the details of the distributed bottom layer, to develop distributed program；The power of cluster is made full use of to carry out high speed fortune
It calculates and stores；Hadoop big data processing technique has more than ten years history, is seen as the solution of preferred big data cluster processing
Scheme；MapReduce is the outstanding solution calculated all the way
Big data cluster Spark is a kind of quick, general cluster computing system, it provide high-level API, Java,
Scala, Python and R and a kind of optimization engine for supporting figure to handle, it also supports that one group of upgrade kit abundant includes
Spark SQL is handled for SQL query and structural data, and MLlib is used for machine learning, graphics process and Spark
Streaming。
But creation big data cluster, need to buy several servers or virtual machine to dispose, and need to match manually
Set big data cluster；A large amount of financial resources and manpower are spent to remove deployment configuration big data cluster；If there can be a kind of technology, do not need
The performance of big data cluster can also be made full use of by buying many servers；And automatically configure big data cluster.
Summary of the invention
The purpose of the present invention is to the deficiencies in the prior art, one kind is provided and does not need to buy many servers and can also fill
Point using big data cluster performance, and automatically configure at the exploration and development big data based on container technique of big data cluster
The system and method for platform.
Technical solution of the present invention includes:
A kind of exploration and development big data processing platform (abbreviation exploration and development big data processing platform) based on container technique is
System, system architecture include:
(1) presentation layer, including user items, message center, virtual desktop, virtual server, cloud storage, resource management, resource
Monitoring, the modeling of big data cluster, big data, big data is submitted and big data display module；
(2) service layer, including service management, service request, service examination & approval, service arrangement, service use and service interface module；
(3) infrastructure layer, including authentication, calculating, mirror image, block device, network block, object storage, virtual machine, resource pipe
Reason, container and monitoring system module.
Above scheme further comprises:
(1) each functional module of presentation layer:
User items module provides user management, project management；Message center module provides user and sends message, and system sends wide
Broadcast message；Virtual desktop module provides virtual desktop creation, use, access；Virtual server module provides virtual server wound
It builds, use, access；Cloud storage module provides cloud storage Dropbox；Resource management module provides network, cloud Host Type, mirror image, master
Machine polymerization；Monitoring resource module provide the cpu busy percentage of virtual machine and server, memory usage, storage and network monitoring；
Big data cluster module provides creation Hadoop, spark cluster, and user inputs big data number of clusters, machine name, IP address,
System meeting automatic deployment big data cluster simultaneously automatically configures；Big data modeling module provides the template of big data processing model,
Setting editor's big data handles model；Big data submits module to provide user and submits big data job request；Big data shows mould
Block, which is provided, shows big data analysis processing result with table and figure.
(2) each functional module of service layer: service layer by one or more different exploration and development software types virtual machine or
Container cluster example set includes whole virtual machines or entirely container at, service cluster example, or include simultaneously virtual machine and
Container；
Wherein: service management module is supplied to system manager's issuing service；Service request module is supplied to tenant administrator Shen
It please service；Service approval module is supplied to examination & approval administrator and examines service；Service arrangement module is supplied to system manager's deployment
Service cluster example；Service is supplied to user using module and uses service cluster example；Service interface module includes certification, meter
The interactive interface of calculation, network, mirror image, object, virtual machine and container module；
(3) each functional module of infrastructure layer:
Authentication module provides unified identity authentication function；Computing module provides creation service cluster example virtual machine or container；Mirror
As module provides the virtual machine image or container mirror image of service cluster；Block device module provides Service Instance virtual machine cloud hard disk；
The network of network module offer service cluster example；Object storage module provides mirror image, picture storage；Virtual machine module provides empty
Quasi- machine creation, is closed and is deleted at starting；Creation, starting, closing, the deletion of container module offer container；Resource management module mentions
For the modulation of quotas of tenant CPU, memory, storage resource；The real time monitoring of monitoring system module offer service cluster example.
A kind of system of the exploration and development big data processing platform based on above-mentioned based on container technique carries out at big data
The method of reason, big data Hadoop, the Spark container mirror image production method of the exploration and development big data processing platform include with
Lower step:
Step 1: installation containment system software, desktop software；
Step 2: installation configuration safety shell protocol (SSH), remote control software (VNC), webpage remote control software
(NOVNC) it services；
Step 3: installation configuration Java software；
Step 4: installation configuration big data Hadoop software；
Step 5: installation configuration big data Spark software；
Step 6: installation configuration big data Hive software；
Step 7: installation configuration big data HBase software；
Step 8: addition automatically configures big data Hadoop service routine；
Step 9: addition automatically configures big data Spark service routine；
Step 10: installation dispensing containers service starts script.
The method of creation big data Hadoop, the Spark cluster of the exploration and development big data processing platform includes following
Step:
Step 1: selection big data Hadoop, Spark container mirror image；
Step 2: input big data cluster module number；
Step 3: the machine name or starting machine name of input big data cluster；
Step 4: the IP address or start ip address of input big data cluster；
Step 5: the title of input big data cluster；
Step 6: inputting the server of big data clustered deploy(ment) or automatically selecting；
Step 7: submitting creation big data cluster request.
Method that the big data operation of the exploration and development big data processing platform is submitted the following steps are included:
Step 1: modifying the database configuration information in data source in processing routine template file；
Step 2: modifying the journal file path in data source, configuration information in processing routine template file；
Step 3: modifying alternative document folder, the configuration information in data source in processing routine template file；
Configuration information is modified Step 4: generating in annual historical data template file in processing routine；
Configuration information is modified Step 5: generating in monthly historical data template file in processing routine；
Configuration information is modified Step 6: generating in daily historical data template file in processing routine；
Step 7: submitting the request of big data processing；
Step 8: system is automatically performed the processing of big data operation；
Step 9: system is automatically performed the .json file that all result datas are generated to dictionary format after big data operation is handled.
The big data of the exploration and development big data processing platform show the method for processing the following steps are included:
Step 1: loading the json file generated when big data operation processing is completed in big data displayed page；
Step 2: loading data according to the time of user's selection, exploration and development software, module in big data displayed page；
Step 3: in big data displayed page Dynamically Announce data table；
Step 4: loading map in big data displayed page, and analysis data are shown according to information；
Step 5: showing analysis data according to the data of dynamically load in big data displayed page with histogram；
Step 6: showing analysis data according to the data of dynamically load in big data displayed page with cake chart；
Step 7: showing analysis data according to the data of dynamically load in big data displayed page with curve graph；
Step 8: dynamic configuration finally table to be shown, map, histogram, cake chart, curve in big data displayed page
Figure.
The present invention provides creation big data cluster for exploration and development big data processing platform user interface, exploration is submitted to open
Feel like jelly part Analysis model of network behaviors, using big data cluster, the clothes such as check big data analysis processing result (report, graphical display)
Business；Big data hadoop, the spark container mirror image that big data cluster is compiled by source code creates big data cluster container, defeated
Enter the parameters such as newly-built big data number of clusters, machine name, IP address, system automatically configures big data cluster；User can be direct
Model template is handled using big data cluster, and according to exploration and development software big data, editor's setting big data handles model, comes
The mass data of analysis processing exploration and development software.
Detailed description of the invention
Attached drawing 1 is a kind of service system architecture diagram of exploration and development big data processing platform based on container technique.
Specific embodiment
Implementation system of the invention is system and the side of a kind of exploration and development big data processing platform based on container technique
Method, system architecture design are as shown in Figure 1, comprising:
(1) exploration and development big data processing platform presentation layer provides user items module, message center module, virtual desktop mould
Block, virtual server module, cloud storage module, resource management module, monitoring resource module, big data cluster module, big data
Model module, big data submit module, big data display module etc.；
(2) exploration and development big data processing platform service layer includes service management module, service request module, service examination & approval mould
Block, service arrangement module are mentioned, are serviced using module, service interface module etc.；
(3) exploration and development big data processing platform service layer includes authentication module, computing module, mirror module, block device
Module, network module, object storage module, virtual machine module, container module, resource management module, monitoring resource etc..
The system of exploration and development big data processing platform based on container technique, each functional module of presentation layer:
(1) user items module provides the functions such as user management, project management；
(2) message center module provides user and sends message, and system sends the functions such as broadcast message；
(3) virtual desktop module provides the function of virtual desktop creation, use, access；
(4) virtual server module provides the function of virtual server creation, use, access；
(5) cloud storage module provides the function of cloud storage Dropbox；
(6) resource management module provides the functions such as network, cloud Host Type, mirror image, host polymerization；
(7) monitoring resource module provides the cpu busy percentage of virtual machine and server, memory usage, storage, the monitoring of network
Function；
(8) big data cluster module provide creation Hadoop, spark cluster function, user input big data number of clusters,
Machine name, IP address etc., system meeting automatic deployment big data cluster simultaneously automatically configure；
(9) big data model module provides the template of big data processing model, and editor's big data processing mould can be set in user
Type；
(10) big data submits module to provide the function that user submits big data job request；
(11) big data display module provides the function that big data analysis processing result is shown with table and figure.
The system of exploration and development big data processing platform based on container technique, each functional module of service layer:
(1) service is by the virtual machine or container cluster example set of one or more different exploration and development software types at services set
Group's example can be entirely virtual machine, can also be entirely container, can also simultaneously include virtual machine and container；
(2) service management module is supplied to system manager's issuing service；
(3) service request module is supplied to tenant administrator and applies servicing；
(4) service approval module is supplied to examination & approval administrator and examines service；
(5) service arrangement module is supplied to system manager's deployment services cluster instance；
(6) service is supplied to user using module and uses service cluster example；
(7) service interface module is encapsulated and is authenticated, calculates, the modules interaction such as network, mirror image, object, virtual machine, container connects
Mouthful；
The system of exploration and development big data processing platform based on container technique, each functional module of infrastructure layer:
(1) authentication module provides middle petrochemical industry unified identity authentication function；
(2) computing module provides the function of creation service cluster example virtual machine or container；
(3) mirror module provides the virtual machine image or container image feature of service cluster；
(4) block device module provides Service Instance virtual machine cloud hard disk function；
(5) network module provides the network function of service cluster example；
(6) object storage module provides the store functions such as mirror image, picture；
(7) virtual machine module provides the functions such as virtual machine creating, starting, closing, deletion；
(8) container module provides the functions such as creation, starting, closing, the deletion of container.
(9) resource management module provides the functions such as the modulation of quotas of the resources such as tenant CPU, memory, storage；
(10) monitoring resource module provides the monitoring function of service cluster example；
Exploration and development big data processing platform by a control node server, a network-node server, one or more
Calculate node server composition, exploration and development big data platform portal website, Database Systems, scheduling system, each service module
Etc. being deployed in control node server；Exploration and development big data processing platform network is deployed in network-node server；Exploration is opened
Hair big data processing platform virtual machine manager and container manager are deployed in calculate node server.
Exploration and development big data processing platform infrastructure services provide the function of creation virtual machine or container, and basis is set
The virtual machine or container instance for applying service, are mounted with basic operating system；Tenant administrator can apply for infrastructure services；
System manager can choose virtual machine or container instance in deployment base facilities services；It can also be allowed in issuing service
Tenant administrator selects to be virtual machine or container instance when applying for infrastructure services.
Exploration and development big data processing platform oil exploration service provides the cluster services function such as creation hadoop, spark
Can, the virtual machine or container instance of big data cluster service are mounted with basic operating system, exploration and development software, database
Deng；System manager, which issues big data cluster service, can specify virtual machine or container, and tenant administrator application can also be allowed big
It is specified when data cluster services, it can also be specified by system manager.
Exploration and development big data processing platform calculate node server is configured with virtual machine manager and Container Management simultaneously
Device, the calculate node can both support creation virtual machine instance, can also create container instance.
Exploration and development big data processing platform encapsulates service interface module, and portal website will increase new functional module,
It only needs to increase function in front end page, and increases function in service interface module, do not need the function mould of change bottom
Block.
Big data Hadoop, the Spark container mirror image production method of exploration and development big data processing platform:
Step 1: installation containment system software, desktop software；
Step 2: installation configuration safety shell protocol (SSH), remote control software (VNC), webpage remote control software
(NOVNC) it services；
Step 3: installation configuration Java software；
Step 4: installation configuration big data Hadoop software；
Step 5: installation configuration big data Spark software；
Step 6: installation configuration big data Hive software；
Step 7: installation configuration big data HBase software；
Step 8: addition automatically configures big data Hadoop service routine；
Step 9: addition automatically configures big data Spark service routine；
Step 10: installation dispensing containers service starts script.
The method of creation big data Hadoop, the Spark cluster of exploration and development big data processing platform:
Step 1: selection big data Hadoop, Spark container mirror image；
Step 2: input big data cluster module number；
Step 3: the machine name or starting machine name of input big data cluster；
Step 4: the IP address or start ip address of input big data cluster；
Step 5: the title of input big data cluster；
Step 6: inputting the server of big data clustered deploy(ment) or automatically selecting；
Step 7: submitting creation big data cluster request.
The method that the big data operation of exploration and development big data processing platform is submitted:
Step 1: modifying the database configuration information in data source in processing routine template file；
Step 2: modifying the journal file path in data source, configuration information in processing routine template file；
Step 3: modifying alternative document folder, the configuration information in data source in processing routine template file；
Configuration information is modified Step 4: generating in annual historical data template file in processing routine；
Configuration information is modified Step 5: generating in monthly historical data template file in processing routine；
Configuration information is modified Step 6: generating in daily historical data template file in processing routine；
Step 7: submitting the request of big data processing；
Step 8: system is automatically performed the processing of big data operation；
Step 9: system is automatically performed the json file that all result datas are generated to dictionary format after big data operation is handled.
The method that the big data of exploration and development big data processing platform shows processing:
Step 1: loading the .json file generated when big data operation processing is completed in big data displayed page；
Step 2: loading data according to time, exploration and development software, the module etc. of user's selection in big data displayed page；
Step 3: in big data displayed page Dynamically Announce data table；
Step 4: loading map in big data displayed page, and analysis data are shown according to information；
Step 5: showing analysis data according to the data of dynamically load in big data displayed page with histogram；
Step 6: showing analysis data according to the data of dynamically load in big data displayed page with cake chart；
Step 7: showing analysis data according to the data of dynamically load in big data displayed page with curve graph；
Step 8: dynamic configuration finally table to be shown, map, histogram, cake chart, curve in big data displayed page
Figure.
Claims (6)
1. a kind of system of the exploration and development big data processing platform based on container technique, which is characterized in that the system architecture
Include:
(1) presentation layer, including user items, message center, virtual desktop, virtual server, cloud storage, resource management, resource
Monitoring, the modeling of big data cluster, big data, big data is submitted and big data display module；
(2) service layer, including service management, service request, service examination & approval, service arrangement, service use and service interface module；
(3) infrastructure layer, including authentication, calculating, mirror image, block device, network block, object storage, virtual machine, resource pipe
Reason, container and monitoring system module.
2. the system of the exploration and development big data processing platform based on container technique as described in claim 1, which is characterized in that
(1) each functional module of presentation layer:
User items module provides user management, project management；Message center module provides user and sends message, and system sends wide
Broadcast message；Virtual desktop module provides virtual desktop creation, use, access；Virtual server module provides virtual server wound
It builds, use, access；Cloud storage module provides cloud storage Dropbox；Resource management module provides network, cloud Host Type, mirror image, master
Machine polymerization；Monitoring resource module provide the cpu busy percentage of virtual machine and server, memory usage, storage and network monitoring；
Big data cluster module provides creation Hadoop, spark cluster, and user inputs big data number of clusters, machine name, IP address,
System meeting automatic deployment big data cluster simultaneously automatically configures；Big data modeling module provides the template of big data processing model,
Setting editor's big data handles model；Big data submits module to provide user and submits big data job request；Big data shows mould
Block, which is provided, shows big data analysis processing result with table and figure；
(2) each functional module of service layer: service layer by one or more different exploration and development software types virtual machine or container
Cluster instance composition, service cluster example include whole virtual machines or entirely container, or simultaneously include virtual machine and container；
Wherein: service management module is supplied to system manager's issuing service；Service request module is supplied to tenant administrator Shen
It please service；Service approval module is supplied to examination & approval administrator and examines service；Service arrangement module is supplied to system manager's deployment
Service cluster example；Service is supplied to user using module and uses service cluster example；Service interface module includes certification, meter
The interactive interface of calculation, network, mirror image, object, virtual machine and container module；
(3) each functional module of infrastructure layer:
Authentication module provides unified identity authentication function；Computing module provides creation service cluster example virtual machine or container；Mirror
As module provides the virtual machine image or container mirror image of service cluster；Block device module provides Service Instance virtual machine cloud hard disk；
The network of network module offer service cluster example；Object storage module provides mirror image, picture storage；Virtual machine module provides empty
Quasi- machine creation, is closed and is deleted at starting；Creation, starting, closing, the deletion of container module offer container；Resource management module mentions
For the modulation of quotas of tenant CPU, memory, storage resource；The real time monitoring of monitoring system module offer service cluster example.
3. a kind of system of the exploration and development big data processing platform based on of any of claims 1 or 2 based on container technique into
The method of row big data processing, which is characterized in that big data Hadoop, Spark of the exploration and development big data processing platform
Container mirror image production method the following steps are included:
Step 1: installation containment system software, desktop software；
Step 2: installation configuration safety shell protocol (SSH), remote control software (VNC), webpage remote control software
(NOVNC) it services；
Step 3: installation configuration Java software；
Step 4: installation configuration big data Hadoop software；
Step 5: installation configuration big data Spark software；
Step 6: installation configuration big data Hive software；
Step 7: installation configuration big data HBase software；
Step 8: addition automatically configures big data Hadoop service routine；
Step 9: addition automatically configures big data Spark service routine；
Step 10: installation dispensing containers service starts script.
4. the exploration and development big data processing platform system according to claim 3 based on container technique carries out at big data
The method of reason, which is characterized in that creation big data Hadoop, the Spark cluster of the exploration and development big data processing platform
Method the following steps are included:
Step 1: selection big data Hadoop, Spark container mirror image；
Step 2: input big data cluster module number；
Step 3: the machine name or starting machine name of input big data cluster；
Step 4: the IP address or start ip address of input big data cluster；
Step 5: the title of input big data cluster；
Step 6: inputting the server of big data clustered deploy(ment) or automatically selecting；
Step 7: submitting creation big data cluster request.
5. the exploration and development big data processing platform system according to claim 3 based on container technique carries out at big data
The method of reason, which is characterized in that the method that the big data operation of the exploration and development big data processing platform is submitted includes following
Step:
Step 1: modifying the database configuration information in data source in processing routine template file；
Step 2: modifying the journal file path in data source, configuration information in processing routine template file；
Step 3: modifying alternative document folder, the configuration information in data source in processing routine template file；
Configuration information is modified Step 4: generating in annual historical data template file in processing routine；
Configuration information is modified Step 5: generating in monthly historical data template file in processing routine；
Configuration information is modified Step 6: generating in daily historical data template file in processing routine；
Step 7: submitting the request of big data processing；
Step 8: system is automatically performed the processing of big data operation；
Step 9: system is automatically performed the .json file that all result datas are generated to dictionary format after big data operation is handled.
6. the exploration and development big data processing platform system according to claim 3 based on container technique carries out at big data
The method of reason, which is characterized in that the method that the big data of the exploration and development big data processing platform shows processing includes following
Step:
Step 1: loading the json file generated when big data operation processing is completed in big data displayed page；
Step 2: loading data according to the time of user's selection, exploration and development software, module in big data displayed page；
Step 3: in big data displayed page Dynamically Announce data table；
Step 4: loading map in big data displayed page, and analysis data are shown according to information；
Step 5: showing analysis data according to the data of dynamically load in big data displayed page with histogram；
Step 6: showing analysis data according to the data of dynamically load in big data displayed page with cake chart；
Step 7: showing analysis data according to the data of dynamically load in big data displayed page with curve graph；
Step 8: dynamic configuration finally table to be shown, map, histogram, cake chart, curve in big data displayed page
Figure.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811467745.3A CN109656682A (en) | 2018-12-03 | 2018-12-03 | A kind of system and method for the exploration and development big data processing platform based on container technique |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811467745.3A CN109656682A (en) | 2018-12-03 | 2018-12-03 | A kind of system and method for the exploration and development big data processing platform based on container technique |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109656682A true CN109656682A (en) | 2019-04-19 |
Family
ID=66111208
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811467745.3A Pending CN109656682A (en) | 2018-12-03 | 2018-12-03 | A kind of system and method for the exploration and development big data processing platform based on container technique |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109656682A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110377399A (en) * | 2019-07-22 | 2019-10-25 | 中国联合网络通信集团有限公司 | HBase containerization method, apparatus, equipment and readable storage medium storing program for executing |
| CN110515602A (en) * | 2019-09-17 | 2019-11-29 | 成都源动数据科技有限公司 | A kind of online interaction formula programming opening experiment system |
| CN111078358A (en) * | 2019-12-05 | 2020-04-28 | 中车株洲电力机车有限公司 | Test data processing method and system based on software container |
| CN111949276A (en) * | 2020-08-17 | 2020-11-17 | 浪潮云信息技术股份公司 | System and method for automatically deploying application program based on container mode |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170039236A1 (en) * | 2015-08-06 | 2017-02-09 | International Business Machines Corporation | Vertical tuning of distributed analytics clusters |
| CN106888254A (en) * | 2017-01-20 | 2017-06-23 | 华南理工大学 | A kind of exchange method between container cloud framework based on Kubernetes and its each module |
| CN107733704A (en) * | 2017-09-29 | 2018-02-23 | 中国石油化工股份有限公司 | A kind of system and method for the exploration and development cloud based on virtualization and container technique |
| CN108037919A (en) * | 2017-12-01 | 2018-05-15 | 北京博宇通达科技有限公司 | A kind of visualization big data workflow configuration method and system based on WEB |
| CN108108226A (en) * | 2017-12-20 | 2018-06-01 | 恒泰艾普（北京）云技术有限公司 | A kind of large data files analysis and processing method under cloud computing environment |
-
2018
- 2018-12-03 CN CN201811467745.3A patent/CN109656682A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170039236A1 (en) * | 2015-08-06 | 2017-02-09 | International Business Machines Corporation | Vertical tuning of distributed analytics clusters |
| CN106888254A (en) * | 2017-01-20 | 2017-06-23 | 华南理工大学 | A kind of exchange method between container cloud framework based on Kubernetes and its each module |
| CN107733704A (en) * | 2017-09-29 | 2018-02-23 | 中国石油化工股份有限公司 | A kind of system and method for the exploration and development cloud based on virtualization and container technique |
| CN108037919A (en) * | 2017-12-01 | 2018-05-15 | 北京博宇通达科技有限公司 | A kind of visualization big data workflow configuration method and system based on WEB |
| CN108108226A (en) * | 2017-12-20 | 2018-06-01 | 恒泰艾普（北京）云技术有限公司 | A kind of large data files analysis and processing method under cloud computing environment |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110377399A (en) * | 2019-07-22 | 2019-10-25 | 中国联合网络通信集团有限公司 | HBase containerization method, apparatus, equipment and readable storage medium storing program for executing |
| CN110515602A (en) * | 2019-09-17 | 2019-11-29 | 成都源动数据科技有限公司 | A kind of online interaction formula programming opening experiment system |
| CN110515602B (en) * | 2019-09-17 | 2023-08-18 | 成都源动数据科技有限公司 | Online interactive programming open experiment system |
| CN111078358A (en) * | 2019-12-05 | 2020-04-28 | 中车株洲电力机车有限公司 | Test data processing method and system based on software container |
| CN111078358B (en) * | 2019-12-05 | 2023-09-05 | 中车株洲电力机车有限公司 | Test data processing method and system based on software container |
| CN111949276A (en) * | 2020-08-17 | 2020-11-17 | 浪潮云信息技术股份公司 | System and method for automatically deploying application program based on container mode |
| CN111949276B (en) * | 2020-08-17 | 2024-02-06 | 浪潮云信息技术股份公司 | System and method for automatically deploying application programs based on container mode |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10616132B2 (en) | Managing user privileges for computer resources in a networked computing environment | |
| CN107766126B (en) | Container mirror image construction method, system and device and storage medium | |
| CN108279932B (en) | Method and device for dynamically configuring user interface of mobile terminal | |
| CN109656682A (en) | A kind of system and method for the exploration and development big data processing platform based on container technique | |
| US9420034B2 (en) | Providing services to multiple tenants of an application | |
| Scolati et al. | A containerized big data streaming architecture for edge cloud computing on clustered single-board devices | |
| CN111459415B (en) | Cross-cloud data migration method, device, equipment and storage medium | |
| US10402746B2 (en) | Computing instance launch time | |
| EP3032442B1 (en) | Modeling and simulation of infrastructure architecture for big data | |
| US20130227563A1 (en) | Mechanism for Creating and Maintaining Multi-Tenant Applications in a Platform-as-a-Service (PaaS) Environment of a Cloud Computing System | |
| CN112560244A (en) | Virtual simulation experiment system and method based on Docker | |
| US9451033B2 (en) | Enhanced command selection in a networked computing environment | |
| AU2017327824B2 (en) | Data integration job conversion | |
| US8825862B2 (en) | Optimization of resource provisioning in a networked computing environment | |
| CN102291445A (en) | Cloud computing management system based on virtual resources | |
| US11960578B2 (en) | Correspondence of external operations to containers and mutation events | |
| Kumar et al. | An Implementation of Load Balancing Policy for Virtual Machines Associated with a Data Centre | |
| US20130212276A1 (en) | Command management in a networked computing environment | |
| CN115202729A (en) | Container service-based mirror image generation method, device, equipment and medium | |
| El-Omari | Cloud IoT as a crucial enabler: a survey and taxonomy | |
| US20220291904A1 (en) | Methods and systems for operationalizing artificial intelligence models using web-based user interfaces | |
| US11500816B2 (en) | Intelligent file recommendation engine | |
| CN107562894A (en) | A kind of API of web application systems behavior simulation obtains and methods for using them | |
| CN103051668A (en) | Server resource management system and server resource management method | |
| WO2022057634A1 (en) | Containerized software discovery and identification |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20190419 |
|
| WD01 | Invention patent application deemed withdrawn after publication |