WO2017161984A1 - 数据集群的部署方法、装置、***及计算机存储介质 - Google Patents

数据集群的部署方法、装置、***及计算机存储介质 Download PDF

Info

Publication number
WO2017161984A1
WO2017161984A1 PCT/CN2017/074073 CN2017074073W WO2017161984A1 WO 2017161984 A1 WO2017161984 A1 WO 2017161984A1 CN 2017074073 W CN2017074073 W CN 2017074073W WO 2017161984 A1 WO2017161984 A1 WO 2017161984A1
Authority
WO
WIPO (PCT)
Prior art keywords
data cluster
cluster system
configuration
information
deployment
Prior art date
Application number
PCT/CN2017/074073
Other languages
English (en)
French (fr)
Inventor
杨桂荣
Original Assignee
中兴通讯股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中兴通讯股份有限公司 filed Critical 中兴通讯股份有限公司
Publication of WO2017161984A1 publication Critical patent/WO2017161984A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/11—File system administration, e.g. details of archiving or snapshots
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/182—Distributed file systems

Definitions

  • the present invention relates to the field of communications, and in particular to a method, an apparatus, a system, and a computer storage medium for deploying a data cluster.
  • a data cluster system is usually used to store massive data.
  • the distributed system infrastructure developed by Apache the Hadoop system, stores massive amounts of data.
  • many data cluster systems often have complex deployment and management problems.
  • the Hadoop system is used as an example.
  • the Hadoop system is an open source software framework for distributed processing of large amounts of data.
  • the system includes a distributed file system (Hadoop Distributed File System, HDFS for short) and a distributed database (Hbase).
  • HDFS Hadoop Distributed File System
  • base distributed database
  • the storage core of the system is distributed.
  • File System (HDFS) is suitable for running on general-purpose hardware and needs to be deployed on a large number of machines to support large-scale data sets and high-throughput data access. HDFS achieves high fault tolerance through multiple copies that can be distributed across different machines.
  • Hadoop system also includes many other components for resource scheduling, data storage, and external connections, such as YARN, HBASE, zookeeper, HIVE, Impala, MapReduce, Oozie, Sqoop, Flume, etc.
  • Hadoop systems include components such as Figure 1 shows.
  • Hadoop clusters are powerful and widely used. However, due to their complex systems, Hadoop clusters are very complicated to deploy and manage. It requires dozens of servers to be installed and deployed. It is very labor-intensive. How to quickly and automatically build available Hadoop clusters and manage and monitor them. Operational status has become an important topic worth studying.
  • the embodiment of the present invention is to provide a method, an apparatus, a system, and a computer storage medium for deploying a data cluster, so as to at least solve the problem that the deployment and management of the data cluster in the related art are complicated.
  • a method for deploying a data cluster includes: acquiring first configuration information for a data cluster system deployment, where the first configuration information includes: And the first deployment request for carrying the first configuration information, where the first deployment request is used to request the cloud platform to create a data cluster system deployment manner according to the first configuration information.
  • Required hardware resources and/or configuration are required to be used to request the cloud platform to create a data cluster system deployment manner according to the first configuration information.
  • the acquiring the first configuration information for the data cluster system deployment comprises: receiving a first configuration file defined by a user; and acquiring the first configuration information according to the first configuration file.
  • the acquiring the first configuration information according to the first configuration file includes: parsing the first configuration file, obtaining a first configuration parameter related to the data cluster system deployment; and calculating the first configuration according to the first configuration parameter information.
  • the first configuration file further includes information for components of the data cluster system deployment; after the first deployment request carrying the first configuration information is sent to the cloud platform, the method further Including: detecting whether the hardware resource is successfully created; in the case that the hardware resource is successfully created, the plug-in is generated according to the information of the component used for the data cluster system deployment.
  • the generating the plug-in according to the component for the data cluster system deployment comprises: acquiring the first data cluster system version file information, wherein the first data cluster system version file information is the deployed data. Information of the version file of the cluster system; corresponding to the information of the component used for data cluster system deployment and the first data cluster system version file information Version file, build plugin.
  • the plugin is a plugin with component functionality.
  • the plugin is a plurality of plugins, and the plurality of plugins are deployed in a layered structure.
  • the method further comprises: running the plugin to implement at least one of the following processes: network environment configuration; component resource allocation; clock Synchronization; configure the component to the target node of the deployed data cluster system.
  • the method further includes: monitoring an operating environment of the data cluster system, where the operating environment includes a resource occupation status, and automatically detecting that an abnormal resource occupancy condition occurs.
  • the data cluster system is tuned; and/or, the operating state of the data cluster system is monitored, and the data cluster system is automatically tuned in the event that the operating state of the data cluster system is abnormal.
  • the method further includes: receiving a second configuration file for upgrading the data cluster system; parsing the second configuration file to obtain a second configuration parameter related to the data cluster system upgrade; determining according to the second configuration parameter Whether the data cluster system upgrade needs to add hardware resources and/or configuration; if the hardware resources and/or configuration need to be added, the second deployment request carrying the second configuration parameter is sent to the cloud platform, where the second deployment request is used.
  • the requesting cloud platform creates hardware resources and/or configurations required for the data cluster system upgrade according to the second configuration parameter.
  • the second configuration file further includes information for components of the data cluster system upgrade; if the hardware resources and/or configuration are not required to be added, the method further includes: acquiring the second data Cluster system version file information, wherein the second data cluster system version file information is information of a version file of the upgraded data cluster system; information according to components used for upgrading the data cluster system and the second data cluster system The version file corresponding to the version file information, and the plugin is generated.
  • a data cluster deployment apparatus including: obtaining The module is configured to obtain first configuration information for the data cluster system deployment, where the first configuration information includes hardware resources and/or configuration information required for deploying the data cluster system; and the sending module is configured to move to the cloud
  • the platform sends a first deployment request that carries the first configuration information, where the first deployment request is used to request the cloud platform to create hardware resources and/or configurations required for the data cluster system deployment according to the first configuration information.
  • the acquiring module is configured to receive a first configuration file defined by a user, and obtain the first configuration information according to the first configuration file.
  • the acquiring module is configured to parse the first configuration file, obtain a first configuration parameter related to the data cluster system deployment, and calculate the first configuration information according to the first configuration parameter. .
  • the first configuration file further includes information for components of the data cluster system deployment; the device further includes a generating module configured to send the carrying module to the cloud platform After the first deployment request of the first configuration information, it is detected whether the hardware resource is successfully created; if the hardware resource is successfully created, the plug-in is generated according to information for components deployed by the data cluster system.
  • the generating module is configured to obtain the first data cluster system version file information, where the first data cluster system version file information is information of a version file of the deployed data cluster system; Generating the plug-in to the version file corresponding to the data cluster system deployment component and the version file corresponding to the first data cluster system version file information.
  • the apparatus further includes an execution module configured to execute the plug-in after the generation module generates the plug-in according to the information for the component deployed by the data cluster system to implement at least one of the following processes : network environment configuration; component resource allocation; clock synchronization; configuring the component to the target node of the deployed data cluster system.
  • the apparatus further includes a monitoring module configured to monitor an operating environment of the data cluster system, wherein the operating environment includes a resource occupancy status, and the monitoring occurs. If the resource occupation status is abnormal, the data cluster system is automatically tuned; and/or the running status of the data cluster system is monitored, and if the running status of the data cluster system is abnormal, the automatic The data cluster system is tuned.
  • a monitoring module configured to monitor an operating environment of the data cluster system, wherein the operating environment includes a resource occupancy status, and the monitoring occurs. If the resource occupation status is abnormal, the data cluster system is automatically tuned; and/or the running status of the data cluster system is monitored, and if the running status of the data cluster system is abnormal, the automatic The data cluster system is tuned.
  • the device further includes a determining module
  • the obtaining module is further configured to receive a second configuration file for upgrading the data cluster system; parse the second configuration file to obtain a second configuration parameter related to the data cluster system upgrade;
  • the determining module is configured to determine, according to the second configuration parameter, whether the data cluster system upgrade needs to increase hardware resources and/or configuration;
  • the sending module is configured to send, to the cloud platform, a second deployment request that carries the second configuration parameter, where the hardware resource and/or configuration needs to be added, where the second deployment request is used for And requesting the cloud platform to create hardware resources and/or configurations required for the data cluster system upgrade according to the second configuration parameter.
  • the apparatus further includes a generating module; the second configuration file further includes information for components of the data cluster system upgrade;
  • the obtaining module is further configured to: when the determining module determines that the hardware resource and/or the configuration is not required to be added, obtain the second data cluster system version file information, where the second data cluster system version file information is Information about the version file of the upgraded data cluster system;
  • the generating module is configured to generate a plug-in according to the information of the component used for the data cluster system upgrade and the version file corresponding to the second data cluster system version file information.
  • a data cluster deployment system includes: a processor configured to acquire first configuration information for a data cluster system deployment, and send the first configuration information to a cloud platform The first deployment request, where the first configuration information includes hardware resources and/or configuration information required to deploy the data cluster system, and the first deployment request is used to request the cloud platform to create a data cluster according to the first configuration information.
  • Hardware resources required for system deployment and / Or a cloud platform configured to create hardware resources and/or configurations required for data cluster system deployment according to the first configuration information.
  • a computer storage medium storing computer executable instructions for executing a data cluster according to an embodiment of the present invention is provided. Deployment method.
  • the embodiment of the present invention obtains the first configuration information for the data cluster system deployment, where the first configuration information includes hardware resources and/or configuration information required for deploying the data cluster system; and the first configuration is sent to the cloud platform.
  • a first deployment request of the information where the first deployment request is used to request the cloud platform to create hardware resources and/or configurations required for the data cluster system deployment according to the first configuration information, and implement batch creation of hardware resources and/or through the cloud platform.
  • the configuration solves the problem of complicated deployment and management of the data cluster, thereby effectively simplifying the deployment process of the actual development environment, the test environment, and the production environment of the data cluster, thereby greatly saving equipment and manpower, and improving the data cluster. Deployment efficiency.
  • FIG. 1 is a schematic structural diagram of components of a Hadoop system in the related art
  • FIG. 2 is a flowchart of a method for deploying a data cluster according to an embodiment of the present invention
  • FIG. 3 is a structural block diagram of a device for deploying a data cluster according to an embodiment of the present invention
  • FIG. 4 is a schematic diagram of a deployment system of a data cluster according to an embodiment of the present invention.
  • FIG. 5 is a schematic diagram of a deployment system of a data cluster according to an embodiment of the present invention.
  • FIG. 2 is a flowchart of a method for deploying a data cluster according to an embodiment of the present invention. As shown in FIG. 2, the process includes the following steps:
  • Step S202 acquiring first configuration information for the data cluster system deployment, where the first configuration information includes information about hardware resources and/or configuration required to deploy the data cluster system;
  • Step S204 Send a first deployment request that carries the first configuration information to the cloud platform, where the first deployment request is used to request the cloud platform to create hardware resources and/or configurations required for the data cluster system deployment according to the first configuration information.
  • the cloud platform is a service platform built by using cloud technology, and can provide services such as infrastructure or virtualized environment.
  • the hardware resources required to deploy a data cluster system include the infrastructure required for system deployment.
  • the cloud platform can be a cloud platform based on PAAS (Platform-as-a-Service, which can be referred to as platform as a service), and the PAAS-based cloud platform can provide sufficient hardware support for the Hadoop system.
  • PAAS Platform-as-a-Service
  • the first configuration information may be related configuration information in a user-defined configuration file.
  • the first configuration information may include at least one of the following data: related data of the virtual machine template selected by the user, and the user.
  • the configuration file can be serialized in advance, and the user only needs to change a small number of parameters to complete the configuration of the data cluster system, and the configuration file is the only deployment list that the user needs to fill in, and the deployment of the data cluster system becomes It's even simpler.
  • the data cluster deployment method of the embodiment of the present invention is applicable to the deployment of a big data cluster, for example, the deployment of a hadoop system.
  • the foregoing embodiment obtains the first configuration information for the data cluster system deployment, where the first configuration information includes hardware resources and/or configuration information required for deploying the data cluster system; and the first configuration is sent to the cloud platform.
  • a first deployment request of the information where the first deployment request is used to request the cloud platform to create hardware resources and/or configurations required for the data cluster system deployment according to the first configuration information, and implement batch creation of hardware resources and/or through the cloud platform.
  • the configuration solves the problem of complicated deployment and management of the data cluster, thereby effectively simplifying the deployment process of the actual development environment, the test environment, and the production environment of the data cluster, thereby greatly saving equipment and manpower, and improving the data cluster. Deployment efficiency.
  • the acquiring the first configuration information for the data cluster system deployment includes: receiving a first configuration file defined by the user; and acquiring the first configuration information according to the first configuration file.
  • the first configuration file is user-defined (the user modifies and/or selects parameters of the standard configuration file according to the requirements of the user to obtain the first configuration file), and the configuration method can better meet the requirements of the user, and is deployed. The operation is more convenient.
  • the acquiring the first configuration information according to the first configuration file includes: parsing the first configuration file, obtaining first configuration parameters related to data cluster system deployment, and calculating first configuration information according to the first configuration parameter.
  • the user provides a first configuration file required for data cluster system deployment, and after receiving the first configuration file, the processor parses parameters in the first configuration file, and automatically calculates required hardware resources (infrastructure). And configuring the situation, and then triggering the automatic deployment request, sending a request to create a hardware resource and/or configuration to the PAAS cloud platform, where the request carries the calculated hardware resource and/or configuration information.
  • the embodiment After receiving the configuration file of the user, the embodiment automatically triggers a series of operations such as parsing and requesting, and the deployment process is very efficient and convenient without requiring the user to re-enter the effort.
  • the first configuration file further includes: information for a component of the data cluster system deployment; after the sending, by the cloud platform, the first deployment request that carries the first configuration information, the method further includes: detecting Whether the hardware resource is successfully created; if the hardware resource is successfully created, the plug-in is generated according to the information of the component used for data cluster system deployment.
  • a main control module can be automatically generated.
  • the plug-in can be automatically generated according to the first configuration file, and the plug-in is configured and run.
  • the generating the plug-in according to the component for the data cluster system deployment includes: acquiring the first data cluster system version file information, where the first data cluster system version file information is the deployed data.
  • the information of the version file of the cluster system; the plug-in is generated according to the information of the component used for the data cluster system deployment and the version file corresponding to the first data cluster system version file information.
  • the first data cluster system version file information may be a version installation package of the big data cluster system to be deployed, and may be obtained by the operator from the version machine and placed in the version directory.
  • a main control module can be automatically generated.
  • the user-defined configuration files (including component parameters) and the hadoop version file can be automatically Generate plugins.
  • the first deployment request is further configured to request to select a required virtual host template from multiple data cluster system version files stored in the cloud platform to generate a virtual machine.
  • the cloud platform may pre-store a plurality of virtual host templates, and the user selects a required virtual host template in the configuration file according to the requirement, and sends the request to the cloud platform by using a request message, and the cloud platform determines the virtual to be generated according to the request. machine.
  • the plug-in is a plug-in with component functionality, the plug-in being able to run independently.
  • a plug-in containing a component function is automatically generated by a remote command, the plug-in Can run independently from the operating system.
  • the plugin can be regarded as the carrier of the component.
  • a big data cluster system can be thought of as consisting of a series of components.
  • a plug-in with component functionality is a plug-in that can be deployed at any time, with the functionality of the components that need to be deployed. For example, the installation of a HIVE version package requires a lot of configuration, and by generating a plug-in with component functions, the embodiment only needs to configure the plug-in to complete the complicated configuration process of the component, thereby effectively improving the configuration efficiency of the component.
  • This embodiment effectively overcomes the inefficiency caused by the requirement that multiple components are deployed one by one in the deployment of the existing big data cluster system, and effectively improves the deployment efficiency of the big data cluster system.
  • the method further includes: running the plug-in to implement at least one of the following processes: network environment configuration; component resource allocation; clock synchronization; The component is configured to the target node of the deployed data cluster system.
  • all the plug-ins can be automatically configured by the batch configuration tool to implement network environment configuration, component resource allocation, clock synchronization, and configuring the components to target nodes of the deployed data cluster system.
  • the plug-in is a plurality of plug-ins, and the plurality of plug-ins are deployed in a layered structure.
  • the layered deployment of the big data plug-in can support flexible use of the plug-in, facilitate system adjustment and expansion, and automatically perform performance tuning.
  • the method further includes: monitoring an operating environment of the data cluster system, where the operating environment includes a resource occupation status, and automatically monitoring the data cluster when the resource occupancy status is abnormal.
  • the system is tuned; and/or, the operating state of the data cluster system is monitored, and the data cluster system is automatically tuned in the event that the operating state of the data cluster system is abnormal.
  • Kerberos Kerberos is a system for secure authentication
  • identity authentication Kerberos is a system for secure authentication
  • number of automatic monitoring is monitored.
  • the master process is automatically notified, and the master process automatically completes the automatic tuning of the data cluster system. excellent.
  • the method further includes: receiving a second configuration file for upgrading the data cluster system; parsing the second configuration file to obtain a second configuration parameter related to the data cluster system upgrade; determining according to the second configuration parameter Whether the data cluster system upgrade needs to add hardware resources and/or configuration; if the hardware resources and/or configuration need to be added, the second deployment request carrying the second configuration parameter is sent to the cloud platform, where the second deployment request is used.
  • the requesting cloud platform creates hardware resources and/or configurations required for the data cluster system upgrade according to the second configuration parameter.
  • the second configuration file further includes information for components of the data cluster system upgrade, and if the hardware resources and/or configuration are not required to be added, the method further includes: acquiring the second data cluster system version.
  • File information wherein the second data cluster system version file information is information of a version file of the upgraded data cluster system; and information corresponding to the component for the data cluster system upgrade and the second data cluster system version file information Version file, build plugin.
  • the second data cluster system version file information may be a version package of the big data cluster system to be upgraded, and may be obtained by the operator from the version machine and placed in the version directory.
  • the foregoing embodiment provides a configuration method when a data cluster system needs to be upgraded.
  • the user provides a configuration file (second configuration file) required for the data cluster system upgrade, and after receiving the configuration file, the processor parses the parameters in the configuration file, and automatically calculates whether the upgrade needs to add hardware resources (infrastructure) and configuration. happening. If necessary, an automatic deployment request is triggered, and a request for creating an infrastructure is sent to the PAAS platform, and the request carries the calculated configuration information. After the infrastructure is successfully created, the main control module is automatically notified to control the corresponding plug-in through the main control module.
  • second configuration file second configuration file
  • the processor parses the parameters in the configuration file, and automatically calculates whether the upgrade needs to add hardware resources (infrastructure) and configuration. happening. If necessary, an automatic deployment request is triggered, and a request for creating an infrastructure is sent to the PAAS platform, and the request carries the calculated configuration information. After the infrastructure is successfully created, the main control module is automatically notified to control the
  • the processor (the plug-in auto-generating module in the processor) passes the main control module and is based on the configuration file and the new data cluster system version file (the second data cluster system version file) The version file corresponding to the information is automatically generated; the processor (the batch configuration module in the processor) stops the plug-in to be upgraded and replaces it with a new plug-in according to the configuration file, and then configures and runs it.
  • the processor can also use the security and monitoring automation functions to authenticate the identity through Kerberos, and automatically monitor the operation of the relevant components of the data cluster system. If the environment is running normally, and/or the data cluster system is running normally, continuous monitoring; if the environment is abnormal in resource occupation and/or the system is in an abnormal state, the main control process can be automatically notified, and the main control process automatically completes the data cluster system. Automatic tuning.
  • the system upgrade can be realized without stopping the operation of the data cluster system, thereby achieving a seamless upgrade and improving the user experience.
  • the data cluster deployment method proposed by the above implementation can deploy a big data environment through plug-in of components, expand an updateable component plug-in warehouse, store and manage through a PAAS cloud platform, deploy and deploy through automated deployment scripts and management scripts. Manage and monitor the health of big data clusters by automatically collecting system information to achieve the goal of automatically deploying and managing big data environments.
  • the big data plug-in layered deployment can support the flexible use of plug-ins, facilitate adjustment and expansion, and support automatic performance tuning.
  • the method is easy to automatically package and deploy applications, create a lightweight PAAS environment, and can greatly save equipment and human resources and improve deployment efficiency in actual development/test environment deployment or production environment deployment. It also enables a seamless upgrade of the big data platform.
  • the following provides a method for deploying a data cluster according to an embodiment of the present invention, where the method includes the following steps:
  • Step 11 The user provides a configuration file required for the Hadoop system deployment; wherein the configuration file is equivalent to the first configuration file described above.
  • Step 12 After receiving the configuration file, the processor parses the parameters in the configuration file, automatically calculates the required infrastructure and configuration, and then triggers an automatic deployment request, and sends a request for creating an infrastructure and/or configuration to the PaaS platform, in the request. Carrying the calculated configuration information; wherein the request is equivalent to the first deployment request described above.
  • Step 13 After the infrastructure is successfully created, the hadoop master module is automatically generated.
  • Step 14 The plug-in automatic generation module automatically generates the component plug-in according to the standard configuration file and the hadoop version file (the version file corresponding to the first data cluster version file information) under the control of the main control module.
  • Step 15 The batch configuration module configures and runs the plugin according to the configuration file.
  • Step 16 The security and monitoring module automatically uses Kerberos for identity authentication to automatically monitor the operation of the relevant components of hadoop.
  • Step 17 If the environment is running normally, and / or the system is running normally, continuous monitoring.
  • Step 18 If the environment is abnormal or the system is abnormal, the master process is automatically notified. The master process automatically performs automatic tuning of the hadoop system.
  • Another method for deploying a data cluster is provided according to an embodiment of the present invention.
  • the method is applied to an upgrade of a big data system, and the method includes the following steps:
  • Step 21 The user provides a configuration file required for the Hadoop system upgrade.
  • Step 22 After receiving the configuration file, the processor parses the parameters in the configuration file, and automatically calculates whether the upgrade needs to add infrastructure and/or configuration. If necessary, triggers step 23; if not, triggers step 25 .
  • Step 23 Trigger an automatic deployment request, and send a creation infrastructure request (the infrastructure request is equivalent to the second deployment request) to the PaaS platform, where the request carries the calculated configuration information.
  • Step 24 After the infrastructure is successfully created, the hadoop master module is automatically notified.
  • Step 25 The plug-in automatic generation module automatically generates the component plug-in according to the configuration file and the new hadoop version file (the version file corresponding to the second data cluster system version file information) under the control of the main control module.
  • Step 26 The batch configuration module stops the plug-in to be upgraded and replaces it with a new plug-in according to the configuration file, and configures and runs the new plug-in.
  • Step 27 The security and monitoring module automatically uses Kerberos for identity authentication to automatically monitor the operation of Hadoop related components.
  • Step 28 If the environment is running normally, and/or the system is running normally, it is continuously monitored.
  • Step 29 If the environment is abnormal or the system is abnormal, the master process is automatically notified, and the master process automatically performs automatic tuning of the hadoop system.
  • the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware, but in many cases, the former is A better implementation.
  • the technical solution of the present invention which is essential or contributes to the prior art, may be embodied in the form of a software product stored in a storage medium (such as ROM/RAM, disk,
  • the optical disc includes a number of instructions for causing a terminal device (which may be a cell phone, a computer, a server, or a network device, etc.) to perform the methods described in various embodiments of the present invention.
  • a device for deploying a data cluster is provided, which is used to implement the foregoing embodiments and preferred embodiments, and details are not described herein.
  • the term "module” may implement a combination of software and/or hardware of a predetermined function.
  • the apparatus described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware, is also possible and contemplated.
  • FIG. 3 is a structural block diagram of a device for deploying a data cluster according to an embodiment of the present invention. As shown in FIG. 3, the device includes:
  • the obtaining module 30 is configured to acquire first configuration information for the data cluster system deployment, where the first configuration information includes information about hardware resources and/or configuration required to deploy the data cluster system;
  • the sending module 32 is configured to send, to the cloud platform, a first deployment request that carries the first configuration information, where the first deployment request is used to request the cloud platform to create hardware resources required for the data cluster system deployment according to the first configuration information. And / or configuration.
  • the first configuration information for the data cluster system deployment is obtained by the acquiring module 30, where the first configuration information includes hardware resources and/or configuration information required for deploying the data cluster system;
  • the sending module 32 sends a first deployment request that carries the first configuration information to the cloud platform, where the first deployment request is used to request the cloud platform to create hardware resources required for the data cluster system deployment according to the first configuration information.
  • / or configuration to achieve the mass creation of hardware resources and / or configuration through the cloud platform, to solve the problem of complex deployment and management of the data cluster, thereby effectively simplifying the actual development environment, test environment and production environment of the data cluster.
  • the deployment process can greatly save equipment and manpower and improve the efficiency of data cluster deployment.
  • the acquiring module 30 is configured to receive a first configuration file defined by a user, and obtain the first configuration information according to the first configuration file.
  • the acquiring module 30 is configured to parse the first configuration file, obtain a first configuration parameter related to the data cluster system deployment, and calculate the first configuration according to the first configuration parameter. information.
  • the first configuration file further includes information for components of the data cluster system deployment; the device further includes a generating module, configured to send the carrying module to the cloud platform After the first deployment request of the first configuration information, it is detected whether the hardware resource is successfully created; if the hardware resource is successfully created, the plug-in is generated according to information for components deployed by the data cluster system.
  • the generating module is configured to obtain a first data cluster system version.
  • the file information wherein the first data cluster system version file information is information of a version file of the deployed data cluster system; information according to components used for the data cluster system deployment and the first data cluster system
  • the version file corresponding to the version file information generates the plugin.
  • the apparatus further includes an operation module configured to run the plug-in after the generation module generates the plug-in according to the information for the component deployed by the data cluster system to implement at least one of the following processes: : network environment configuration; component resource allocation; clock synchronization; configuring the component to the target node of the deployed data cluster system.
  • the device further includes a monitoring module configured to monitor an operating environment of the data cluster system, where the operating environment includes a resource occupation status, and automatically detects that an abnormal resource occupancy condition occurs. Tuning the data cluster system; and/or monitoring the running status of the data cluster system, and automatically tuning the data cluster system if an abnormality occurs in the running state of the data cluster system.
  • the device further includes a determining module
  • the obtaining module 30 is further configured to receive a second configuration file for upgrading the data cluster system; and parse the second configuration file to obtain a second configuration parameter related to the data cluster system upgrade;
  • the determining module is configured to determine, according to the second configuration parameter, whether the data cluster system upgrade needs to increase hardware resources and/or configuration;
  • the sending module 32 is configured to send, to the cloud platform, a second deployment request that carries the second configuration parameter, where the hardware resource and/or configuration needs to be added, where the second deployment request is used by the second deployment request. And requesting the cloud platform to create hardware resources and/or configurations required for the data cluster system upgrade according to the second configuration parameter.
  • the apparatus further includes a generating module, where the second configuration file further includes information for components of the data cluster system upgrade;
  • the obtaining module 30 is further configured to determine, at the determining module, that the hardware resource does not need to be added. And obtaining the second data cluster system version file information, where the second data cluster system version file information is information of the version file of the upgraded data cluster system;
  • the generating module is configured to generate a plug-in according to the information of the component used for the data cluster system upgrade and the version file corresponding to the second data cluster system version file information.
  • each of the above modules may be implemented by software or hardware: the above modules are all located in the same processor; or the modules are respectively located in multiple processors.
  • FIG. 4 is a schematic diagram of a deployment system of a data cluster according to an embodiment of the present invention. As shown in FIG. 4, the system includes:
  • the processor 40 is configured to acquire first configuration information for the data cluster system deployment, and send, to the cloud platform, a first deployment request that carries the first configuration information, where the first configuration information includes hardware required to deploy the data cluster system.
  • the resource and/or configuration information, the first deployment request is used to request the cloud platform to create hardware resources and/or configurations required for the data cluster system deployment according to the first configuration information;
  • the cloud platform 42 is configured to create hardware resources and/or configurations required for data cluster system deployment according to the first configuration information.
  • the first configuration information for the data cluster system deployment is obtained by the processor 40, and the first deployment request carrying the first configuration information is sent to the cloud platform, where the first configuration information includes the deployment.
  • the first deployment request is used to request the cloud platform to create hardware resources and/or configurations required for the data cluster system deployment according to the first configuration information, where the cloud platform 42 is configured according to the hardware resources and/or configuration information required by the data cluster system.
  • the first configuration information creates hardware resources and/or configurations required for data cluster system deployment, and implements batch creation of hardware resources and/or configurations through the cloud platform, thereby solving the problem of complicated deployment and management of the data cluster, thereby effectively simplifying the problem.
  • the deployment process of the actual development environment, test environment and production environment of the data cluster can greatly save equipment and manpower, and improve the efficiency of data cluster deployment.
  • FIG. 5 is a schematic diagram of a deployment system of a data cluster according to an embodiment of the present invention.
  • the processor 40 may further include: a main control module 502, a plug-in generating module 504, a standard configuration module 506, a batch configuration module 508, and a security and monitoring module 510.
  • the functions of each module are as follows:
  • the main control module 502 the module is configured to assemble and run the plug-in, and has the function of assembling the data cluster system version file into a plug-in that can operate independently of the hardware and the operating system.
  • big data components such as YARN, HBASE, zookeeper, HIVE, Impala, MapReduce, Oozie, Sqoop, Flume, etc. can be packaged into separate plug-ins, so that plug-ins can be deployed independently from the operating system and hardware, with complete isolation. Sexuality, and resources for hardware and operating systems can be reused.
  • the plug-in generating module 504 is configured to utilize the container technology and the container technology to assemble the data cluster system version file into a plug-in according to a customized function through the main control module for deployment and use.
  • the standard configuration module 506 (the function of acquiring the module and the sending module in the deployment device of the data cluster) is configured to serialize and standardize the data cluster system configuration file, and only need to change a small number of parameters to complete the data when using.
  • the configuration of the cluster system is the only deployment list that the user needs to fill out.
  • the standard configuration module standardizes the configuration file. By extracting the configurable items of the data cluster and forming a standard configuration file, the user operation is simplified and the user experience is improved.
  • the batch configuration module 508 is configured to use the batch configuration tool to automatically configure all the plug-ins according to the configuration file (such as the first configuration file and the second configuration file) to support network environment construction, component resource allocation, Clock synchronization and application configuration to node operations, etc., to provide a unified configuration environment for data cluster system components.
  • the configuration file such as the first configuration file and the second configuration file
  • the security and monitoring module 510 is configured to utilize Kerberos (hadoop's own Kerberos identity authentication system) for identity authentication.
  • Kerberos hadoop's own Kerberos identity authentication system
  • the module can automatically monitor the running of the relevant components of the data cluster system and complete the tuning of the data cluster system according to the resource occupancy.
  • the cloud platform 42 is used to provide sufficient hardware support for the data cluster system.
  • the virtual host template of the common operating system is stored in the cloud computing platform (the cloud platform), and the cloud platform can set the master computer required by the data cluster for the user through the virtual machine template and configuration parameters selected by the user, and then pass the configuration file. Determine the components that need to be installed. After confirming the correctness, the processor 40 will automatically generate the required plug-ins and deploy them as a layered structure to build a big data cluster.
  • the deployment of the plug-in-based data cluster system is implemented, and the automatic deployment and component allocation are supported, and the plug-in is deployed in the cluster in the form of a layered plug-in, which can support flexible use and convenience. Adjustment and expansion for automatic performance tuning.
  • the deployment system of the data cluster implements batch construction of hardware resources and/or configurations through the cloud platform, and solves the problem that the deployment and management of the data clusters in the prior art are complicated.
  • Embodiments of the present invention also provide a storage medium.
  • the above storage medium may be configured to store program code for performing the following steps:
  • the first deployment request that carries the first configuration information is sent to the cloud platform, where the first deployment request is used to request the cloud platform to create hardware resources and/or configurations required for the data cluster system deployment according to the first configuration information.
  • the foregoing storage medium may include, but is not limited to, a USB flash drive, a Read-Only Memory (ROM), a Random Access Memory (RAM), and a mobile hard disk.
  • ROM Read-Only Memory
  • RAM Random Access Memory
  • the disclosed apparatus and method may be implemented in other manners.
  • the device embodiments described above are merely illustrative.
  • the division of the unit is only a logical function division.
  • there may be another division manner such as: multiple units or components may be combined, or Can be integrated into another system, or some features can be ignored or not executed.
  • the coupling, or direct coupling, or communication connection of the components shown or discussed may be indirect coupling or communication connection through some interfaces, devices or units, and may be electrical, mechanical or other forms. of.
  • the units described above as separate components may or may not be physically separated, and the components displayed as the unit may or may not be physical units, that is, may be located in one place or distributed to multiple network units; Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.
  • each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may be separately used as one unit, or two or more units may be integrated into one unit;
  • the unit can be implemented in the form of hardware or in the form of hardware plus software functional units.
  • the foregoing program may be stored in a computer readable storage medium, and the program is executed when executed.
  • the foregoing storage device includes the following steps: the foregoing storage medium includes: a mobile storage device, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
  • ROM read-only memory
  • RAM random access memory
  • magnetic disk or an optical disk.
  • optical disk A medium that can store program code.
  • the above-described integrated unit of the present invention may be stored in a computer readable storage medium if it is implemented in the form of a software function module and sold or used as a standalone product. Based on such understanding, the technical solution of the embodiments of the present invention is made substantially or prior to the prior art.
  • the contributed portion may be embodied in the form of a software product stored in a storage medium, including instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to perform the various aspects of the present invention. All or part of the methods described in the examples.
  • the foregoing storage medium includes various media that can store program codes, such as a mobile storage device, a ROM, a RAM, a magnetic disk, or an optical disk.
  • the technical solution of the embodiment of the present invention obtains the first configuration information for the data cluster system deployment, where the first configuration information includes hardware resources and/or configuration information required for deploying the data cluster system; a first deployment request of the first configuration information, where the first deployment request is used to request the cloud platform to create hardware resources and/or configurations required for the data cluster system deployment according to the first configuration information, and implement batch creation of hardware resources by using the cloud platform.
  • And/or configuration which solves the problem of complicated deployment and management of data clusters, thereby effectively simplifying the deployment process of the actual development environment, test environment and production environment of the data cluster, and can greatly save equipment and manpower, and improve the system. The efficiency of data cluster deployment.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Stored Programmes (AREA)

Abstract

本发明实施例公开了一种数据集群的部署方法、装置、***及计算机存储介质。所述方法包括：获取用于数据集群***部署的第一配置信息，其中，第一配置信息包括部署数据集群***所需的硬件资源和/或配置的信息；向云平台发送携带有第一配置信息的第一部署请求，其中，第一部署请求用于请求云平台根据第一配置信息创建数据集群***部署所需的硬件资源和/或配置。

Description

数据集群的部署方法、装置、***及计算机存储介质 技术领域
本发明涉及通信领域，具体而言，涉及一种数据集群的部署方法、装置、***及计算机存储介质。
背景技术
随着互联网的发展，为了满足用户对海量数据的存储需求，通常会使用数据集群***存储海量数据。例如，使用Apache开发的分布式***基础架构，即海杜普(Hadoop)***，存储海量数据。但是，很多数据集群***往往存在部署和管理复杂的问题，下面以Hadoop***为例进行说明。
Hadoop***是一个能够对大量数据进行分布式处理的开源软件框架，该***包括分布式文件***(Hadoop Distributed File System，简称为HDFS)和分布式数据库(Hbase)，该***的存储核心是分布式文件***(HDFS)。HDFS适合运行在通用的硬件上，并且需要部署在大量机器上，以此来支持大规模的数据集和高吞吐量的数据访问。HDFS通过可以分布在不同机器上的多个副本数来实现高容错性。
另外，Hadoop***还包括很多其他组件，用来实现资源调度、数据存储以及和外部的连接，如YARN、HBASE、zookeeper、HIVE、Impala、MapReduce、Oozie、Sqoop、Flume等，Hadoop***包括的组件如图1所示。
Hadoop集群功能强大，应用广泛，但是由于其***复杂，Hadoop集群的部署和管理非常繁杂，动辄需要数十台服务器的安装部署，十分耗费精力，如何快速自动化地构建可用的Hadoop集群并管理监控其运行状态成为值得研究的重要课题。
针对相关技术中，数据集群的部署和管理较为复杂的问题，目前尚未提出有效的解决方案。
发明内容
本发明实施例期望提供一种数据集群的部署方法、装置、***及计算机存储介质，以至少解决相关技术中数据集群的部署和管理较为复杂的问题。
根据本发明实施例的第一方面，提供了一种数据集群的部署方法，包括：获取用于数据集群***部署的第一配置信息，其中，所述第一配置信息包括部署数据集群***所需的硬件资源和/或配置的信息；向云平台发送携带有第一配置信息的第一部署请求，其中，所述第一部署请求用于请求云平台根据第一配置信息创建数据集群***部署所需的硬件资源和/或配置。
在一实施例中，所述获取用于数据集群***部署的第一配置信息包括：接收用户定义的第一配置文件；根据第一配置文件获取所述第一配置信息。
在一实施例中，所述根据第一配置文件获取第一配置信息包括：解析第一配置文件，获得与数据集群***部署相关的第一配置参数；根据所述第一配置参数计算第一配置信息。
在一实施例中，所述第一配置文件中还包括用于数据集群***部署的组件的信息；所述在向云平台发送携带有第一配置信息的第一部署请求之后，所述方法还包括：检测硬件资源是否创建成功；在硬件资源创建成功的情况下，根据用于数据集群***部署的组件的信息生成插件。
在一实施例中，所述根据用于数据集群***部署的组件的信息生成插件包括：获取第一数据集群***版本文件信息，其中，所述第一数据集群***版本文件信息为所部署的数据集群***的版本文件的信息；根据用于数据集群***部署的组件的信息和第一数据集群***版本文件信息对应的 版本文件，生成插件。
在一实施例中，所述插件为具备组件功能的插件。
在一实施例中，所述插件为多个插件，所述多个插件以层状结构部署。
在一实施例中，在根据用于数据集群***部署的组件的信息生成插件之后，所述方法还包括：运行所述插件，以实现以下过程至少之一：网络环境配置；组件资源分配；时钟同步；将组件配置到所部署的数据集群***的目标节点。
在一实施例中，在运行所述插件之后，所述方法还包括：监测数据集群***的运行环境，其中，运行环境包括资源占用状况，在监测到发生资源占用状况异常的情况下，自动对数据集群***进行调优；和/或，监测数据集群***的运行状态，在发生数据集群***的运行状态异常的情况下，自动对数据集群***进行调优。
在一实施例中，所述方法还包括：接收用于数据集群***升级的第二配置文件；解析第二配置文件，得到与数据集群***升级相关的第二配置参数；根据第二配置参数判断数据集群***升级是否需要增加硬件资源和/或配置；在需要增加硬件资源和/或配置的情况下，向云平台发送携带有第二配置参数的第二部署请求，其中，第二部署请求用于请求云平台根据第二配置参数创建数据集群***升级所需的硬件资源和/或配置。
在一实施例中，所述第二配置文件中还包括用于数据集群***升级的组件的信息；在不需要增加硬件资源和/或配置的情况下，所述方法还包括：获取第二数据集群***版本文件信息，其中，所述第二数据集群***版本文件信息为升级后的数据集群***的版本文件的信息；根据用于所述数据集群***升级的组件的信息和第二数据集群***版本文件信息对应的版本文件，生成插件。
根据本发明的第二方面，提供了一种数据集群的部署装置，包括：获 取模块，配置为获取用于数据集群***部署的第一配置信息，其中，所述第一配置信息包括部署数据集群***所需的硬件资源和/或配置的信息；发送模块，配置为向云平台发送携带有第一配置信息的第一部署请求，其中，所述第一部署请求用于请求云平台根据第一配置信息创建数据集群***部署所需的硬件资源和/或配置。
在一实施例中，所述获取模块，配置为接收用户定义的第一配置文件；根据所述第一配置文件获取所述第一配置信息。
在一实施例中，所述获取模块，配置为解析所述第一配置文件，获得与所述数据集群***部署相关的第一配置参数；根据所述第一配置参数计算所述第一配置信息。
在一实施例中，所述第一配置文件中还包括用于所述数据集群***部署的组件的信息；所述装置还包括生成模块，配置为在所述发送模块向云平台发送携带有所述第一配置信息的第一部署请求之后，检测所述硬件资源是否创建成功；在所述硬件资源创建成功的情况下，根据用于所述数据集群***部署的组件的信息生成插件。
在一实施例中，所述生成模块，配置为获取第一数据集群***版本文件信息，其中，所述第一数据集群***版本文件信息为所部署的数据集群***的版本文件的信息；根据用于所述数据集群***部署的组件的信息和所述第一数据集群***版本文件信息对应的版本文件，生成所述插件。
在一实施例中，所述装置还包括运行模块，配置为在所述生成模块根据用于所述数据集群***部署的组件的信息生成插件之后，运行所述插件，以实现以下过程至少之一：网络环境配置；组件资源分配；时钟同步；将所述组件配置到所部署的数据集群***的目标节点。
在一实施例中，所述装置还包括监测模块，配置为监测所述数据集群***的运行环境，其中，所述运行环境包括资源占用状况，在监测到发生 资源占用状况异常的情况下，自动对所述数据集群***进行调优；和/或，监测所述数据集群***的运行状态，在发生数据集群***的运行状态异常的情况下，自动对所述数据集群***进行调优。
在一实施例中，所述装置还包括判断模块；
所述获取模块，还配置为接收用于所述数据集群***升级的第二配置文件；解析所述第二配置文件，得到与所述数据集群***升级相关的第二配置参数；
所述判断模块，配置为根据所述第二配置参数判断所述数据集群***升级是否需要增加硬件资源和/或配置；
所述发送模块，配置为在需要增加硬件资源和/或配置的情况下，向所述云平台发送携带有所述第二配置参数的第二部署请求，其中，所述第二部署请求用于请求所述云平台根据所述第二配置参数创建所述数据集群***升级所需的硬件资源和/或配置。
在一实施例中，所述装置还包括生成模块；所述第二配置文件中还包括用于所述数据集群***升级的组件的信息；
所述获取模块，还配置为在所述判断模块判定不需要增加硬件资源和/或配置的情况下，获取第二数据集群***版本文件信息，其中，所述第二数据集群***版本文件信息为升级后的数据集群***的版本文件的信息；
所述生成模块，配置为根据用于所述数据集群***升级的组件的信息和所述第二数据集群***版本文件信息对应的版本文件，生成插件。
根据本发明的第三方面，提供了一种数据集群的部署***，包括：处理器，配置为获取用于数据集群***部署的第一配置信息，向云平台发送携带有所述第一配置信息的第一部署请求，其中，所述第一配置信息包括部署数据集群***所需的硬件资源和/或配置的信息，所述第一部署请求用于请求云平台根据第一配置信息创建数据集群***部署所需的硬件资源和/ 或配置；云平台，配置为根据所述第一配置信息创建数据集群***部署所需的硬件资源和/或配置。
根据本发明实施例的第四方面，提供了一种计算机存储介质，所述计算机存储介质中存储有计算机可执行指令，所述计算机可执行指令用于执行本发明实施例所述的数据集群的部署方法。
本发明实施例通过获取用于数据集群***部署的第一配置信息，其中，第一配置信息包括部署数据集群***所需的硬件资源和/或配置的信息；向云平台发送携带有第一配置信息的第一部署请求，其中，第一部署请求用于请求云平台根据第一配置信息创建数据集群***部署所需的硬件资源和/或配置，实现了通过云平台批量创建硬件资源和/或配置，解决了数据集群的部署和管理较为复杂的问题，进而有效简化了数据集群的实际开发环境、测试环境及生产环境等方面的部署过程，能够大幅度节约设备和人力，提高了数据集群的部署效率。
附图说明
此处所说明的附图用来提供对本发明的进一步理解，构成本申请的一部分，本发明的示意性实施例及其说明用于解释本发明，并不构成对本发明的不当限定。在附图中：
图1是相关技术中Hadoop***的组件结构示意图；
图2是根据本发明实施例的数据集群的部署方法的流程图；
图3是根据本发明实施例的数据集群的部署装置的结构框图；
图4是根据本发明实施例的数据集群的部署***的示意图；
图5是根据本发明实施例的一种数据集群的部署***的示意图。
具体实施方式
下文中将参考附图并结合实施例来详细说明本发明。需要说明的是， 在不冲突的情况下，本申请中的实施例及实施例中的特征可以相互组合。
需要说明的是，本发明的说明书和权利要求书及上述附图中的术语“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。
在本实施例中提供了一种数据集群的部署方法，图2是根据本发明实施例的数据集群的部署方法的流程图，如图2所示，该流程包括如下步骤：
步骤S202，获取用于数据集群***部署的第一配置信息，其中，第一配置信息包括部署数据集群***所需的硬件资源和/或配置的信息；
步骤S204，向云平台发送携带有第一配置信息的第一部署请求，其中，第一部署请求用于请求云平台根据第一配置信息创建数据集群***部署所需的硬件资源和/或配置。
其中，云平台为利用云技术搭建起来的一个服务平台，可以提供基础设施或者虚拟化环境等服务。
其中，部署数据集群***所需的硬件资源包括***部署所需的基础设施。
其中，云平台可以是基于PAAS(Platform-as-a-Service，可简称为平台即服务)的云平台，基于PAAS的云平台能够为hadoop***提供足够的硬件支持。
在本实施中，第一配置信息可以是用户自定义的配置文件中的相关配置信息，例如，第一配置信息可以包括以下数据的至少之一：用户选定的虚拟机模板的相关数据、用户自定义的待部署的数据集群***中需要安装的组件、基础设施搭建相关的参数等。另外，可以预先将配置文件系列化标准化，在使用时用户只需要更改少量的参数就可以完成数据集群***的配置，并且该配置文件是用户唯一需要填写的部署清单，数据集群***的部署变得更加简单。
作为一种实施方式，本发明实施例的数据集群的部署方法适用于大数据集群的部署，例如，hadoop***的部署。
上述实施例，通过获取用于数据集群***部署的第一配置信息，其中，第一配置信息包括部署数据集群***所需的硬件资源和/或配置的信息；向云平台发送携带有第一配置信息的第一部署请求，其中，第一部署请求用于请求云平台根据第一配置信息创建数据集群***部署所需的硬件资源和/或配置，实现了通过云平台批量创建硬件资源和/或配置，解决了数据集群的部署和管理较为复杂的问题，进而有效简化了数据集群的实际开发环境、测试环境及生产环境等方面的部署过程，能够大幅度节约设备和人力，提高了数据集群的部署效率。
作为一种实施方式，所述获取用于数据集群***部署的第一配置信息包括：接收用户定义的第一配置文件；根据第一配置文件获取第一配置信息。其中，所述第一配置文件是用户自定义的(用户根据自身需求对标准配置文件进行参数的修改和/或选择，以获得第一配置文件)，配置方法更能满足用户的需求，并且部署操作更加便捷。
作为一种实施方式，所述根据第一配置文件获取第一配置信息包括：解析第一配置文件，获得与数据集群***部署相关的第一配置参数；根据第一配置参数计算第一配置信息。
在本实施例中，用户提供数据集群***部署需要的第一配置文件，处理器接收到该第一配置文件后，解析第一配置文件中的参数，自动计算出需要的硬件资源(基础设施)和配置情况，然后触发自动部署请求，向PAAS云平台发送创建硬件资源和/或配置的请求，所述请求中携带有计算得到的硬件资源和/或配置信息。该实施例在接收到用户的配置文件后，会自动触发一系列的解析、请求等操作，无需用户再投入精力，部署过程十分高效、便捷。
作为一种实施方式，所述第一配置文件中还包括用于数据集群***部署的组件的信息；所述在向云平台发送携带有第一配置信息的第一部署请求之后，还包括：检测硬件资源是否创建成功；在硬件资源创建成功的情况下，根据用于数据集群***部署的组件的信息生成插件。
在本实施例中，在硬件资源创建成功之后，可自动生成一个主控模块，在主控模块的控制下，可根据第一配置文件自动生成插件，并将插件配置并运行起来。
作为一种实施方式，所述根据用于数据集群***部署的组件的信息生成插件包括：获取第一数据集群***版本文件信息，其中，所述第一数据集群***版本文件信息为所部署的数据集群***的版本文件的信息；根据用于数据集群***部署的组件的信息和第一数据集群***版本文件信息对应的版本文件，生成插件。
其中，所述第一数据集群***版本文件信息可以是待部署的大数据集群***的版本安装包，可通过操作人员从版本机上获取并放至版本目录中。
例如，在hadoop***的部署中，在硬件资源创建成功之后，可自动生成一个主控模块，在主控模块的控制下，可以根据用户定义的配置文件(包括组件的参数)和hadoop版本文件自动生成插件。
作为一种实施方式，所述第一部署请求还用于请求从云平台中存储的多个数据集群***版本文件中选择需要的虚拟主机模板，以生成虚拟机。
在本实施例中，云平台可预先存储多个虚拟主机模板，用户根据需要在配置文件中选定需要的虚拟主机模板，并通过请求消息发送至云平台，云平台根据请求确定需要生成的虚拟机。
作为一种实施方式，所述插件为带有组件功能的插件，所述插件能够独立运行。
在本实施例中，通过远程命令自动生成包含组件功能的插件，该插件 可脱离操作***独立运行。其中，插件可以看作是组件的载体。大数据集群***可以看作是由一系列的组件构成的。带有组件功能的插件是指一个可随时部署的插件，该插件具有需要部署的组件的功能。例如，一个HIVE版本包的安装需要涉及诸多的配置，而该实施例通过生成具有组件功能的插件，仅仅需要配置该插件即可完成对组件的复杂配置过程，有效提高了组件的配置效率。该实施例有效克服了现有大数据集群***的部署中需要多个组件按照次序逐个部署而造成的效率低下的缺陷，有效提升了大数据集群***的部署效率。
作为一种实施方式，在根据用于数据集群***部署的组件的信息生成插件之后，还包括：运行插件，以实现以下过程至少之一：网络环境配置；组件资源分配；时钟同步；将所述组件配置到所部署的数据集群***的目标节点。
在本实施例中，可以通过批量配置工具自动配置所有的插件，以实现网络环境配置、组件资源分配、时钟同步、将所述组件配置到所部署的数据集群***的目标节点等操作。
作为一种实施方式，所述插件为多个插件，多个插件以层状结构部署。
在本实施例中，通过大数据插件层状化部署，可以支持插件的灵活使用，方便***的调整和扩容，以及自动进行性能调优。
作为一种实施方式，在运行插件之后，所述方法还包括：监测数据集群***的运行环境，其中，运行环境包括资源占用状况，在监测到发生资源占用状况异常的情况下，自动对数据集群***进行调优；和/或，监测数据集群***的运行状态，在发生数据集群***的运行状态异常的情况下，自动对数据集群***进行调优。
本实施例可以实现安全和监控的自动化，作为一种实施方式，利用Kerberos(Kerberos为一种安全认证的***)进行身份认证，自动化监控数 据集群***的相关组件的运行情况。环境若运行正常，和/或***正常运行，则持续监控；环境若发生资源占用异常，和/或***处于异常状态，则自动通知主控进程，主控进程自动完成对数据集群***的自动调优。
作为一种实施方式，所述方法还包括：接收用于数据集群***升级的第二配置文件；解析第二配置文件，得到与数据集群***升级相关的第二配置参数；根据第二配置参数判断数据集群***升级是否需要增加硬件资源和/或配置；在需要增加硬件资源和/或配置的情况下，向云平台发送携带有第二配置参数的第二部署请求，其中，第二部署请求用于请求云平台根据第二配置参数创建数据集群***升级所需的硬件资源和/或配置。
作为一种实施方式，所述第二配置文件中还包括用于数据集群***升级的组件的信息，在不需要增加硬件资源和/或配置的情况下，还包括：获取第二数据集群***版本文件信息，其中，所述第二数据集群***版本文件信息为升级后的数据集群***的版本文件的信息；根据用于数据集群***升级的组件的信息和第二数据集群***版本文件信息对应的版本文件，生成插件。
其中，所述第二数据集群***版本文件信息可以是待升级到的大数据集群***的版本包，可通过操作人员从版本机上获取并放至版本目录中。
上述实施例提供了数据集群***需要升级时的配置方法。具体地，用户提供数据集群***升级需要的配置文件(第二配置文件)，处理器接收到配置文件后，解析配置文件中的参数，自动计算出升级是否需要增加硬件资源(基础设施)和配置情况。若需要，则触发自动部署请求，向PAAS平台发送创建基础设施的请求，请求携带有计算出的配置信息，在基础设施创建成功后，自动通知主控模块，以通过主控模块控制相应的插件生成；若不需要，则处理器(处理器中的插件自动生成模块)通过主控模块，并基于配置文件和新的数据集群***版本文件(第二数据集群***版本文件 信息对应的版本文件)自动生成插件；处理器(处理器中的批量配置模块)按照配置文件把待升级插件停止并替换成新的插件然后配置并运行起来。
在上述过程中，处理器同样可利用安全和监控自动化功能通过Kerberos进行身份认证，自动化监控数据集群***相关组件的运行情况。其中，环境若运行正常，和/或数据集群***正常运行，持续监控；环境若发生资源占用异常和/或***出现异常状态，可自动通知主控进程，主控进程自动完成对数据集群***的自动调优。
通过上述实施例，在数据集群***需要升级时，无需停止数据集群***的运行即可实现***的升级，从而实现了无缝升级，提高了用户体验。
上述实施提出的数据集群的部署方法，可通过组件的插件化进行大数据环境的部署，扩展可更新的组件插件仓库，通过PAAS云平台进行存储和管理，通过自动化部署脚本和管理脚本进行部署和管理，并通过自动收集***信息，对大数据集群的健康情况进行监控，从而达到了自动部署和管理大数据环境的效果。另外，大数据插件层状化部署，可以支持插件的灵活使用，方便调整和扩容，支持自动执行性能调优。此外，该方法易于对应用进行自动打包和部署、创建轻量PAAS环境等优点，在实际开发/测试环境部署或者生产环境部署方面能够大幅度地节约设备和人力资源，提高部署效率，同时该方法还实现了大数据平台的无缝升级。
下面，根据两种具体实施方式来进一步说明本发明提供的数据集群的部署方法。
实施例一
下面根据本发明实施例提供了一种数据集群的部署方法，该方法包括以下的步骤：
步骤11：用户提供hadoop***部署需要的配置文件；其中，所述配置文件相当于上述的第一配置文件。
步骤12：处理器接收到配置文件后，解析配置文件中参数，自动计算出需要的基础设施和配置情况，然后触发自动部署请求，向PaaS平台发送创建基础设施和/或配置的请求，请求中携带计算得到的配置信息；其中，所述请求相当于上述的第一部署请求。
步骤13：基础设施创建成功后，自动生成hadoop主控模块。
步骤14：插件自动生成模块在主控模块的控制下，根据标准配置文件和hadoop版本文件(第一数据集群版本文件信息对应的版本文件)自动生成组件插件。
步骤15：批量配置模块按照配置文件把插件配置并运行起来。
步骤16：安全和监控模块自动利用Kerberos进行身份认证，自动化监控hadoop相关组件的运行情况
步骤17：环境若运行正常，和/或***正常运行，则持续监控。
步骤18：环境若发生资源占用异常或者***出现异常状况，则自动通知主控进程，主控进程自动完成对hadoop***的自动调优。
实施例二
下面根据本发明实施例提供了另外一种数据集群的部署方法，该方法应用于大数据***的升级，该方法包括以下的步骤：
步骤21：用户提供hadoop***升级需要的配置文件。
步骤22：处理器接收到配置文件后，解析配置文件中的参数，自动计算出升级是否需要增加基础设施和/或配置，若需要，则触发执行步骤23；若不需要，则触发执行步骤25。
步骤23：触发自动部署请求，向PaaS平台发送创建基础设施请求(所述基础设施请求相当于上述的第二部署请求)，所述请求携带有计算得到的配置信息。
步骤24：基础设施创建成功后，自动通知hadoop主控模块。
步骤25：插件自动生成模块在主控模块的控制下，根据配置文件和新的hadoop版本文件(第二数据集群***版本文件信息对应的版本文件)自动生成组件插件。
步骤26：批量配置模块按照配置文件把待升级插件停止并替换成新的插件，并将新的插件配置并运行起来。
步骤27：安全和监控模块自动利用Kerberos进行身份认证，自动化监控Hadoop相关组件的运行情况。
步骤28：环境若运行正常，和/或***正常运行，则持续监控。
步骤29：环境若发生资源占用异常或者***出现异常状况，则自动通知主控进程，主控进程自动完成对hadoop***的自动调优。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到根据上述实施例的方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本发明的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台终端设备(可以是手机，计算机，服务器，或者网络设备等)执行本发明各个实施例所述的方法。
在本实施例中还提供了一种数据集群的部署装置，该装置用于实现上述实施例及优选实施方式，已经进行过说明的不再赘述。如以下所使用的，术语“模块”可以实现预定功能的软件和/或硬件的组合。尽管以下实施例所描述的装置较佳地以软件来实现，但是硬件，或者软件和硬件的组合的实现也是可能并被构想的。
图3是根据本发明实施例的数据集群的部署装置的结构框图，如图3所示，该装置包括：
获取模块30，配置为获取用于数据集群***部署的第一配置信息，其中，所述第一配置信息包括部署数据集群***所需的硬件资源和/或配置的信息；
发送模块32，配置为向云平台发送携带有第一配置信息的第一部署请求，其中，所述第一部署请求用于请求云平台根据第一配置信息创建数据集群***部署所需的硬件资源和/或配置。
在本实施例中，通过所述获取模块30获取用于数据集群***部署的第一配置信息，其中，所述第一配置信息包括部署数据集群***所需的硬件资源和/或配置的信息；所述发送模块32向云平台发送携带有第一配置信息的第一部署请求，其中，所述第一部署请求用于请求云平台根据第一配置信息创建数据集群***部署所需的硬件资源和/或配置，实现了通过云平台批量创建硬件资源和/或配置，解决了数据集群的部署和管理较为复杂的问题，进而有效简化了数据集群的实际开发环境、测试环境及生产环境等方面的部署过程，能够大幅度节约设备和人力，提高了数据集群的部署效率。
作为一种实施方式，所述获取模块30，配置为接收用户定义的第一配置文件；根据所述第一配置文件获取所述第一配置信息。
作为一种实施方式，所述获取模块30，配置为解析所述第一配置文件，获得与所述数据集群***部署相关的第一配置参数；根据所述第一配置参数计算所述第一配置信息。
作为一种实施方式，所述第一配置文件中还包括用于所述数据集群***部署的组件的信息；所述装置还包括生成模块，配置为在所述发送模块向云平台发送携带有所述第一配置信息的第一部署请求之后，检测所述硬件资源是否创建成功；在所述硬件资源创建成功的情况下，根据用于所述数据集群***部署的组件的信息生成插件。
作为一种实施方式，所述生成模块，配置为获取第一数据集群***版 本文件信息，其中，所述第一数据集群***版本文件信息为所部署的数据集群***的版本文件的信息；根据用于所述数据集群***部署的组件的信息和所述第一数据集群***版本文件信息对应的版本文件，生成所述插件。
作为一种实施方式，所述装置还包括运行模块，配置为在所述生成模块根据用于所述数据集群***部署的组件的信息生成插件之后，运行所述插件，以实现以下过程至少之一：网络环境配置；组件资源分配；时钟同步；将所述组件配置到所部署的数据集群***的目标节点。
作为一种实施方式，所述装置还包括监测模块，配置为监测所述数据集群***的运行环境，其中，所述运行环境包括资源占用状况，在监测到发生资源占用状况异常的情况下，自动对所述数据集群***进行调优；和/或，监测所述数据集群***的运行状态，在发生数据集群***的运行状态异常的情况下，自动对所述数据集群***进行调优。
作为一种实施方式，所述装置还包括判断模块；
所述获取模块30，还配置为接收用于所述数据集群***升级的第二配置文件；解析所述第二配置文件，得到与所述数据集群***升级相关的第二配置参数；
所述判断模块，配置为根据所述第二配置参数判断所述数据集群***升级是否需要增加硬件资源和/或配置；
所述发送模块32，配置为在需要增加硬件资源和/或配置的情况下，向所述云平台发送携带有所述第二配置参数的第二部署请求，其中，所述第二部署请求用于请求所述云平台根据所述第二配置参数创建所述数据集群***升级所需的硬件资源和/或配置。
作为一种实施方式，所述装置还包括生成模块；所述第二配置文件中还包括用于所述数据集群***升级的组件的信息；
所述获取模块30，还配置为在所述判断模块判定不需要增加硬件资源 和/或配置的情况下，获取第二数据集群***版本文件信息，其中，所述第二数据集群***版本文件信息为升级后的数据集群***的版本文件的信息；
所述生成模块，配置为根据用于所述数据集群***升级的组件的信息和所述第二数据集群***版本文件信息对应的版本文件，生成插件。
需要说明的是，上述各个模块是可以通过软件或硬件来实现的：上述模块均位于同一处理器中；或者，上述模块分别位于多个处理器中。
在本实施例中还提供了一种数据集群的部署***，该***用于实现上述实施例及优选实施方式，已经进行过说明的不再赘述。
图4是根据本发明实施例的数据集群的部署***的示意图，如图4所示，所述***包括：
处理器40，配置为获取用于数据集群***部署的第一配置信息，向云平台发送携带有第一配置信息的第一部署请求，其中，第一配置信息包括部署数据集群***所需的硬件资源和/或配置的信息，第一部署请求用于请求云平台根据第一配置信息创建数据集群***部署所需的硬件资源和/或配置；
云平台42，配置为根据第一配置信息创建数据集群***部署所需的硬件资源和/或配置。
在本实施例中，通过处理器40获取用于数据集群***部署的第一配置信息，并向云平台发送携带有第一配置信息的第一部署请求，其中，所述第一配置信息包括部署数据集群***所需的硬件资源和/或配置的信息，第一部署请求用于请求云平台根据第一配置信息创建数据集群***部署所需的硬件资源和/或配置，所述云平台42根据第一配置信息创建数据集群***部署所需的硬件资源和/或配置，实现了通过云平台批量创建硬件资源和/或配置，解决了数据集群的部署和管理较为复杂的问题，进而有效简化了 数据集群的实际开发环境、测试环境及生产环境等方面的部署过程，能够大幅度节约设备和人力，提高了数据集群的部署效率。
图5是根据本发明实施例的一种数据集群的部署***的示意图。其中，在该数据集群的部署***中，上述的处理器40还可以包括：主控模块502、插件生成模块504、标准配置模块506、批量配置模块508以及安全和监控模块510。各模块的功能具体如下：
所述主控模块502：该模块配置为将插件组装并运行起来，以及具有将数据集群***版本文件组装成可以不依赖于硬件和操作***独立运行的插件的功能。例如，可以将大数据的组件，如YARN、HBASE、zookeeper、HIVE、Impala、MapReduce、Oozie、Sqoop、Flume等，封装成单独的插件，使得插件可以脱离操作***和硬件独立部署，具有完全的隔离性，并且可以复用硬件和操作***的资源。
所述插件生成模块504：配置为利用容器技术和集装箱技术，通过主控模块，按照定制的功能将数据集群***版本文件组装成插件，以供部署和使用。
所述标准配置模块506(可实现数据集群的部署装置中获取模块和发送模块的功能)：配置为将数据集群***配置文件系列化、标准化，在使用时只需更改少量的参数即可完成数据集群***的配置。其中，配置文件是用户唯一需要填写的部署清单。标准配置模块可实现配置文件的标准化，通过提取数据集群的可配置项，形成标准配置文件，简化了用户操作，提高了用户体验。
所述批量配置模块508：配置为使用批量配置工具，根据配置文件(如上述的第一配置文件和第二配置文件)，自动配置所有的插件，用于支持包括网络环境构建、组件资源分配、时钟同步以及应用配置到节点等操作，用于为数据集群***组件提供统一的配置环境。
所述安全和监控模块510：配置为利用Kerberos(hadoop自带的Kerberos身份认证***)进行身份认证。该模块可自动化监控数据集群***相关组件的运行情况，并根据资源占用情况完成对数据集群***的调优。
所述云平台42用于为数据集群***提供足够的硬件支持。在云计算平台(云平台)中存储常用操作***的虚拟主机模版，云平台可通过用户选定的虚拟机模版以及配置参数为用户搭建数据集群所需的主控计算机，然后再通过配置文件，确定需要安装的组件，确认无误后，处理器40会自动生成需要的插件，并将其部署为层状结构，搭建好大数据集群。
根据该实施例的数据集群的部署***，实现了基于插件化的数据集群***的部署，并且支持自动化部署和组件分配，将插件以层状插件的形式部署在集群中，可支持灵活使用，方便调整和扩容，可实现自动性能调优。该数据集群的部署***通过云平台实现了批量构造硬件资源和/或配置，解决了现有技术中数据集群的部署和管理较为复杂的问题。
本发明的实施例还提供了一种存储介质。作为一种实施方式，在本实施例中，上述存储介质可以被设置为存储用于执行以下步骤的程序代码：
S202，获取用于数据集群***部署的第一配置信息，其中，第一配置信息包括部署数据集群***所需的硬件资源和/或配置的信息；
S204，向云平台发送携带有第一配置信息的第一部署请求，其中，第一部署请求用于请求云平台根据第一配置信息创建数据集群***部署所需的硬件资源和/或配置。
作为一种实施方式，在本实施例中，上述存储介质可以包括但不限于：U盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、移动硬盘、磁碟或者光盘等各种可以存储程序代码的介质。
作为一种实施方式，本实施例中的具体示例可以参考上述实施例及可 选实施方式中所描述的示例，本实施例在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的设备和方法，可以通过其它的方式实现。以上所描述的设备实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，如：多个单元或组件可以结合，或可以集成到另一个***，或一些特征可以忽略，或不执行。另外，所显示或讨论的各组成部分相互之间的耦合、或直接耦合、或通信连接可以是通过一些接口，设备或单元的间接耦合或通信连接，可以是电性的、机械的或其它形式的。
上述作为分离部件说明的单元可以是、或也可以不是物理上分开的，作为单元显示的部件可以是、或也可以不是物理单元，即可以位于一个地方，也可以分布到多个网络单元上；可以根据实际的需要选择其中的部分或全部单元来实现本实施例方案的目的。
另外，在本发明各实施例中的各功能单元可以全部集成在一个处理单元中，也可以是各单元分别单独作为一个单元，也可以两个或两个以上单元集成在一个单元中；上述集成的单元既可以采用硬件的形式实现，也可以采用硬件加软件功能单元的形式实现。
本领域普通技术人员可以理解：实现上述方法实施例的全部或部分步骤可以通过程序指令相关的硬件来完成，前述的程序可以存储于一计算机可读取存储介质中，该程序在执行时，执行包括上述方法实施例的步骤；而前述的存储介质包括：移动存储设备、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质。
或者，本发明上述集成的单元如果以软件功能模块的形式实现并作为独立的产品销售或使用时，也可以存储在一个计算机可读取存储介质中。基于这样的理解，本发明实施例的技术方案本质上或者说对现有技术做出 贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机、服务器、或者网络设备等)执行本发明各个实施例所述方法的全部或部分。而前述的存储介质包括：移动存储设备、ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本发明的具体实施方式，但本发明的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本发明揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本发明的保护范围之内。因此，本发明的保护范围应以所述权利要求的保护范围为准。
工业实用性
本发明实施例的技术方案通过获取用于数据集群***部署的第一配置信息，其中，第一配置信息包括部署数据集群***所需的硬件资源和/或配置的信息；向云平台发送携带有第一配置信息的第一部署请求，其中，第一部署请求用于请求云平台根据第一配置信息创建数据集群***部署所需的硬件资源和/或配置，实现了通过云平台批量创建硬件资源和/或配置，解决了数据集群的部署和管理较为复杂的问题，进而有效简化了数据集群的实际开发环境、测试环境及生产环境等方面的部署过程，能够大幅度节约设备和人力，提高了数据集群的部署效率。

Claims (22)

  1. 一种数据集群的部署方法，包括：
    获取用于数据集群***部署的第一配置信息，其中，所述第一配置信息包括部署所述数据集群***所需的硬件资源和/或配置的信息；
    向云平台发送携带有所述第一配置信息的第一部署请求，其中，所述第一部署请求用于请求所述云平台根据所述第一配置信息创建所述数据集群***部署所需的硬件资源和/或配置。
  2. 根据权利要求1所述的方法，其中，所述获取用于数据集群***部署的第一配置信息包括：
    接收用户定义的第一配置文件；
    根据所述第一配置文件获取所述第一配置信息。
  3. 根据权利要求2所述的方法，其中，所述根据所述第一配置文件获取所述第一配置信息包括：
    解析所述第一配置文件，获得与所述数据集群***部署相关的第一配置参数；
    根据所述第一配置参数计算所述第一配置信息。
  4. 根据权利要求2所述的方法，其中，所述第一配置文件中还包括用于所述数据集群***部署的组件的信息；所述在向云平台发送携带有所述第一配置信息的第一部署请求之后，所述方法还包括：
    检测所述硬件资源是否创建成功；
    在所述硬件资源创建成功的情况下，根据用于所述数据集群***部署的组件的信息生成插件。
  5. 根据权利要求4所述的方法，其中，所述根据用于所述数据集群***部署的组件的信息生成插件包括：
    获取第一数据集群***版本文件信息，其中，所述第一数据集群*** 版本文件信息为所部署的数据集群***的版本文件的信息；
    根据用于所述数据集群***部署的组件的信息和所述第一数据集群***版本文件信息对应的版本文件，生成所述插件。
  6. 根据权利要求4-5中任一项所述的方法，其中，所述插件为具备组件功能的插件。
  7. 根据权利要求4-5中任一项所述的方法，其中，所述插件为多个插件，所述多个插件以层状结构部署。
  8. 根据权利要求4所述的方法，其中，在根据用于所述数据集群***部署的组件的信息生成插件之后，所述方法还包括：
    运行所述插件，以实现以下过程至少之一：
    网络环境配置；组件资源分配；时钟同步；将所述组件配置到所部署的数据集群***的目标节点。
  9. 根据权利要求8所述的方法，其中，在运行所述插件之后，所述方法还包括：
    监测所述数据集群***的运行环境，其中，所述运行环境包括资源占用状况，在监测到发生资源占用状况异常的情况下，自动对所述数据集群***进行调优；和/或，
    监测所述数据集群***的运行状态，在发生数据集群***的运行状态异常的情况下，自动对所述数据集群***进行调优。
  10. 根据权利要求1所述的方法，其中，所述方法还包括：
    接收用于所述数据集群***升级的第二配置文件；
    解析所述第二配置文件，得到与所述数据集群***升级相关的第二配置参数；
    根据所述第二配置参数判断所述数据集群***升级是否需要增加硬件资源和/或配置；
    在需要增加硬件资源和/或配置的情况下，向所述云平台发送携带有所述第二配置参数的第二部署请求，其中，所述第二部署请求用于请求所述云平台根据所述第二配置参数创建所述数据集群***升级所需的硬件资源和/或配置。
  11. 根据权利要求10所述的方法，其中，所述第二配置文件中还包括用于所述数据集群***升级的组件的信息；在不需要增加硬件资源和/或配置的情况下，所述方法还包括：
    获取第二数据集群***版本文件信息，其中，所述第二数据集群***版本文件信息为升级后的数据集群***的版本文件的信息；
    根据用于所述数据集群***升级的组件的信息和所述第二数据集群***版本文件信息对应的版本文件，生成插件。
  12. 一种数据集群的部署装置，包括：
    获取模块，配置为获取用于数据集群***部署的第一配置信息，其中，所述第一配置信息包括部署所述数据集群***所需的硬件资源和/或配置的信息；
    发送模块，配置为向云平台发送携带有所述第一配置信息的第一部署请求，其中，所述第一部署请求用于请求所述云平台根据所述第一配置信息创建所述数据集群***部署所需的硬件资源和/或配置。
  13. 根据权利要求12所述的装置，其中，所述获取模块，配置为接收用户定义的第一配置文件；根据所述第一配置文件获取所述第一配置信息。
  14. 根据权利要求13所述的装置，其中，所述获取模块，配置为解析所述第一配置文件，获得与所述数据集群***部署相关的第一配置参数；根据所述第一配置参数计算所述第一配置信息。
  15. 根据权利要求13所述的装置，其中，所述第一配置文件中还包括用于所述数据集群***部署的组件的信息；所述装置还包括生成模块，配 置为在所述发送模块向云平台发送携带有所述第一配置信息的第一部署请求之后，检测所述硬件资源是否创建成功；在所述硬件资源创建成功的情况下，根据用于所述数据集群***部署的组件的信息生成插件。
  16. 根据权利要求15所述的装置，其中，所述生成模块，配置为获取第一数据集群***版本文件信息，其中，所述第一数据集群***版本文件信息为所部署的数据集群***的版本文件的信息；根据用于所述数据集群***部署的组件的信息和所述第一数据集群***版本文件信息对应的版本文件，生成所述插件。
  17. 根据权利要求15所述的装置，其中，所述装置还包括运行模块，配置为在所述生成模块根据用于所述数据集群***部署的组件的信息生成插件之后，运行所述插件，以实现以下过程至少之一：网络环境配置；组件资源分配；时钟同步；将所述组件配置到所部署的数据集群***的目标节点。
  18. 根据权利要求17所述的装置，其中，所述装置还包括监测模块，配置为监测所述数据集群***的运行环境，其中，所述运行环境包括资源占用状况，在监测到发生资源占用状况异常的情况下，自动对所述数据集群***进行调优；和/或，监测所述数据集群***的运行状态，在发生数据集群***的运行状态异常的情况下，自动对所述数据集群***进行调优。
  19. 根据权利要求12所述的装置，其中，所述装置还包括判断模块；
    所述获取模块，还配置为接收用于所述数据集群***升级的第二配置文件；解析所述第二配置文件，得到与所述数据集群***升级相关的第二配置参数；
    所述判断模块，配置为根据所述第二配置参数判断所述数据集群***升级是否需要增加硬件资源和/或配置；
    所述发送模块，配置为在需要增加硬件资源和/或配置的情况下，向所 述云平台发送携带有所述第二配置参数的第二部署请求，其中，所述第二部署请求用于请求所述云平台根据所述第二配置参数创建所述数据集群***升级所需的硬件资源和/或配置。
  20. 根据权利要求19所述的装置，其中，所述装置还包括生成模块；所述第二配置文件中还包括用于所述数据集群***升级的组件的信息；
    所述获取模块，还配置为在所述判断模块判定不需要增加硬件资源和/或配置的情况下，获取第二数据集群***版本文件信息，其中，所述第二数据集群***版本文件信息为升级后的数据集群***的版本文件的信息；
    所述生成模块，配置为根据用于所述数据集群***升级的组件的信息和所述第二数据集群***版本文件信息对应的版本文件，生成插件。
  21. 一种数据集群的部署***，包括：
    处理器，配置为获取用于数据集群***部署的第一配置信息，向云平台发送携带有所述第一配置信息的第一部署请求，其中，所述第一配置信息包括部署所述数据集群***所需的硬件资源和/或配置的信息，所述第一部署请求用于请求所述云平台根据所述第一配置信息创建所述数据集群***部署所需的硬件资源和/或配置；
    所述云平台，配置为根据所述第一配置信息创建所述数据集群***部署所需的硬件资源和/或配置。
  22. 一种计算机存储介质，所述计算机存储介质中存储有计算机可执行指令，所述计算机可执行指令用于执行权利要求1至11任一项所述的数据集群的部署方法。
PCT/CN2017/074073 2016-03-24 2017-02-20 数据集群的部署方法、装置、***及计算机存储介质 WO2017161984A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610177977.X 2016-03-24
CN201610177977.XA CN107229646A (zh) 2016-03-24 2016-03-24 数据集群的部署方法、装置及***

Publications (1)

Publication Number Publication Date
WO2017161984A1 true WO2017161984A1 (zh) 2017-09-28

Family

ID=59899194

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/074073 WO2017161984A1 (zh) 2016-03-24 2017-02-20 数据集群的部署方法、装置、***及计算机存储介质

Country Status (2)

Country Link
CN (1) CN107229646A (zh)
WO (1) WO2017161984A1 (zh)

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110311831A (zh) * 2019-06-14 2019-10-08 平安科技（深圳）有限公司 基于容器云的***资源监控方法及相关设备
CN110389766A (zh) * 2019-06-21 2019-10-29 深圳市汇川技术股份有限公司 HBase容器集群部署方法、***、设备及计算机可读存储介质
CN111597536A (zh) * 2020-05-19 2020-08-28 重庆第二师范学院 一种hadoop集群kerberos高可用认证方法
CN112052095A (zh) * 2020-09-11 2020-12-08 成都深思科技有限公司 一种分布式高可用的大数据挖掘任务调度***
CN112068935A (zh) * 2020-09-15 2020-12-11 北京值得买科技股份有限公司 kubernetes程序部署监控方法、装置以及设备
CN112395072A (zh) * 2019-08-14 2021-02-23 北京三快在线科技有限公司 模型部署的方法、装置、存储介质及电子设备
CN112749085A (zh) * 2020-12-31 2021-05-04 北京知因智慧科技有限公司 一种跨集群的任务发布方法、装置、电子设备及存储介质
CN113312153A (zh) * 2021-06-25 2021-08-27 深信服科技股份有限公司 一种集群部署方法、装置、电子设备及存储介质
CN113419746A (zh) * 2021-05-21 2021-09-21 济南浪潮数据技术有限公司 一种Cinder-CSI插件安装方法、***、存储介质及设备
CN113448686A (zh) * 2021-06-22 2021-09-28 深信服科技股份有限公司 一种资源部署方法、装置、电子设备及存储介质
CN113448837A (zh) * 2020-11-03 2021-09-28 北京新氧科技有限公司 开发及测试环境部署方法、***、电子设备及介质
CN114116066A (zh) * 2021-11-30 2022-03-01 湖南麒麟信安科技股份有限公司 一种云平台集群控制方法、***及存储介质
CN115297096A (zh) * 2022-06-28 2022-11-04 青岛海尔科技有限公司 资源包的部署方法、装置、存储介质及电子装置

Families Citing this family (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101998564B1 (ko) * 2018-07-19 2019-07-10 나무기술 주식회사 클라우드 플랫폼에서의 멀티 클러스터 프로비저닝 및 관리 방법
CN110210924B (zh) * 2019-04-15 2024-03-26 腾讯科技（深圳）有限公司 产品功能定制方法、装置以及服务器
CN110532022B (zh) * 2019-08-07 2022-06-17 浪潮电子信息产业股份有限公司 一种获取不同云平台版本的方法、装置、设备和存储介质
CN113296823A (zh) * 2020-02-21 2021-08-24 北京国双科技有限公司 一种生成资源配置文件的方法、装置、设备及存储介质
CN111625270A (zh) * 2020-05-21 2020-09-04 浪潮电子信息产业股份有限公司 一种开发环境创建方法、装置、设备及存储介质
CN112346816A (zh) * 2020-09-01 2021-02-09 北京京东尚科信息技术有限公司 云平台更新方法和装置
CN112306626B (zh) * 2020-09-01 2024-04-05 北京京东尚科信息技术有限公司 用于更新云平台的方法和装置
CN112738206B (zh) * 2020-12-25 2022-08-23 北京浪潮数据技术有限公司 一种分布式集群和云平台的对接方法及组件
CN113608832A (zh) * 2021-07-17 2021-11-05 济南浪潮数据技术有限公司 一种应用部署方法、***、设备以及介质
CN113645314B (zh) * 2021-08-31 2023-04-07 新华三技术有限公司成都分公司 一种私有云的部署方法和服务器
CN113839821B (zh) * 2021-10-14 2024-05-24 京东科技信息技术有限公司 部署集群和构建基础设施的方法、装置、***、设备及介质
CN114095498B (zh) * 2021-10-22 2022-08-19 广西中科曙光云计算有限公司 集群环境的部署方法、***、计算机设备及存储介质
CN116489025B (zh) * 2023-04-26 2024-01-26 国网河北省电力有限公司 一种电网调控云控制平台及其维护方法
CN116866180B (zh) * 2023-07-04 2024-03-01 北京志凌海纳科技有限公司 集群升级的测试方法及***

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102394939A (zh) * 2011-11-21 2012-03-28 苏州阔地网络科技有限公司 一种分布式***的部署方法及***
CN102455915A (zh) * 2010-10-18 2012-05-16 卓望数码技术（深圳）有限公司 一种分布式***的自动化配置方法及服务器
CN104750780A (zh) * 2015-03-04 2015-07-01 北京航空航天大学 一种基于统计分析的Hadoop配置参数优化方法
CN104915407A (zh) * 2015-06-03 2015-09-16 华中科技大学 一种基于Hadoop多作业环境下的资源调度方法
CN105260203A (zh) * 2015-09-25 2016-01-20 福州大学 一种基于模型的Hadoop部署以及配置方法

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103283209B (zh) * 2011-04-18 2015-12-09 北京新媒传信科技有限公司 一种应用服务平台***及其实现方法
CN103209197B (zh) * 2012-01-12 2016-08-03 百度在线网络技术（北京）有限公司 集群服务器部署方法及***
CN103167041B (zh) * 2013-03-28 2016-04-20 广州中国科学院软件应用技术研究所 一种支持云环境应用集群自动化部署的***及方法
CN104601622B (zh) * 2013-10-31 2018-04-17 国际商业机器公司 一种部署集群的方法和***
CN104836819A (zh) * 2014-02-10 2015-08-12 阿里巴巴集团控股有限公司 动态负载均衡的方法、***及监控调度设备
US9596143B2 (en) * 2014-07-25 2017-03-14 Cohesity, Inc. Node discovery and cluster formation for a secondary storage appliance
CN104579761B (zh) * 2014-12-24 2018-03-23 西安工程大学 一种基于云计算的nosql集群自动配置***及自动配置方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102455915A (zh) * 2010-10-18 2012-05-16 卓望数码技术（深圳）有限公司 一种分布式***的自动化配置方法及服务器
CN102394939A (zh) * 2011-11-21 2012-03-28 苏州阔地网络科技有限公司 一种分布式***的部署方法及***
CN104750780A (zh) * 2015-03-04 2015-07-01 北京航空航天大学 一种基于统计分析的Hadoop配置参数优化方法
CN104915407A (zh) * 2015-06-03 2015-09-16 华中科技大学 一种基于Hadoop多作业环境下的资源调度方法
CN105260203A (zh) * 2015-09-25 2016-01-20 福州大学 一种基于模型的Hadoop部署以及配置方法

Cited By (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110311831A (zh) * 2019-06-14 2019-10-08 平安科技（深圳）有限公司 基于容器云的***资源监控方法及相关设备
CN110311831B (zh) * 2019-06-14 2022-03-25 平安科技（深圳）有限公司 基于容器云的***资源监控方法及相关设备
CN110389766A (zh) * 2019-06-21 2019-10-29 深圳市汇川技术股份有限公司 HBase容器集群部署方法、***、设备及计算机可读存储介质
CN110389766B (zh) * 2019-06-21 2022-12-27 深圳市汇川技术股份有限公司 HBase容器集群部署方法、***、设备及计算机可读存储介质
CN112395072A (zh) * 2019-08-14 2021-02-23 北京三快在线科技有限公司 模型部署的方法、装置、存储介质及电子设备
CN111597536A (zh) * 2020-05-19 2020-08-28 重庆第二师范学院 一种hadoop集群kerberos高可用认证方法
CN112052095B (zh) * 2020-09-11 2024-04-19 成都锋卫科技有限公司 一种分布式高可用的大数据挖掘任务调度***
CN112052095A (zh) * 2020-09-11 2020-12-08 成都深思科技有限公司 一种分布式高可用的大数据挖掘任务调度***
CN112068935A (zh) * 2020-09-15 2020-12-11 北京值得买科技股份有限公司 kubernetes程序部署监控方法、装置以及设备
CN113448837A (zh) * 2020-11-03 2021-09-28 北京新氧科技有限公司 开发及测试环境部署方法、***、电子设备及介质
CN113448837B (zh) * 2020-11-03 2024-03-19 北京新氧科技有限公司 开发及测试环境部署方法、***、电子设备及介质
CN112749085A (zh) * 2020-12-31 2021-05-04 北京知因智慧科技有限公司 一种跨集群的任务发布方法、装置、电子设备及存储介质
CN112749085B (zh) * 2020-12-31 2023-10-24 北京知因智慧科技有限公司 一种跨集群的任务发布方法、装置、电子设备及存储介质
CN113419746A (zh) * 2021-05-21 2021-09-21 济南浪潮数据技术有限公司 一种Cinder-CSI插件安装方法、***、存储介质及设备
CN113419746B (zh) * 2021-05-21 2022-11-08 济南浪潮数据技术有限公司 一种Cinder-CSI插件安装方法、***、存储介质及设备
CN113448686A (zh) * 2021-06-22 2021-09-28 深信服科技股份有限公司 一种资源部署方法、装置、电子设备及存储介质
CN113448686B (zh) * 2021-06-22 2024-05-24 深信服科技股份有限公司 一种资源部署方法、装置、电子设备及存储介质
CN113312153A (zh) * 2021-06-25 2021-08-27 深信服科技股份有限公司 一种集群部署方法、装置、电子设备及存储介质
CN114116066A (zh) * 2021-11-30 2022-03-01 湖南麒麟信安科技股份有限公司 一种云平台集群控制方法、***及存储介质
CN114116066B (zh) * 2021-11-30 2024-05-07 湖南麒麟信安科技股份有限公司 一种云平台集群控制方法、***及存储介质
CN115297096A (zh) * 2022-06-28 2022-11-04 青岛海尔科技有限公司 资源包的部署方法、装置、存储介质及电子装置

Also Published As

Publication number Publication date
CN107229646A (zh) 2017-10-03

Similar Documents

Publication Publication Date Title
WO2017161984A1 (zh) 数据集群的部署方法、装置、***及计算机存储介质
CN110389900B (zh) 一种分布式数据库集群测试方法、装置及存储介质
CN108809722B (zh) 一种部署Kubernetes集群的方法、装置和存储介质
CN106533753B (zh) 一种分布式***的角色配置方法及装置
CN107463582B (zh) 分布式部署Hadoop集群的方法及装置
US9419856B1 (en) Network device configuration deployment pipeline
US20210263714A1 (en) System and method for autowiring of a microservice architecture
CN107800565B (zh) 巡检方法、装置、***、计算机设备和存储介质
CN110752947A (zh) 一种k8s集群部署方法及装置，一种部署平台
CN112214330A (zh) 集群中主节点的部署方法、装置及计算机可读存储介质
CN106776998B (zh) 一种数据库服务提供方法及服务器
CN105099739A (zh) 一种基于插件式软件部署方法、装置及应用服务器
WO2017194990A1 (en) Orchestrator for a virtual network platform as a service (vnpaas)
CN105429839A (zh) 虚拟化网络功能vnf优化方法、装置及***
CN112968789B (zh) 数据采集方法、装置、计算机设备及计算机可读介质
US9959157B1 (en) Computing instance migration
CN113778486B (zh) 一种代码流水线的容器化处理方法、装置、介质及设备
CN112860282B (zh) 集群插件的升级方法、装置和服务器
CN109684038B (zh) Docker服务容器日志的处理方法、装置和电子设备
KR20150108230A (ko) 클러스터 시스템 구축 방법 및 장치
CN112905337A (zh) 软硬件混合部署的MySQL集群调度方法及装置
US9832137B1 (en) Provisioning system and method for a distributed computing environment using a map reduce process
CN113900670B (zh) 集群服务器应用部署***
EP4024761A1 (en) Communication method and apparatus for multiple management domains
CN115632944B (zh) 一种节点配置方法、装置、设备、可读存储介质及服务器

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17769261

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 17769261

Country of ref document: EP

Kind code of ref document: A1