CN116032745A - Automatic configuration method and device of hadoop cluster - Google Patents
Automatic configuration method and device of hadoop cluster Download PDFInfo
- Publication number
- CN116032745A CN116032745A CN202310092781.0A CN202310092781A CN116032745A CN 116032745 A CN116032745 A CN 116032745A CN 202310092781 A CN202310092781 A CN 202310092781A CN 116032745 A CN116032745 A CN 116032745A
- Authority
- CN
- China
- Prior art keywords
- configuration
- information
- hadoop
- automatic
- web page
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 54
- 238000013515 script Methods 0.000 claims abstract description 44
- 238000012423 maintenance Methods 0.000 claims abstract description 32
- 238000012360 testing method Methods 0.000 claims abstract description 25
- BUGBHKTXTAQXES-UHFFFAOYSA-N Selenium Chemical compound [Se] BUGBHKTXTAQXES-UHFFFAOYSA-N 0.000 claims description 23
- 229910052711 selenium Inorganic materials 0.000 claims description 23
- 239000011669 selenium Substances 0.000 claims description 23
- 238000004590 computer program Methods 0.000 claims description 20
- 238000010586 diagram Methods 0.000 description 10
- 230000015654 memory Effects 0.000 description 10
- 238000004891 communication Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 6
- 230000000007 visual effect Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 4
- 238000009434 installation Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000007405 data analysis Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 239000008186 active pharmaceutical agent Substances 0.000 description 1
- 230000000712 assembly Effects 0.000 description 1
- 238000000429 assembly Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000000758 substrate Substances 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Landscapes
- Stored Programmes (AREA)
Abstract
The disclosure provides an automatic configuration method of hadoop clusters, which can be applied to the technical field of Yu Yun dimensions. The method comprises the following steps: responding to the hadoop cluster deployment instruction, and generating a configuration file according to the cluster landing planning information, the configuration key value pair information and the operation and maintenance tool access address information; uploading the configuration file and the automation test tool script package to a master node; and running the automatic test tool script package, and simulating the configuration operation of the web page according to the configuration file so as to complete the automatic configuration of the hadoop cluster component. The disclosure also provides an automatic configuration device, equipment, a storage medium and a program product of the hadoop cluster.
Description
Technical Field
The disclosure relates to the technical field of cloud computing, in particular to the technical field of operation and maintenance, and especially relates to an automatic configuration method, device, equipment, storage medium and program product of a hadoop cluster.
Background
With the vigorous development of technologies such as the internet of things, cloud computing and cloud storage, big data analysis begins to be integrated into various industries, and Hadoop is widely applied to various fields as an infrastructure of big data as a general big data storage and analysis platform recognized in the industry. Because the Hadoop has a plurality of components and more than thousands of configurations, in order to facilitate the installation configuration and maintenance of the Hadoop platform, the original version of the Hadoop is not directly used, but Cloudera or a self-grinding operation and maintenance suite is selected, the Hadoop platform is installed by using the visual WebUI unified configuration, and then platform interfaces are provided for all service parties for use.
In the related art, there are two main ways to build the Hadoop platform, one is pure script installation, all binary and configuration are prepared in advance, and then related configuration files are distributed and configured by using scripts. However, the mode does not have a unified interface for checking the starting and running conditions of the assembly; if the script executes error reporting, it cannot visually display which machines are installed successfully or fail, i.e. the script is not suitable for large-scale cluster deployment; meanwhile, as the component operation logs are scattered in all physical machines of the cluster, bug investigation is difficult. The second method is to use Cloudera manager to install, namely, web visual UI to install, an installer only needs to manually select components according to steps, then add nodes, modify configuration and save, and then the whole cluster can be managed, and the configuration can be automatically rolled and synchronized to all node configurations after saving. However, due to complex and numerous component configuration, hundreds of configuration items of hadoop are required to be configured, and deployment failure is caused by manual operation errors, so that subsequent service development is affected.
It should be noted that the information disclosed in the above background section is only for enhancing understanding of the background of the present disclosure and thus may include information that does not constitute prior art known to those of ordinary skill in the art.
Disclosure of Invention
In view of the foregoing, the present disclosure provides a hadoop cluster automatic configuration method, apparatus, device, storage medium and program product for improving hadoop deployment efficiency.
According to a first aspect of the present disclosure, there is provided an automatic configuration method of hadoop clusters, including:
responding to the hadoop cluster deployment instruction, and generating a configuration file according to the cluster landing planning information, the configuration key value pair information and the operation and maintenance tool access address information;
uploading the configuration file and the automation test tool script package to a master node; and
and running the automatic test tool script package, and simulating configuration operation of the web page according to the configuration file to complete automatic configuration of the hadoop cluster component.
According to an embodiment of the present disclosure, the generating a configuration file according to the cluster landing planning information, the configuration key pair information and the operation and maintenance tool access address information includes:
determining host list information according to the cluster planning information and the network planning information;
determining service list information of each host according to the component role planning information; and
and generating a configuration file according to the host list information, the service list information, the operation and maintenance tool access address information and the configuration key value pair information.
According to an embodiment of the present disclosure, the simulating the configuration operation of the web page according to the configuration file to complete the automatic configuration of the hadoop cluster component includes:
accessing a web page of the operation and maintenance tool according to the operation and maintenance tool access address information;
and simulating the configuration operation of the web page according to the configuration key value pair information to complete the automatic configuration of the hadoop cluster component.
According to an embodiment of the present disclosure, the simulating the configuration operation of the web page according to the configuration key value pair information to complete the automatic configuration of the hadoop cluster component includes:
determining configuration operation of configuration items in the web page according to the configuration key value pair information;
and circularly executing the configuration operation until the configuration of the hadoop cluster component is completed.
According to an embodiment of the present disclosure, the configuration key value pair information includes a key field, a key value, and a key value type, and the determining a configuration operation of a configuration item in a web page according to the configuration key value pair information includes:
determining the position information of the configuration items in the web page according to the key fields; and
and determining configuration operation information of the web page according to the key value type, wherein the configuration operation information comprises clicking operation, input operation and pull-down selection operation.
According to an embodiment of the present disclosure, the automated test tool script package is a selenium script.
A second aspect of the present disclosure provides an automatic configuration apparatus for hadoop clusters, including: the acquisition module is used for responding to the hadoop cluster deployment instruction, and generating a configuration file according to the cluster landing planning information, the configuration key value pair information and the operation and maintenance tool access address information;
the uploading module is used for uploading the configuration file and the automatic test tool script package to the master node; and
and the configuration module is used for running the automatic test tool script package, simulating the configuration operation of the web page according to the configuration file, and completing the automatic configuration of the hadoop cluster component.
A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method for automatically configuring hadoop clusters described above.
A fourth aspect of the present disclosure also provides a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the above-described method of auto-configuration of hadoop clusters.
A fifth aspect of the present disclosure also provides a computer program product comprising a computer program which, when executed by a processor, implements the method for automatically configuring hadoop clusters described above.
According to the automatic configuration method of the hadoop cluster, a configuration file is generated according to cluster landing planning information, configuration key value pair information and operation and maintenance tool access address information by responding to a hadoop cluster deployment instruction; uploading the configuration file and the automation test tool script package to a master node; and running an automatic test tool script package, and simulating configuration operation of the web page, thereby realizing automatic configuration of the hadoop cluster component. By running the automatic test tool script, the page position of each configuration item is determined according to the pre-generated configuration file, the operation of filling the configuration value is completed by using the Selenium PI, the automatic configuration of the hadoop cluster assembly is realized, and compared with the manual operation configuration scheme of operation and maintenance personnel in the related technology, the hadoop cluster configuration deployment efficiency is greatly improved, and the workload of the operation and maintenance personnel is reduced.
Drawings
The foregoing and other objects, features and advantages of the disclosure will be more apparent from the following description of embodiments of the disclosure with reference to the accompanying drawings, in which:
FIG. 1 schematically illustrates an application scenario diagram of a method, apparatus, device, storage medium, and program product for automatic configuration of hadoop clusters according to an embodiment of the present disclosure;
FIG. 2 schematically illustrates a flow chart of a method of automatic configuration of hadoop clusters according to an embodiment of the disclosure;
FIG. 3 schematically illustrates a flow chart of a method of generating a configuration file according to an embodiment of the disclosure;
FIG. 4 schematically illustrates a flow chart of a hadoop cluster component auto-configuration method provided in accordance with an embodiment of the present disclosure;
FIG. 5 schematically illustrates a block diagram of an automatic configuration apparatus of a hadoop cluster according to an embodiment of the present disclosure;
fig. 6 schematically illustrates a block diagram of an electronic device adapted to implement an automatic configuration method of hadoop clusters according to an embodiment of the disclosure.
Detailed Description
Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is only exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the concepts of the present disclosure.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and/or the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.
All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein should be construed to have meanings consistent with the context of the present specification and should not be construed in an idealized or overly formal manner.
Where expressions like at least one of "A, B and C, etc. are used, the expressions should generally be interpreted in accordance with the meaning as commonly understood by those skilled in the art (e.g.," a system having at least one of A, B and C "shall include, but not be limited to, a system having a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).
The terms appearing in the embodiments of the present disclosure will first be explained:
selenium: the Selenium is a browser remote control interface tool, and can translate operation codes written by a developer aiming at a browser (such as clicking on an object) into actual actions of a user to operate the browser.
Cloudera: a visual platform for building, operating and managing Hadoop is provided.
Hadoop: the big data assembly set forms a Hadoop platform and generally comprises Hdfs, hive, zookeeper, kafka common assemblies.
Based on the above technical problems, an embodiment of the present disclosure provides an automatic configuration method of a hadoop cluster, including: responding to the hadoop cluster deployment instruction, and generating a configuration file according to the cluster landing planning information, the configuration key value pair information and the operation and maintenance tool access address information; uploading the configuration file and the automation test tool script package to a master node; and running the automatic test tool script package, and simulating the configuration operation of the web page according to the configuration file so as to complete the automatic configuration of the hadoop cluster component.
Fig. 1 schematically illustrates an application scenario diagram of a hadoop cluster auto-configuration method, apparatus, device, storage medium and program product according to an embodiment of the present disclosure.
As shown in fig. 1, an application scenario 100 according to this embodiment may include a hadoop cluster deployment scenario. The network 104 is used as a medium to provide communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
The user may interact with the server 105 via the network 104 using the terminal devices 101, 102, 103 to receive or send messages or the like. Various communication client applications, such as shopping class applications, web browser applications, search class applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only) may be installed on the terminal devices 101, 102, 103.
The terminal devices 101, 102, 103 may be a variety of electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.
The server 105 may be a server providing various services, such as providing an automatic configuration service of a cluster (by way of example only) to a user using a cluster deployment instruction issued by the terminal devices 101, 102, 103. Server 105 may run a Selenium script to automatically deploy and configure the components of the hadoop cluster based on the cluster drop plan information, configuration key pair information, and configuration files generated by the operation and maintenance tool access address information.
It should be noted that, the method for automatically configuring hadoop clusters provided by the embodiments of the present disclosure may be generally performed by the server 105. Accordingly, the hadoop cluster auto-configuration device provided in the embodiments of the present disclosure may be generally disposed in the server 105. The method for automatically configuring hadoop clusters provided by the embodiments of the present disclosure may also be performed by a server or a server cluster that is different from the server 105 and is capable of communicating with the terminal devices 101, 102, 103 and/or the server 105. Accordingly, the hadoop cluster auto-configuration apparatus provided by the embodiments of the present disclosure may also be provided in a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and/or the server 105.
It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
The automatic configuration method of hadoop clusters of the disclosed embodiments will be described in detail below with reference to the scenario described in fig. 1 through fig. 2 to 6.
Fig. 2 schematically illustrates a flowchart of a method for automatic configuration of hadoop clusters according to an embodiment of the disclosure.
As shown in fig. 2, the hadoop cluster automatic configuration method of this embodiment includes operations S210 to S230, and the hadoop cluster automatic configuration method may be executed by a server or other computing devices.
In operation S210, in response to the hadoop cluster deployment instruction, a configuration file is generated according to the cluster landing planning information, the configuration key pair information, and the operation and maintenance tool access address information.
In operation S220, the configuration file and the automated test tool script package are uploaded to the master node.
According to an embodiment of the present disclosure, the automated test tool script package is a selenium script.
In operation S230, the automated test tool script package is run, and the configuration operation of the web page is simulated according to the configuration file, so as to complete the automatic configuration of the hadoop cluster component.
In one example, cloudera is used as a visual platform for operation and maintenance management hadoop, an operation and maintenance person uses Cloudera manager Web visual UI to install, manually selects components according to certain steps, then adds nodes, modifies configuration and stores, and manages the whole cluster. When hadoop integrates a plurality of big data components, for example, hdfs, hive, zookeeper, kafka common components, more than hundreds of configuration items are needed to be manually configured, and due to the complex and numerous configuration items, the components needed by different clusters are different, and the corresponding configurations are also different. Therefore, depending on manual configuration of operation and maintenance personnel, configuration operation errors are easy to cause, and deployment failure is caused.
In the embodiment of the disclosure, the automated testing tool is Selenium, or may be a similar tool for automatic testing of a Web application, taking Selenium as an example, the operation manually configured by an operation and maintenance person is replaced by a Selenium script, where the Selenium script uses a browser API to simulate complex Web operations such as user access, clicking, input, and selection. The Selenium script can read an external configuration script (comprising configuration information of various components of Hadoop), and then uses the Selenium to simulate clicking on a WebUI interface of ClouderaManager to modify the configuration of hundreds of key components, so that the whole deployment and configuration of Hadoop are automatically completed.
When a hadoop cluster deployment instruction is received, a configuration file and a Selenium script package are acquired, wherein the configuration file is pre-precipitated by an operation and maintenance personnel according to configuration items of all components of hadoop, and a specific generation process of the configuration file can refer to operation 310 shown in fig. 3 and 4, which is not repeated herein. Uploading the configuration file and the Selenium script package acquired in the operation S210 and the hadoop publishing package to a main node of a cluster to be deployed, and performing machine initialization and environment initialization operations, such as JDK installation, ip configuration, hosts configuration, SSH encryption-free communication and the like, before the Selenium script package is operated for configuration, after the Cloudera is built, operating the Selenium script package for automatic configuration of the hadoop cluster assembly, and circularly simulating configuration items of a user operation related set by combining configuration items. The specific configuration process may be referred to as operation S231 and operation S232 shown in fig. 5. After the deployment is completed, the deployment and configuration results can be verified through the running log.
According to the hadoop cluster configuration method based on the Selenium script, the Selenium script is operated, page positions of configuration items are determined according to the configuration files generated in advance, the Selenium PI is utilized to finish operation of filling configuration values, automatic configuration of hadoop cluster components is achieved, and compared with the scheme of manual operation configuration of operation and maintenance personnel in the related art, the hadoop cluster configuration deployment efficiency can be improved, and the workload of the operation and maintenance personnel is reduced.
The method for generating a configuration file in an embodiment of the present disclosure will be described below with reference to fig. 3. Fig. 3 schematically illustrates a flowchart of a method of generating a configuration file according to an embodiment of the present disclosure. As shown in fig. 3, operation S210 includes operations S211 to S213.
In operation S211, host list information is determined according to the cluster planning information and the network planning information.
Service list information of each host is determined according to the component role plan information in operation S212.
In operation S213, a configuration file is generated according to the host list information, the service list information, the operation and maintenance tool access address information, and configuration key value pair information.
In one example, the configuration file (named cdh_config.yml) generated according to operations S211 to S213 is illustrated as follows, in the configuration file, hive, hdfs, zookeeper are individual Hadoop services that need complex configuration, each Hadoop cluster may run many services such as hive, hdfs, zookeeper, etc., and each service needs to specify a batch of machines such as bigdata01, bigdata02 to take different roles in the services.
And determining a host list of the current deployment cluster according to the cluster planning information and the network planning information, specifically determining a host set forming the cluster according to the cluster planning, determining an IP address of each host in the cluster through the network planning, and configuring a corresponding router or switch.
Component role planning refers to the fact that each host may run one or more services, where it takes on different roles, which may be understood as different modules of a service, such as a producer module, a consumer module, together serving a message queue service. The operation and maintenance tool accesses address information, namely the url website of the ClouderManager and the user name and password of the ClouderManager. The configuration key value pair information is information of configuration items in each configuration page, the configuration keys represent the configuration items, and the configuration values represent specific configuration parameters. The following is shown: hive: -key: "Hive Metastore database name"; val: "hive"; type: "text", the value characterizing a key as "Hive Metastere database name" is "Hive", and the type is input. hdfs: -key: "use dustbin"; val: "true"; type: "checkbox", the value that characterizes a key as "use garbage" is "true", and the type is selected. zookeeper: -key: "Server maximum Log size"; val: "50 megabytes"; type: "text, list".
The process of running an automated test tool script package to automatically configure hadoop cluster components is described below in connection with fig. 4. Fig. 4 schematically illustrates a flowchart of a hadoop cluster component auto-configuration method provided according to an embodiment of the present disclosure. As shown in fig. 4, operation S230 includes operations S231 to S232.
In operation S231, a web page of the operation tool is accessed according to the operation tool access address information.
In operation S232, the configuration operation of the web page is simulated according to the configuration key value pair information, so as to complete the automatic configuration of the hadoop cluster component.
According to the embodiment of the disclosure, determining configuration operation of configuration items in the web page according to the configuration key value pair information; and circularly executing the configuration operation until the configuration of the hadoop cluster component is completed.
According to an embodiment of the present disclosure, the configuration key value pair information includes a key field, a key value, and a key value type, and the determining a configuration operation of a configuration item in a web page according to the configuration key value pair information includes: determining the position information of the configuration items in the web page according to the key fields; and determining configuration operation information of the web page according to the key value type, wherein the configuration operation information comprises clicking operation, input operation and pull-down selection operation.
In one example, the configuration key pair information includes a key field, a key value, and a key value type. The key field is a configuration item, the key value is a configuration parameter corresponding to the configuration item, and the key value type corresponds to some types of configuration items in the configuration page, such as "whether a" checkbox "is selected", "text, list", "input box" or "drop-down selection box", and the like. First, the web page of the operation tool ClouderManager is accessed according to the operation tool access address information. Determining configuration operations of the web page, such as selection, drop-down selection, input and the like, according to the configuration key value pair information, specifically determining position information of configuration items in the web page according to the key field, namely positioning to specific configuration items; and determining configuration operation information of the web page according to the key value type, and then utilizing the Selenium to simulate clicking on a WebUI interface of the Cloudera manager to modify the configuration of hundreds of key components, thereby automatically completing the whole deployment and configuration of Hadoop.
After deployment is completed, the running result of the Selenium script can be judged by combining the log of the running result of the script and the log of each component in the Cloudera manager, and the running success or failure can be judged.
According to the hadoop cluster configuration method provided by the embodiment of the disclosure, when a plurality of clusters are required to be configured, as configuration key values in configuration files are always fixed, the configuration files are only required to be finely tuned according to specific services, so that the Selenium script can be operated to perform automatic configuration, the configuration efficiency is greatly improved, and meanwhile, the configuration results are controllable, so that the hadoop cluster configuration method is suitable for application scenes of large-scale cluster deployment.
Based on the automatic configuration method of the hadoop cluster, the disclosure also provides an automatic configuration device of the hadoop cluster. The device will be described in detail below in connection with fig. 5.
Fig. 5 schematically illustrates a block diagram of an automatic configuration apparatus of hadoop clusters according to an embodiment of the present disclosure.
As shown in fig. 5, the hadoop cluster automatic configuration apparatus 700 of this embodiment includes an acquisition module 710, an upload module 720, and a configuration module 730.
The obtaining module 710 is configured to generate a configuration file according to the cluster landing planning information, the configuration key pair information, and the operation and maintenance tool access address information in response to the hadoop cluster deployment instruction. In an embodiment, the obtaining module 710 may be configured to perform the operation S210 described above, which is not described herein.
The upload module 720 is configured to upload the configuration file and the automation test tool script package to the master node. In an embodiment, the uploading module 720 may be configured to perform the operation S220 described above, which is not described herein.
The configuration module 730 is configured to run the automated test tool script package, and simulate the configuration operation of the web page according to the configuration file, so as to complete the automatic configuration of the hadoop cluster component. In an embodiment, the configuration module 730 may be configured to perform the operation S230 described above, which is not described herein.
Any of the acquisition module 710, the upload module 720, and the configuration module 730 may be combined in one module to be implemented, or any of them may be split into a plurality of modules, according to an embodiment of the present disclosure. Alternatively, at least some of the functionality of one or more of the modules may be combined with at least some of the functionality of other modules and implemented in one module. According to embodiments of the present disclosure, at least one of the acquisition module 710, the upload module 720, and the configuration module 730 may be implemented at least in part as hardware circuitry, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system-on-chip, a system-on-substrate, a system-on-package, an Application Specific Integrated Circuit (ASIC), or in hardware or firmware in any other reasonable manner of integrating or packaging the circuitry, or in any one of or a suitable combination of any of the three. Alternatively, at least one of the acquisition module 710, the upload module 720 and the configuration module 730 may be at least partially implemented as a computer program module which, when executed, may perform the respective functions.
Fig. 6 schematically illustrates a block diagram of an electronic device adapted to implement an automatic configuration method of hadoop clusters according to an embodiment of the disclosure.
As shown in fig. 6, an electronic device 900 according to an embodiment of the present disclosure includes a processor 901 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 902 or a program loaded from a storage portion 908 into a Random Access Memory (RAM) 903. The processor 901 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or an associated chipset and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), or the like. Processor 901 may also include on-board memory for caching purposes. Processor 901 may include a single processing unit or multiple processing units for performing the different actions of the method flows according to embodiments of the present disclosure.
In the RAM 903, various programs and data necessary for the operation of the electronic device 900 are stored. The processor 901, the ROM 902, and the RAM 903 are connected to each other by a bus 904. The processor 901 performs various operations of the method flow according to the embodiments of the present disclosure by executing programs in the ROM 902 and/or the RAM 903. Note that the program may be stored in one or more memories other than the ROM 902 and the RAM 903. The processor 901 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.
According to an embodiment of the disclosure, the electronic device 900 may also include an input/output (I/O) interface 905, the input/output (I/O) interface 905 also being connected to the bus 904. The electronic device 900 may also include one or more of the following components connected to the I/O interface 905: an input section 906 including a keyboard, a mouse, and the like; an output portion 907 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and a speaker; a storage portion 908 including a hard disk or the like; and a communication section 909 including a network interface card such as a LAN card, a modem, or the like. The communication section 909 performs communication processing via a network such as the internet. The drive 910 is also connected to the I/O interface 905 as needed. A removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is installed as needed on the drive 910 so that a computer program read out therefrom is installed into the storage section 908 as needed.
The present disclosure also provides a computer-readable storage medium that may be embodied in the apparatus/device/system described in the above embodiments; or may exist alone without being assembled into the apparatus/device/system. The computer-readable storage medium carries one or more programs which, when executed, implement methods in accordance with embodiments of the present disclosure.
According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example, but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, the computer-readable storage medium may include ROM 902 and/or RAM 903 and/or one or more memories other than ROM 902 and RAM 903 described above.
Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowcharts. When the computer program product runs in a computer system, the program code is used for enabling the computer system to realize the hadoop cluster automatic configuration method provided by the embodiment of the disclosure.
The above-described functions defined in the system/apparatus of the embodiments of the present disclosure are performed when the computer program is executed by the processor 901. The systems, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.
In one embodiment, the computer program may be based on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted, distributed, and downloaded and installed in the form of a signal on a network medium, via communication portion 909, and/or installed from removable medium 911. The computer program may include program code that may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.
In such an embodiment, the computer program may be downloaded and installed from the network via the communication portion 909 and/or installed from the removable medium 911. The above-described functions defined in the system of the embodiments of the present disclosure are performed when the computer program is executed by the processor 901. The systems, devices, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.
According to embodiments of the present disclosure, program code for performing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, such computer programs may be implemented in high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. Programming languages include, but are not limited to, such as Java, c++, python, "C" or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
Those skilled in the art will appreciate that the features recited in the various embodiments of the disclosure and/or in the claims may be provided in a variety of combinations and/or combinations, even if such combinations or combinations are not explicitly recited in the disclosure. In particular, the features recited in the various embodiments of the present disclosure and/or the claims may be variously combined and/or combined without departing from the spirit and teachings of the present disclosure. All such combinations and/or combinations fall within the scope of the present disclosure.
The embodiments of the present disclosure are described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the embodiments cannot be used advantageously in combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be made by those skilled in the art without departing from the scope of the disclosure, and such alternatives and modifications are intended to fall within the scope of the disclosure.
Claims (10)
1. An automatic configuration method of hadoop clusters is characterized by comprising the following steps:
responding to the hadoop cluster deployment instruction, and generating a configuration file according to the cluster landing planning information, the configuration key value pair information and the operation and maintenance tool access address information;
uploading the configuration file and the automation test tool script package to a master node; and
and running the automatic test tool script package, and simulating configuration operation of the web page according to the configuration file to complete automatic configuration of the hadoop cluster component.
2. The method of claim 1, wherein generating the configuration file from the cluster drop plan information, the configuration key pair information, and the operation and maintenance tool access address information comprises:
determining host list information according to the cluster planning information and the network planning information;
determining service list information of each host according to the component role planning information; and
and generating a configuration file according to the host list information, the service list information, the operation and maintenance tool access address information and the configuration key value pair information.
3. The method of claim 1, wherein simulating the configuration operation of the web page according to the configuration file to complete the automatic configuration of the hadoop cluster component comprises:
accessing a web page of the operation and maintenance tool according to the operation and maintenance tool access address information;
and simulating the configuration operation of the web page according to the configuration key value pair information to complete the automatic configuration of the hadoop cluster component.
4. A method according to claim 3, wherein simulating the configuration operation of the web page according to the configuration key value to complete the automatic configuration of the hadoop cluster component comprises:
determining configuration operation of configuration items in the web page according to the configuration key value pair information;
and circularly executing the configuration operation until the configuration of the hadoop cluster component is completed.
5. The method of claim 4, wherein the configuration key pair information includes a key field, a key value, and a key value type, and wherein determining a configuration operation for a configuration item in a web page based on the configuration key pair information comprises:
determining the position information of the configuration items in the web page according to the key fields; and
and determining configuration operation information of the web page according to the key value type, wherein the configuration operation information comprises clicking operation, input operation and pull-down selection operation.
6. The method of any one of claims 1 to 5, wherein the automated test tool script package is a selenium script.
7. An automatic configuration device of hadoop cluster, which is characterized by comprising:
the acquisition module is used for responding to the hadoop cluster deployment instruction, and generating a configuration file according to the cluster landing planning information, the configuration key value pair information and the operation and maintenance tool access address information;
the uploading module is used for uploading the configuration file and the automatic test tool script package to the master node; and
and the configuration module is used for running the automatic test tool script package, simulating the configuration operation of the web page according to the configuration file, and completing the automatic configuration of the hadoop cluster component.
8. An electronic device, comprising:
one or more processors;
storage means for storing one or more programs,
wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any of claims 1-6.
9. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform the method according to any of claims 1-6.
10. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310092781.0A CN116032745A (en) | 2023-01-31 | 2023-01-31 | Automatic configuration method and device of hadoop cluster |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310092781.0A CN116032745A (en) | 2023-01-31 | 2023-01-31 | Automatic configuration method and device of hadoop cluster |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116032745A true CN116032745A (en) | 2023-04-28 |
Family
ID=86077686
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310092781.0A Pending CN116032745A (en) | 2023-01-31 | 2023-01-31 | Automatic configuration method and device of hadoop cluster |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116032745A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100306286A1 (en) * | 2009-03-05 | 2010-12-02 | Chi-Hsien Chiu | Distributed steam processing |
| US9426030B1 (en) * | 2015-12-11 | 2016-08-23 | International Business Machines Coporation | Automatically generating configuration images and deploying computer components in a computing environment that comprises a shared pool of configurable computing resources |
| CN107273527A (en) * | 2017-06-27 | 2017-10-20 | 北京云和时空科技有限公司 | A kind of Hadoop clusters and distributed system |
| CN108536829A (en) * | 2018-04-11 | 2018-09-14 | 中国中医科学院中药研究所 | A method of it improving aerial survey of unmanned aerial vehicle data and generates tile map efficiency |
| CN115408016A (en) * | 2022-09-01 | 2022-11-29 | 上海承蓝数据科技有限公司 | Computing cluster deployment method and device, electronic equipment and storage medium |
| CN115550164A (en) * | 2022-08-26 | 2022-12-30 | 陕西天诚软件有限公司 | Method and system for one-key installation and management of single-node big data assembly |
-
2023
- 2023-01-31 CN CN202310092781.0A patent/CN116032745A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100306286A1 (en) * | 2009-03-05 | 2010-12-02 | Chi-Hsien Chiu | Distributed steam processing |
| US9426030B1 (en) * | 2015-12-11 | 2016-08-23 | International Business Machines Coporation | Automatically generating configuration images and deploying computer components in a computing environment that comprises a shared pool of configurable computing resources |
| CN107273527A (en) * | 2017-06-27 | 2017-10-20 | 北京云和时空科技有限公司 | A kind of Hadoop clusters and distributed system |
| CN108536829A (en) * | 2018-04-11 | 2018-09-14 | 中国中医科学院中药研究所 | A method of it improving aerial survey of unmanned aerial vehicle data and generates tile map efficiency |
| CN115550164A (en) * | 2022-08-26 | 2022-12-30 | 陕西天诚软件有限公司 | Method and system for one-key installation and management of single-node big data assembly |
| CN115408016A (en) * | 2022-09-01 | 2022-11-29 | 上海承蓝数据科技有限公司 | Computing cluster deployment method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR102493449B1 (en) | Edge computing test methods, devices, electronic devices and computer-readable media | |
| US11593084B2 (en) | Code development for deployment on a cloud platform | |
| US10007513B2 (en) | Edge intelligence platform, and internet of things sensor streams system | |
| US20180300124A1 (en) | Edge Computing Platform | |
| US20190065241A1 (en) | Orchestration service for multi-step recipe composition with flexible, topology-aware, and massive parallel execution | |
| US11561889B2 (en) | Orchestration for automated performance testing | |
| US10977167B2 (en) | Application monitoring with a decoupled monitoring tool | |
| US11474892B2 (en) | Graph-based log sequence anomaly detection and problem diagnosis | |
| CN107608901B (en) | Jmeter-based testing method and device, storage medium and electronic equipment | |
| US10608953B2 (en) | Platform with multiple execution engines | |
| US20140123114A1 (en) | Framework for integration and execution standardization (fiesta) | |
| CN109901985B (en) | Distributed test apparatus and method, storage medium, and electronic device | |
| US9507683B2 (en) | Monitoring service in a distributed platform | |
| CN110647332A (en) | Software deployment method and device based on container cloud | |
| CN113778486B (en) | Method, device, medium and equipment for containerization processing of code pipeline | |
| US11055205B1 (en) | Regression testing using automation technologies | |
| Rattanapoka et al. | An MQTT-based IoT cloud platform with flow design by Node-RED | |
| CN113014445A (en) | Operation and maintenance method, device and platform for server and electronic equipment | |
| CN113127009A (en) | Automatic deployment method and device for big data management platform | |
| US20190205182A1 (en) | Unified monitoring interface | |
| US20230032516A1 (en) | Common platform for implementing rpa services on customer premises | |
| CN116032745A (en) | Automatic configuration method and device of hadoop cluster | |
| CN112559001B (en) | Method and device for updating application | |
| CN112114833A (en) | Device and method for determining middleware installation and deployment | |
| CN112783757A (en) | Test system, test method, computer system, and computer-readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |