CN115913929A

CN115913929A - Kafka cluster deployment method and device, storage medium and electronic equipment

Info

Publication number: CN115913929A
Application number: CN202211313026.2A
Authority: CN
Inventors: 李昕鹏
Original assignee: Qingdao Haier Technology Co Ltd; Haier Smart Home Co Ltd; Haier Uplus Intelligent Technology Beijing Co Ltd
Current assignee: Qingdao Haier Technology Co Ltd; Haier Smart Home Co Ltd; Haier Uplus Intelligent Technology Beijing Co Ltd
Priority date: 2022-10-25
Filing date: 2022-10-25
Publication date: 2023-04-04

Abstract

The application discloses a Kafka cluster deployment method, a storage medium and electronic equipment, and relates to the technical field of intelligent configuration, wherein the Kafka cluster deployment method comprises the following steps: acquiring the deployment configuration information of the Kafka cluster input by a user; determining a connection configuration selection result of the Zookeeper cluster; based on the connection configuration selection result, acquiring the deployment configuration information of the Zookeeper cluster, which is input by the user and used for providing the coordination service for the Kafka cluster; and generating a corresponding template file based on the deployment configuration information, and executing the deployment operation of the Kafka cluster and the Zookeeper cluster according to the deployment task in the template file based on a preset configuration management tool. The method provided by the application can effectively improve the deployment efficiency of the Kafka cluster, saves the labor cost, and reduces the error rate, thereby improving the use experience of the user.

Description

Kafka cluster deployment method and device, storage medium and electronic equipment

Technical Field

The application relates to the technical field of intelligent configuration, in particular to a Kafka cluster deployment method and device, a storage medium and electronic equipment.

Background

Kafka is a high throughput distributed publish-subscribe messaging system. The Kafka cluster includes one or more server nodes, called brokers, each message queue consisting of a plurality of partitions, distributed with their copies across multiple brokers, whose partition acting as a Leader transfers the identity of the Leader to other brokers when one Broker stops. Deploying Kafka clusters requires separate configuration for each Broker, and often a larger number of brokers, and also relies on external Zookeeper clusters. However, kafka is a distributed architecture design, and may be a cluster composed of one or more services, and needs to be managed by a Zookeeper cluster, which needs to be composed of at least three nodes. Therefore, the Zookeeper cluster and the Kafka cluster need to be configured at the same time when a set of Kafka cluster is deployed, because the roles of each server are different, command operations to be executed are different, the efficiency of manually executing the cluster deployment one by the server nodes is low, the specific deployment and configuration process is laborious and error is easy to make.

Therefore, how to provide a more effective deployment scheme of the Kafka cluster to improve the deployment configuration efficiency of the Kafka cluster becomes a difficult problem to be solved urgently.

Disclosure of Invention

The application provides a Kafka cluster deployment method, which is used for solving the defects that in the prior art, the deployment configuration process of the Kafka cluster is low in efficiency and high in error rate.

The application provides a Kafka cluster deployment method, which comprises the following steps:

acquiring Kafka cluster deployment configuration information input by a user;

determining a connection configuration selection result of the Zookeeper cluster; acquiring deployment configuration information of the Zookeeper cluster, which is input by a user and used for providing coordination service for the Kafka cluster, based on the connection configuration selection result;

and generating a corresponding template file based on the deployment configuration information, and executing the deployment operation of the Kafka cluster and the Zookeeper cluster according to the deployment task in the template file based on a preset configuration management tool.

Further, the acquiring of the deployment configuration information of the Kafka cluster input by the user specifically includes:

based on an interactive inquiry mode, acquiring deployment configuration information of the Kafka cluster, which is input by a user based on a first inquiry configuration item;

the obtaining, based on the connection configuration selection result, deployment configuration information of the Zookeeper cluster that is input by the user and used for providing the coordination service for the Kafka cluster specifically includes: and if the connection configuration selection result is that the new Zookeeper cluster is selected to be deployed and used, acquiring deployment configuration information of the Zookeeper cluster input by the user based on the second query configuration item based on an interactive query mode.

Further, the acquiring, based on the interactive query mode, the deployment configuration information of the Kafka cluster input by the user based on the first query configuration item specifically includes:

based on an interactive query mode, acquiring the node number of the Kafka cluster input by a user based on a first node number query configuration item, acquiring the deployment position of the Kafka cluster input by the user based on a first deployment position query configuration item, acquiring the data storage position of the Kafka cluster input by the user based on a first data storage position query configuration item and acquiring the monitoring port number of the Kafka cluster input by the user based on a first port number query configuration item;

the acquiring, based on the interactive query mode, deployment configuration information of the Zookeeper cluster input by the user based on the second query configuration item specifically includes:

based on the interactive inquiry mode, acquiring the node number of the Zookeeper cluster input by the user based on the second node number inquiry configuration item, acquiring the deployment position of the Zookeeper cluster input by the user based on the second deployment position inquiry configuration item, acquiring the data storage position of the Zookeeper cluster input by the user based on the second data storage position inquiry configuration item and acquiring the monitoring port number of the Zookeeper cluster input by the user based on the second port number inquiry configuration item.

Further, the generating a corresponding template file based on the deployment configuration information specifically includes: and generating a corresponding Playbook template file based on the deployment configuration information of the Kafka cluster, the deployment configuration information of the Zookeeper cluster and a preset standard template format.

Further, the executing, by the preset-based configuration management tool, the deployment operations of the Kafka cluster and the Zookeeper cluster according to the deployment task in the template file specifically includes: and respectively executing the deployment operation of the Kafka cluster and the Zookeeper cluster according to the deployment task in the Playbook template file based on an Angle tool.

Further, before executing the deployment operation of the Kafka cluster and the Zookeeper cluster according to the deployment task in the template file based on a preset configuration management tool, the method further includes: and respectively establishing authentication connection between the server nodes of the Kafka cluster and the server nodes of the Zookeeper cluster based on the asymmetric encryption key.

Further, after executing the deployment operations of the Kafka cluster and the Zookeeper cluster according to the deployment task in the template file based on a preset configuration management tool, the method further includes: and monitoring the deployment operation process of executing the Kafka cluster and the Zookeeper cluster in real time, sending the deployment operation process and the monitoring result thereof to a display terminal, judging whether the deployment operation process is abnormal or not according to the monitoring result, and determining a corresponding abnormal execution step if the deployment operation process is abnormal.

The present application further provides a Kafka cluster deployment device, including:

the first deployment configuration information acquisition unit is used for acquiring the deployment configuration information of the Kafka cluster input by a user;

the second deployment configuration information acquisition unit is used for determining a connection configuration selection result of the Zookeeper cluster; acquiring deployment configuration information of the Zookeeper cluster, which is input by a user and used for providing coordination service for the Kafka cluster, based on the connection configuration selection result;

and the deployment management unit is used for generating a corresponding template file based on the deployment configuration information, and executing the deployment operation of the Kafka cluster and the Zookeeper cluster according to the deployment task in the template file based on a preset configuration management tool.

Further, the first deployment configuration information obtaining unit is specifically configured to:

the second deployment configuration information obtaining unit is specifically configured to:

and if the connection configuration selection result is that a new Zookeeper cluster is selected to be deployed and used, acquiring deployment configuration information of the Zookeeper cluster input by the user based on a second query configuration item based on an interactive query mode.

the method for acquiring the Zookeeper cluster deployment configuration information input by the user based on the second query configuration item based on the interactive query mode specifically comprises the following steps:

Further, the deployment management unit is specifically configured to: and generating a corresponding Playbook template file based on the deployment configuration information of the Kafka cluster, the deployment configuration information of the Zookeeper cluster and a preset standard template format.

Further, the deployment management unit is specifically configured to: and respectively executing the deployment operation of the Kafka cluster and the Zookeeper cluster according to the deployment task in the Playbook template file based on an Angle tool.

Further, before executing the deployment operation of the Kafka cluster and the Zookeeper cluster according to the deployment task in the template file based on a preset configuration management tool, the method further includes: and the authentication connection unit is used for respectively establishing authentication connection with the server nodes of the Kafka cluster and the server nodes of the Zookeeper cluster based on the asymmetric encryption key.

Further, after executing the deployment operations of the Kafka cluster and the Zookeeper cluster according to the deployment task in the template file based on a preset configuration management tool, the method further includes: and the process display and abnormity determining unit is used for monitoring the deployment operation process of executing the Kafka cluster and the Zookeeper cluster in real time, sending the deployment operation process and the monitoring result thereof to a display terminal, judging whether the deployment operation process is abnormal or not according to the monitoring result, and determining a corresponding abnormal execution step if the deployment operation process is abnormal.

The present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute a deployment method implementing the Kafka cluster according to any one of the above embodiments.

The present application also provides a computer-readable storage medium comprising a stored program, wherein the program is run to perform a deployment method that implements the Kafka cluster as described in any of the above.

The present application further provides a computer program product, including a computer program, which when executed by a processor, implements the Kafka cluster deployment method as described in any of the above.

According to the Kafka cluster deployment method, the Kafka cluster deployment configuration information input by a user is obtained firstly; determining a connection configuration selection result of the Zookeeper cluster, and acquiring deployment configuration information of the Zookeeper cluster, which is input by a user and used for providing coordination service for the Kafka cluster, based on the connection configuration selection result; and then generating a corresponding template file based on the deployment configuration information, and executing the deployment operation of the Kafka cluster and the Zookeeper cluster according to the deployment task in the template file based on a preset configuration management tool, so that the deployment efficiency of the Kafka cluster can be effectively improved, the labor cost is saved, the error rate is reduced, and the user experience is improved.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art to obtain other drawings without inventive exercise.

Fig. 1 is a schematic flow chart of a Kafka cluster deployment method provided in the present application;

fig. 2 is a schematic diagram of a specific application of the Kafka cluster deployment method provided in the present application;

fig. 3 is a schematic structural diagram of a deployment apparatus of a Kafka cluster provided in the present application;

fig. 4 is a schematic structural diagram of an electronic device provided in the present application.

Detailed Description

In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making any creative effort shall fall within the protection scope of the present application.

It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the application described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

The following describes an embodiment of the Kafka cluster in detail based on the deployment method of the Kafka cluster described in the present application. As shown in fig. 1, which is a schematic flow chart of the Kafka cluster deployment method provided in the present application, a specific implementation process includes the following steps:

step 101: and acquiring the deployment configuration information of the Kafka cluster input by the user.

Wherein kafka is a distributed publish/subscribe schema-based Message Queue (Message Queue). First Kafka will partition (partition) the received message, with each topic (topic) message having a different partition. Therefore, on one hand, the storage of the message is not limited by the size of the storage space of a single server node, and on the other hand, the message can be processed on a plurality of server nodes in parallel. Secondly, to ensure high availability, there will be a certain number of copies (replica) per partition. Therefore, if part of the server nodes are unavailable, the server node where the copy is located can take over, and the continuity of the application is guaranteed.

In the specific implementation process of the step, the deployment configuration information of the Kafka cluster, which is input by the user based on the first query configuration item, can be acquired in an interactive query mode. Specifically, obtaining the deployment configuration information of the Kafka cluster, which is input by the user based on the first query configuration item, in an interactive query manner may specifically include: based on an interactive query mode, the node number of the Kafka cluster input by a user based on a first node number query configuration item is obtained, the deployment position of the Kafka cluster input by the user based on a first deployment position query configuration item is obtained, the data storage position of the Kafka cluster input by the user based on a first data storage position query configuration item is obtained, the monitoring port number of the Kafka cluster input by the user based on a first port number query configuration item is obtained, and the like.

In the embodiment of the present invention, the parameters related in the query configuration item include, but are not limited to, parameters related to the number of nodes configuring the Kafka cluster and the Zookeeper cluster, a monitoring port number, a deployment location (an IP address of a server node), a data storage location, and the like, and these parameters are all configured by default in order to improve the rapidity of automated deployment, and if part of the default configuration is not satisfied, they may also be modified according to the actual requirements according to the prompt.

Specifically, the parameter configuration process based on the query configuration item is as follows: the Kafka cluster deployment version selection process comprises: the query configuration item corresponding to the deployment version may be "please select the Kafka software package that is desired to be deployed: "the user enters the complete path of the installation package after colon based on the query configuration item, such as: /home/kafka-2.13-3.2.1. Tgz.

The Kafka cluster deployment configuration query selection process includes: the first software package query configuration item may be "please select the Kafka software package that wants to be deployed: "the user inquires about the configuration item to input the complete path of the installation package after the colons based on the first data storage location, such as: /home/kafka-2.13-3.2.1. Tgz; the first node number query configuration item may be "Kafka cluster node number: "the user may input a specific number after colon based on the query configuration item, such as: 3; the first IP address query configuration item may be "please enter the IP address of Kafka node 1: ", the user configures the entry IP address of the item based on the first IP address, such as: 192.168.0.1; the first port number query configuration item may be "please enter the listening port of Kafka node 1 (default 9092): "for inquiring the configuration item based on the first port number, the default port can be used by directly inputting the enter key, or the self-defined port can be input, the input port number is more than 6000 and less than 10000, and the input port number outside the range can prompt: "the input port number is out of range (6000 to 10000)"; the first deployment location query configuration item may be "please enter Kafka installation path for Kafka node 1 (default/apps/Kafka-node 01): "the user can directly input the enter key to use the default directory based on the first deployment location query configuration item, or can input the custom directory as required; the first data-deposit-location query configuration item may be "please enter the data-log saving location (default/tmp/Kafka-logs) of Kafka node 1: "the user can directly input the enter key to use the default directory based on the first data storage location query configuration item, or can input the custom directory as required. And repeating the steps until the deployment configuration information of all the nodes is input.

Step 102: determining a connection configuration selection result of the Zookeeper cluster; and acquiring deployment configuration information of the Zookeeper cluster, which is input by a user and used for providing the coordination service for the Kafka cluster, based on the connection configuration selection result.

ZooKeeper is a distributed, open source distributed application program coordination service, which is software that provides a consistent service for distributed applications, and provides functions including: configuration maintenance, domain name service, distributed synchronization, group service, etc.

In the embodiment of the application, a Zookeeper cluster needs to be distributed and coordinated when a plurality of server nodes are prepared to build the Kafka cluster, so that the Zookeeper cluster is prepared first, for example, the Zookeeper cluster can be built in Linux.

In the specific implementation process of this step, the deployment configuration information of the Zookeeper cluster input by the user based on the second query configuration item may be obtained in an interactive query manner. Specifically, the obtaining of the deployment configuration information of the Zookeeper cluster input by the user based on the second query configuration item through the interactive query mode may specifically include: based on the interactive inquiry mode, acquiring the node number of the Zookeeper cluster input by the user based on the second node number inquiry configuration item, acquiring the deployment position of the Zookeeper cluster input by the user based on the second deployment position inquiry configuration item, acquiring the data storage position of the Zookeeper cluster input by the user based on the second data storage position inquiry configuration item, acquiring the monitoring port number of the Zookeeper cluster input by the user based on the second port number inquiry configuration item, and the like.

Specifically, the Zookeeper cluster connection configuration inquiry process includes: the second cluster selection configuration item may be "whether to use an existing Zookeeper cluster (yes/no): ", yes can be entered if the existing Zookeeper cluster is selected for use; the second data storage location query configuration item may be "please enter the Zookeeper cluster connection address: ", the user inquires about the configuration item based on the second data storage location and inputs the available Zookeeper cluster connection address after the colon. Such as: zookeeper01:2181, zookeeper 02 2181, zookeeper 03; if a deployment is selected and a new Zookeeper cluster is used, a no can be entered in the first query, followed by a Zookeeper cluster deployment configuration query.

The Zookeeper cluster deployment configuration query process comprises: the second node number query configuration item may be "Zookeeper cluster node number: "the user inquires about the configuration item based on the second node number and inputs a specific number after the colon, such as: 3; the second IP address query configuration item "please enter the IP address of Zookeeper node 1: ", the user enters an IP address; the second port number query configuration item may be "please enter the listening port of Zookeeper node 1 (default 2181): "the user may directly input the enter key to use the default port or may input the custom port number based on the second port number query configuration item, where the input port number is greater than 2000 and smaller than 4000, and the user may prompt, after inputting the port outside the range: "the input port number is out of range (2000-4000)"; the second deployment location query configuration item may be "please enter the installation path of Zookeeper node 1 (default/apps/Zookeeper-node 01): "the user can directly input the enter key to use the default directory based on the second deployment location query configuration item, or can input the custom directory as required; and so on until the deployment configuration information of all the nodes is input.

Other query configuration items are not listed here, and can be configured according to actual needs. It should be noted that, in the process of querying configuration, the configured parameters cannot be changed, and if it is desired to change, the deployment tool (configuration management tool) can only be exited and then re-executed for configuration. As shown in fig. 2, after the query configuration task is completed, the deployment tool generates an anchor Playbook template file according to the input configuration information, and then performs remote installation and configuration commands according to the tasks in the Playbook template file by using the anchor deployment tool (i.e., the deployment tool), thereby completing the deployment work. Kafka and Zookeeper generally use official binary packages for installation, and after installation, relevant parameters in configuration files need to be modified respectively according to cluster configuration conditions. According to the deployment tool, related installation and configuration operations do not need to be manually operated on each server node, deployment instructions can be executed through a preset step of a Playbook template file in an alarm tool, and deployment is completed only through a waiting set. Thus, the present application promotes the efficiency of deploying a replica set in an operating system (such as a CentOS). It should be noted that, the deployment tool sets default configurations recommended in advance for the configuration items of Kafka and Zookeeper clusters except the configuration items defined in the query step, and if the user has higher requirements for some configurations, the configuration items may be modified according to the above process.

According to the method, the complex and tedious deployment process of the Kafka cluster is achieved through a plurality of simple questions and answers, the architecture information and the deployment configuration information of the cluster are obtained, the obtained information is perfected into a Playbook template file, and deployment is completed simply and rapidly by utilizing an ansable deployment tool. The Playbook template file is a template which is arranged after multiple practical verifications, the cluster deployment process is uniformly arranged in the Playbook template file, and a user can be helped to well know the cluster deployment process and the writing specification of the Playbook template file.

Step 103: and generating a corresponding template file based on the deployment configuration information, and executing the deployment operation of the Kafka cluster and the Zookeeper cluster according to the deployment task in the template file based on a preset configuration management tool.

Generating a corresponding template file based on the deployment configuration information, wherein the corresponding specific implementation process comprises the following steps: and generating a corresponding Playbook template file based on the deployment configuration information of the Kafka cluster, the deployment configuration information of the Zookeeper cluster and a preset standard template format. The deployment operation of the Kafka cluster and the Zookeeper cluster is executed according to the deployment task in the template file based on the preset configuration management tool, and the corresponding specific implementation process comprises the following steps: and respectively executing the deployment operation of the Kafka cluster and the Zookeeper cluster according to the deployment task in the Playbook template file based on an Angle tool. The infrastructure is a new automatic operation and maintenance tool, is developed based on Python, integrates the advantages of a plurality of operation and maintenance tools (puppet, cfengine, chef, func, fabric), and can realize functions such as batch system configuration, batch program deployment, batch operation commands and the like. The Playbook is an interactive script for configuring, deploying and managing the nodes, and a series of tasks can be executed in the Playbook to enable the remote host to reach a desired state. The Playbook is like the ansable controller lists a series of to-do-lists to the controlled node, and the controlled node must do so. The specific working principle and working mode of the Ansible as an open source tool are out of the scope of the present application. The deployment configuration information includes deployment configuration information of the Kafka cluster and deployment configuration information of a Zookeeper cluster for providing coordination service for the Kafka cluster.

It should be noted that, before executing the deployment operation of the Kafka cluster and the Zookeeper cluster according to the deployment task in the template file based on the preset configuration management tool, authentication connections between the server nodes of the Kafka cluster and the server nodes of the Zookeeper cluster need to be established in advance based on the asymmetric encryption key respectively. For example, before executing the deployment tool, it is necessary to ensure that each server node to be deployed can be remotely connected by using the ssh key through the management host. Each server node needs to install a version more than Python2.4, and if the version is lower than Python2.5, pythonsimplejson needs to be installed; if selinux is enabled, then libselinux-python needs to be installed.

In addition, after the preset configuration management tool executes the deployment operations of the Kafka cluster and the Zookeeper cluster according to the deployment tasks in the template file, in order to avoid abandonment of the whole deployment process due to the occurrence of abnormalities in some sub-steps in the deployment process, the technical solution disclosed in the present application further includes: and monitoring the deployment operation process of executing the Kafka cluster and the Zookeeper cluster in real time, sending the deployment operation process and the monitoring result thereof to a display terminal, judging whether the deployment operation process is abnormal or not according to the monitoring result, and determining a corresponding abnormal execution step if the deployment operation process is abnormal. The execution process and the result record are displayed on a screen by monitoring in real time in the execution process, whether the execution process of the deployment operation of the copy set is abnormal or not is judged according to the real-time monitoring result, and if so, the corresponding abnormal execution step and reason are determined for a user to troubleshoot errors and solve the problems. When the cluster is deployed successfully, the current state and the node number of the cluster are printed on a screen for a user to check. If the deployment tool does not normally complete the automatic deployment of the cluster, the deployment tool needs to be re-executed no matter which step the deployment tool exits.

The method aims to inquire and configure the Kafka cluster and the Zookeeper cluster, and finally, the collected deployment configuration information is executed on each server node by combining with an infrastructure deployment tool, so that the aim of automatic cluster deployment is finally fulfilled. The method solves the problem of complicated manual deployment process of installing and deploying the Kafka cluster in the Linux system, generates corresponding deployment instructions according to the Kafka version and the deployment configuration information, and performs corresponding deployment in a plurality of CentOS operating systems, so that the effect of deploying the cluster by one key is achieved, the working efficiency is greatly improved, the labor cost is saved, and the error rate is reduced. By the Kafka cluster deployment method, operation and maintenance personnel or beginners can flexibly and quickly deploy the Kafka cluster, and the Kafka cluster deployment method is suitable for development, testing and various online production environments.

The following describes the Kafka cluster deployment apparatus provided in the present application, and the Kafka cluster deployment apparatus described below and the Kafka cluster deployment method described above may be referred to in correspondence with each other.

Fig. 3 is a schematic structural diagram of a deployment apparatus of a Kafka cluster provided in the present application.

The Kafka cluster deployment device specifically comprises the following parts:

a first deployment configuration information obtaining unit 301, configured to obtain deployment configuration information of the Kafka cluster input by a user;

a second deployment configuration information obtaining unit 302, configured to determine a connection configuration selection result of the Zookeeper cluster; acquiring deployment configuration information of the Zookeeper cluster, which is input by a user and used for providing coordination service for the Kafka cluster, based on the connection configuration selection result;

and the deployment management unit 303 is configured to generate a corresponding template file based on the deployment configuration information, and execute deployment operations of the Kafka cluster and the Zookeeper cluster according to a deployment task in the template file based on a preset configuration management tool.

and if the connection configuration selection result is that the new Zookeeper cluster is selected to be deployed and used, acquiring deployment configuration information of the Zookeeper cluster input by the user based on the second query configuration item based on an interactive query mode.

the method for acquiring the deployment configuration information of the Zookeeper cluster, which is input by the user based on the second query configuration item based on the interactive query mode specifically comprises the following steps:

Further, the deployment management unit is specifically configured to: and respectively executing the deployment operation of the Kafka cluster and the Zookeeper cluster according to the deployment task in the template file of the Playbook based on an ansable tool.

Further, after executing the deployment operation of the Kafka cluster and the Zookeeper cluster according to the deployment task in the template file based on a preset configuration management tool, the method further includes: and the process display and abnormity determining unit is used for monitoring the deployment operation process of executing the Kafka cluster and the Zookeeper cluster in real time, sending the deployment operation process and the monitoring result thereof to a display terminal, judging whether the deployment operation process is abnormal or not according to the monitoring result, and determining a corresponding abnormal execution step if the deployment operation process is abnormal.

The Kafka cluster deployment device provided by the application firstly acquires Kafka cluster deployment configuration information input by a user; determining a connection configuration selection result of the Zookeeper cluster, and acquiring deployment configuration information of the Zookeeper cluster, which is input by a user and used for providing coordination service for the Kafka cluster, based on the connection configuration selection result; and then generating a corresponding template file based on the deployment configuration information, and executing the deployment operation of the Kafka cluster and the Zookeeper cluster based on a preset configuration management tool according to the deployment task in the template file, so that the deployment efficiency of the Kafka cluster can be effectively improved, the labor cost is saved, the error rate is reduced, and the use experience of a user is improved.

Fig. 4 illustrates a physical structure diagram of an electronic device. As shown in fig. 4, the electronic device may include: a processor (processor) 401, a communication Interface (Communications Interface) 404, a memory (memory) 402 and a communication bus 403, wherein the processor 401, the communication Interface 404 and the memory 402 complete communication with each other through the communication bus 403. Processor 401 may call logic instructions in memory 402 to perform a Kafka cluster deployment method comprising: acquiring the deployment configuration information of the Kafka cluster input by a user; determining a connection configuration selection result of the Zookeeper cluster; acquiring deployment configuration information of the Zookeeper cluster, which is input by a user and used for providing coordination service for the Kafka cluster, based on the connection configuration selection result; and generating a corresponding template file based on the deployment configuration information, and executing the deployment operation of the Kafka cluster and the Zookeeper cluster according to the deployment task in the template file based on a preset configuration management tool.

Furthermore, the logic instructions in the memory 402 may be implemented in software functional units and stored in a computer readable storage medium when sold or used as a stand-alone product. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

In another aspect, the present application further provides a computer program product, where the computer program product includes a computer program, the computer program can be stored on a computer-readable storage medium, and when the computer program is executed by a processor, a computer can execute the method for deploying Kafka cluster provided by the above methods, where the method includes: acquiring the deployment configuration information of the Kafka cluster input by a user; determining a connection configuration selection result of the Zookeeper cluster; based on the connection configuration selection result, acquiring the deployment configuration information of the Zookeeper cluster, which is input by a user and used for providing coordination service for the Kafka cluster; and generating a corresponding template file based on the deployment configuration information, and executing the deployment operation of the Kafka cluster and the Zookeeper cluster according to the deployment task in the template file based on a preset configuration management tool.

In another aspect, the present application further provides a computer-readable storage medium, where the computer-readable storage medium includes a stored program, where the program when executed performs the method for deploying a Kafka cluster provided by the foregoing methods, where the method includes: acquiring the deployment configuration information of the Kafka cluster input by a user; determining a connection configuration selection result of the Zookeeper cluster; acquiring deployment configuration information of the Zookeeper cluster, which is input by a user and used for providing coordination service for the Kafka cluster, based on the connection configuration selection result; and generating a corresponding template file based on the deployment configuration information, and executing the deployment operation of the Kafka cluster and the Zookeeper cluster according to the deployment task in the template file based on a preset configuration management tool.

The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.

Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. Based on the understanding, the above technical solutions substantially or otherwise contributing to the prior art may be embodied in the form of a software product, which may be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the various embodiments or some parts of the embodiments.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.

Claims

1. A Kafka cluster deployment method is characterized by comprising the following steps:

acquiring the deployment configuration information of the Kafka cluster input by a user;

2. The Kafka cluster deployment method according to claim 1, wherein the acquiring of the Kafka cluster deployment configuration information input by the user specifically includes:

based on an interactive query mode, acquiring deployment configuration information of the Kafka cluster input by a user based on a first query configuration item;

the obtaining, based on the connection configuration selection result, deployment configuration information of the Zookeeper cluster that is input by the user and used for providing the coordination service for the Kafka cluster specifically includes: and if the connection configuration selection result is that a new Zookeeper cluster is selected to be deployed and used, acquiring deployment configuration information of the Zookeeper cluster input by the user based on a second query configuration item based on an interactive query mode.

3. The Kafka cluster deployment method according to claim 2, wherein the obtaining of the Kafka cluster deployment configuration information input by the user based on the first query configuration item based on the interactive query manner specifically includes:

4. The Kafka cluster deployment method of claim 1, wherein the generating a corresponding template file based on the deployment configuration information specifically includes:

and generating a corresponding Playbook template file based on the deployment configuration information of the Kafka cluster, the deployment configuration information of the Zookeeper cluster and a preset standard template format.

5. The Kafka cluster deployment method according to claim 4, wherein the performing, by the preset-based configuration management tool, the deployment operation of the Kafka cluster and the Zookeeper cluster according to the deployment task in the template file specifically includes:

and respectively executing the deployment operation of the Kafka cluster and the Zookeeper cluster according to the deployment task in the Playbook template file based on an Angle tool.

6. The Kafka cluster deployment method according to claim 1, before executing, based on a preset configuration management tool, the deployment operation of the Kafka cluster and the Zookeeper cluster according to the deployment task in the template file, the method further comprising:

and respectively establishing authentication connection between the server nodes of the Kafka cluster and the server nodes of the Zookeeper cluster based on the asymmetric encryption key.

7. The Kafka cluster deployment method according to claim 1, further comprising, after executing, based on a preset configuration management tool, the deployment operations of the Kafka cluster and the Zookeeper cluster according to the deployment tasks in the template file:

and monitoring the deployment operation process of executing the Kafka cluster and the Zookeeper cluster in real time, sending the deployment operation process and the monitoring result thereof to a display terminal, judging whether the deployment operation process is abnormal or not according to the monitoring result, and determining a corresponding abnormal execution step if the deployment operation process is abnormal.

8. A Kafka cluster deployment apparatus, comprising:

9. A computer-readable storage medium comprising a stored program, wherein the program when executed performs the method of deployment of a Kafka cluster of any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the Kafka cluster deployment method of any one of claims 1 to 7 by the computer program.