CN111142880B - Partition allocation method and device in message system and electronic equipment - Google Patents
Partition allocation method and device in message system and electronic equipment Download PDFInfo
- Publication number
- CN111142880B CN111142880B CN201911356977.6A CN201911356977A CN111142880B CN 111142880 B CN111142880 B CN 111142880B CN 201911356977 A CN201911356977 A CN 201911356977A CN 111142880 B CN111142880 B CN 111142880B
- Authority
- CN
- China
- Prior art keywords
- allocated
- partition
- partitions
- allocation
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000005192 partition Methods 0.000 title claims abstract description 524
- 238000000034 method Methods 0.000 title claims abstract description 87
- 238000009826 distribution Methods 0.000 claims abstract description 18
- 238000004891 communication Methods 0.000 claims description 19
- 238000004590 computer program Methods 0.000 claims description 8
- 238000000638 solvent extraction Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 description 31
- 230000005012 migration Effects 0.000 description 5
- 238000013508 migration Methods 0.000 description 5
- 238000003860 storage Methods 0.000 description 5
- 230000009471 action Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 238000004519 manufacturing process Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 238000005538 encapsulation Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004883 computer application Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a partition allocation method, a partition allocation device and electronic equipment in a message system, wherein the method can comprise the following steps: obtaining allocation information of a target partition to be allocated; the target partition to be allocated is part of the partitions to be allocated; distributing the target partition to be distributed according to the distribution information of the target partition to be distributed; when the allocation of the target partition to be allocated is completed, obtaining allocation information of the remaining partition to be allocated, and allocating the remaining partition to be allocated according to the allocation information of the remaining partition to be allocated until the allocation of all the partitions to be allocated is completed; the remaining partitions to be allocated are unallocated partitions among the partitions to be allocated. The partition allocation method, the partition allocation device and the electronic equipment in the message system can reallocate the partitions under the condition that the Kafka cluster of the message system changes.
Description
Technical Field
The present invention relates to the field of computer application technologies, and in particular, to a partition allocation method and apparatus in a message system, and an electronic device.
Background
Kafka is a high throughput distributed publish-subscribe messaging system that can handle all action stream data of consumers in websites, and has extremely wide application in internet companies and the like. The Kafka provides a message service through a Kafka cluster, and in particular, the Kafka cluster may deploy a plurality of nodes to provide a message service through the nodes.
In Kafka, topic is a logical concept of storing messages, also called channels, and may be considered as a collection of messages, each message sent to the Kafka cluster having a category. In order to extend the throughput of Kafka, the Topic is physically divided into one or more partitions (partitions), each of which can be physically understood as corresponding to a folder, and the messages are stored through a plurality of partitions in Kafka. To increase the security of messages in Kafka, a partition may have multiple copies deployed on different nodes. One of the copies of each partition is taken as a Leader, the other copies of the partition except the Leader are taken as a Follower, and the Follower is responsible for synchronizing messages from the Leader, and when the current Leader is not available, the other copy is taken as a new Leader. The production, consumption, etc. of the message in Kafka are all based on the Leader, e.g. the producer stores the message in the Leader, from which the consumer consumes the message. The process of determining at which node a partition is deployed may be understood as an allocation process to a partition.
With the demands of business development and the like, the scale of the Kafka cluster may change, for example, the Kafka cluster is expanded, i.e., nodes are added in the Kafka cluster. At this time, a need arises for reassignment of the plurality of partitions.
Disclosure of Invention
The embodiment of the invention aims to provide a partition allocation method, a partition allocation device and electronic equipment in a message system, so that a plurality of partitions are reallocated under the condition that Kafka clusters change. The specific technical scheme is as follows:
in a first aspect, an embodiment of the present invention provides a method for allocating partitions in a message system, including:
obtaining allocation information of a target partition to be allocated, wherein the allocation information of the target partition to be allocated is used for indicating the corresponding relation between the target partition to be allocated and nodes in the message system; the target partition to be allocated is part of the partitions to be allocated;
distributing the target partition to be distributed according to the distribution information of the target partition to be distributed;
when the allocation of the target partition to be allocated is completed, obtaining allocation information of the remaining partition to be allocated, and allocating the remaining partition to be allocated according to the allocation information of the remaining partition to be allocated until allocation of all the partitions to be allocated is completed; the remaining partitions to be allocated are unallocated partitions to be allocated among the partitions to be allocated.
Optionally, before the obtaining the allocation information of the target partition to be allocated, the method further includes:
acquiring node related information and partition information of the partition to be allocated; wherein the node-related information is information related to a node in the message system;
and generating the allocation information of the target partition to be allocated according to the node related information and the partition information.
Optionally, the target partition to be allocated is one partition to be allocated among the partitions to be allocated;
the remaining partition to be allocated is an unallocated partition among the partitions to be allocated.
Optionally, the allocating the target partition to be allocated according to the allocation information of the target partition to be allocated includes:
and deploying the target partition to be allocated on a target node, wherein the target node is a node which is indicated in allocation information of the target partition to be allocated and has a corresponding relation with the target partition to be allocated.
Optionally, after the target partition to be allocated is allocated according to the allocation information of the target partition to be allocated, the method further includes:
and synchronizing the information stored in the target partition to be allocated to a memory space corresponding to the target partition to be allocated in the target node.
Optionally, the partition to be allocated includes a partition deployed on a node to be deleted in the message system, where the node to be deleted is a node deleted when the message system is scaled.
In a second aspect, an embodiment of the present invention provides a partition allocation apparatus in a message system, including:
the first obtaining module is used for obtaining the allocation information of the target partition to be allocated, wherein the allocation information of the target partition to be allocated is used for indicating the corresponding relation between the target partition to be allocated and the nodes in the message system; the target partition to be allocated is part of the partitions to be allocated;
the allocation module is used for allocating the target partition to be allocated according to the allocation information of the target partition to be allocated;
the first obtaining module is further configured to obtain allocation information of the remaining partitions to be allocated when allocation of the target partition to be allocated is completed;
the distribution module is further configured to distribute the remaining partitions to be distributed according to the distribution information of the remaining partitions to be distributed until all the partitions to be distributed are distributed; the remaining partitions to be allocated are unallocated partitions to be allocated among the partitions to be allocated.
Optionally, the apparatus further includes:
the second obtaining module is used for obtaining node related information and partition information of the partition to be allocated before obtaining allocation information of the target partition to be allocated; wherein the node-related information is information related to a node in the message system;
and the generation module is used for generating the allocation information of the target partition to be allocated according to the node related information and the partition information.
Optionally, the allocation module is specifically configured to deploy the target partition to be allocated to a target node, where the target node is a node indicated in allocation information of the target partition to be allocated and having a corresponding relationship with the target partition to be allocated.
Optionally, the apparatus further includes: and the synchronization module is used for synchronizing the information stored in the target partition to be allocated to the memory space corresponding to the target partition to be allocated in the target node after the target partition to be allocated is allocated according to the allocation information of the target partition to be allocated.
In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus;
The memory is used for storing a computer program;
the processor is configured to implement the method steps of the method described in the first aspect when executing the program stored in the memory.
In yet another aspect of the present invention, there is also provided a computer readable storage medium having stored therein instructions which, when run on a computer, cause the computer to perform the method steps of the method of the first aspect described above.
In a further aspect of the invention, the embodiments of the invention also provide a computer program product comprising instructions which, when run on a computer, cause the computer to perform the method steps of the method of the first aspect described above.
The partition allocation method, the partition allocation device and the electronic equipment in the message system provided by the embodiment of the invention can obtain the allocation information of the target partition to be allocated; distributing the target partition to be distributed according to the distribution information of the target partition to be distributed; when the allocation of the target partition to be allocated is completed, the allocation information of the remaining partition to be allocated is obtained, and the remaining partition to be allocated is allocated according to the allocation information of the remaining partition to be allocated until the allocation of all the partitions to be allocated is completed. The partition allocation method, the partition allocation device and the electronic equipment in the message system can reallocate the partitions under the condition that Kafka clusters are changed. Of course, it is not necessary for any one product or method of practicing the invention to achieve all of the advantages set forth above at the same time.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.
FIG. 1 is a flowchart of a method for partitioning in a messaging system according to an embodiment of the present invention;
FIG. 2 is another flow chart of a partition allocation method in a message system according to an embodiment of the present invention;
FIG. 3a is a flow chart of a conventional capacity expansion method;
fig. 3b is a schematic flow chart of a capacity expansion manner according to an embodiment of the present invention;
FIG. 4 is a schematic structural diagram of a partition allocation device in a message system according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of another configuration of a partition allocation apparatus in a message system according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of another configuration of a partition allocation apparatus in a message system according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be described below with reference to the accompanying drawings in the embodiments of the present invention.
With the demands of business development and the like, the scale of the Kafka cluster may change, and at this time, the demands for reassignment of a plurality of partitions may arise. In the prior art, the following problems exist in the process of distributing the partitions:
On the one hand, the Kafka cluster is expanded, in the process of reassigning the plurality of partitions, in the prior art, the plurality of partitions are assigned at the same time in consideration of the assignment efficiency, and after all the partitions are assigned, the Leader corresponding to each partition is elected, so that message service can be provided for each partition. While it takes a certain time to allocate multiple partitions at the same time, so that there is no corresponding Leader for each partition in this period, and the production, consumption, etc. of the message in Kafka are performed based on the Leader, that is, the message service is provided based on the Leader, which results in interruption of the message service in this period.
On the other hand, the Kafka cluster may not be available or the traffic becomes smaller, and the Kafka cluster may be scaled, for example, when one or several nodes in the Kafka cluster are deleted, in the prior art, when one or several nodes in the Kafka cluster are to be deleted, a partition deployed on a node may be deleted, and a partition deployed on the node may be a Leader corresponding to the partition, and a time is required for re-electing the Leader, which may cause a message service interruption for the partition in this time period, or even if a Follower corresponding to the partition is deployed on the node to be deleted, there may be a hidden danger for the security of a message for the partition.
In order to solve the problems existing in the process of partition allocation in the prior art, the embodiment of the invention provides a partition allocation method in a message system. The embodiment of the invention realizes the single concurrent distribution of the plurality of partitions to be distributed, so as to reduce the influence on the message service in the distribution process of the partitions, and can understand that the distribution process of the partitions which cannot be perceived by the outside can be realized, and realize the transparent migration of the partitions. And the capacity reduction of the Kafka cluster can be realized, and the partition deployed on the node to be deleted can be redistributed, so that the partition deployed on the node to be deleted is migrated to other nodes. The method can reduce the influence on the message service in the capacity shrinking process of the Kafka cluster, avoid hidden danger of the safety of the message and ensure the integrity of the message.
The following describes the partition allocation method in the message system provided by the embodiment of the invention in detail. The partition allocation method in the message system provided by the embodiment of the invention can be applied to a management platform for managing a plurality of nodes in the message system, and particularly can be realized through a scheduling service deployed in the management platform; alternatively, an independent scheduling server may be deployed in the message system, and the partition allocation method in the message system provided by the embodiment of the present invention is executed through the scheduling server.
The embodiment of the invention provides a partition allocation method in a message system, which can comprise the following steps:
obtaining allocation information of a target partition to be allocated, wherein the allocation information of the target partition to be allocated is used for indicating the corresponding relation between the target partition to be allocated and nodes in a message system; the target partition to be allocated is part of the partitions to be allocated;
distributing the target partition to be distributed according to the distribution information of the target partition to be distributed;
when the allocation of the target partition to be allocated is completed, obtaining allocation information of the remaining partition to be allocated, and allocating the remaining partition to be allocated according to the allocation information of the remaining partition to be allocated until the allocation of all the partitions to be allocated is completed; the remaining partitions to be allocated are unallocated partitions to be allocated in the partitions to be allocated, and allocation information of the remaining partitions to be allocated is used for indicating a corresponding relationship between the remaining partitions to be allocated and nodes in the message system.
In the embodiment of the invention, the allocation information of the target partition to be allocated is obtained; distributing the target partition to be distributed according to the distribution information of the target partition to be distributed; when the allocation of the target partition to be allocated is completed, the allocation information of the remaining partition to be allocated is obtained, and the remaining partition to be allocated is allocated according to the allocation information of the remaining partition to be allocated until the allocation of all the partitions to be allocated is completed. The partition allocation method, the partition allocation device and the electronic equipment in the message system can reallocate the partitions under the condition that Kafka clusters are changed.
And the allocation can be performed for the partial partition each time until all the partitions to be allocated are completely allocated. Therefore, the situation that message service cannot be provided for all the partitions to be allocated within the occupied time of simultaneously allocating all the partitions to be allocated can be avoided, when the partial partitions are allocated, the message service of other partitions is not affected, when the allocation of the partial partitions is completed, the allocation of other partitions is not waited, the message service of the partial partitions can be provided, and the influence on the message service in the allocation process of the partitions can be reduced.
Referring to fig. 1, a detailed description is given of a partition allocation method in a message system according to an embodiment of the present invention.
S101, obtaining allocation information of a target partition to be allocated.
The allocation information of the target partition to be allocated is used for indicating the corresponding relation between the target partition to be allocated and the nodes in the message system. In one implementation, the assignment information may be in the form of JSON (JavaScript Object Notation) files, JSON being a lightweight data exchange format.
The target partition to be allocated is a part of the partitions to be allocated. Such as may be one or more of the partitions to be allocated. Specifically, one or more of the partitions to be allocated may be randomly selected as the target partition to be allocated. Or, each partition to be allocated may be numbered, and one or more of the partitions to be allocated are sequentially used as target partitions to be allocated according to the sequence of the numbers.
In the message system, the message is stored through a plurality of partitions, and the partition to be allocated can be determined by actual requirements. In an alternative embodiment, the message system is Kafka, in the process of expanding the Kafka cluster, it may be also understood that the number of nodes in the Kafka cluster after expansion is greater than the number of nodes before expansion, and all or part of the partitions in the message system may be reassigned, that is, the partition to be assigned may be all or part of the partitions in the message system. In another alternative embodiment, when the message system is scaled, for example, when the nodes in the Kafka cluster are deleted, the number of the nodes in the message system after scaling is smaller than the number of the nodes in the message system before scaling, and the partition to be allocated may include a partition deployed on the node to be deleted, where the node to be deleted is the node deleted when the message system is scaled. Therefore, the partitions deployed on the nodes to be deleted can be redistributed, and the partitions deployed on the nodes to be deleted can be migrated to other nodes in the message system.
S102, distributing the target partition to be distributed according to the distribution information of the target partition to be distributed.
Specifically, the allocation of the target partition to be allocated may include: and deploying the target partition to be allocated at the target node.
The target node is a node which has a corresponding relation with the target partition to be allocated and is indicated in the allocation information of the target partition to be allocated.
In an alternative embodiment, it may be understood that the target partition to be allocated is registered, specifically, a directory table of the partition corresponding to each node may be maintained, and the target partition to be allocated is registered, that is, it may be understood that identification information of the target partition to be allocated is added to the directory table corresponding to the target node.
And S103, when the allocation of the target partition to be allocated is completed, obtaining the allocation information of the residual partition to be allocated, and allocating the residual partition to be allocated according to the allocation information of the residual partition to be allocated until the allocation of all the partitions to be allocated is completed.
The remaining partitions to be allocated are unallocated partitions among the partitions to be allocated. Such as one or more of the unassigned partitions to be assigned.
The allocation information of the remaining partitions to be allocated is used for indicating the corresponding relation between the remaining partitions to be allocated and nodes in the message system.
When the allocation of the target partition to be allocated is completed, judging whether an unallocated partition to be allocated exists in the partition to be allocated; and when the unallocated partition to be allocated exists in the partition to be allocated, obtaining allocation information of the remaining partitions to be allocated.
Specifically, the allocating the remaining partition to be allocated according to the allocation information of the remaining partition to be allocated may include: and deploying the remaining partitions to be allocated on nodes with corresponding relations with the remaining partitions to be allocated.
In the embodiment of the invention, the reassignment of a plurality of partitions can be realized under the condition that the Kafka cluster changes. And the allocation can be performed for the partial partition each time until all the partitions to be allocated are completely allocated. Therefore, the situation that message service cannot be provided for all the partitions to be allocated within the occupied time of simultaneously allocating all the partitions to be allocated can be avoided, when the partial partitions are allocated, the message service of other partitions is not affected, when the allocation of the partial partitions is completed, the allocation of other partitions is not waited, the message service of the partial partitions can be provided, and the influence on the message service in the allocation process of the partitions can be reduced. The partition allocation process which cannot be perceived by the outside can be realized, and the transparent migration of the partition can be realized. And when the message system is scaled, the partitions deployed on the nodes to be deleted can be redistributed, so that the partitions deployed on the nodes to be deleted can be migrated to other nodes. The method can reduce the influence on the message service in the process of shrinking the message system, avoid hidden danger of the safety of the message and ensure the integrity of the message in the message system.
In an alternative embodiment, before obtaining the allocation information of the target partition to be allocated, the method further includes:
acquiring node related information and partition information of a partition to be allocated; wherein the node-related information is information related to a node in the message system; and generating allocation information of the target partition to be allocated according to the node related information and the partition information.
In an embodiment of the invention, the message system can comprise a metadata generation module, and metadata in the message system is stored through the metadata generation module. In an alternative embodiment, the message system is Kafka, the metadata generation module may store the Topic metadata in the Kafka, such as partition information including the number of partitions corresponding to each Topic, and the metadata generation module may also store node related information, such as the number of nodes in the Kafka cluster, performance information of each node, and so on.
In the embodiment of the invention, the allocation information can be generated according to the node related information and the partition information of the partition to be allocated, namely, the corresponding relation between each partition to be allocated and the nodes in the message system is determined, and the node to which the partition to be allocated is to be allocated can be understood as aiming at each partition to be allocated. In one implementation, JSON files may be generated. The JSON file corresponding to one partition to be allocated can be understood as allocation information, and is used for indicating the corresponding relation between the partition to be allocated and nodes in a message system. According to the JSON file corresponding to the partition to be allocated, allocation of the partition to be allocated can be completed, namely the partition to be allocated is deployed on the node, corresponding to the partition to be allocated, of the JSON file corresponding to the partition to be allocated, and the node has a corresponding relation with the partition to be allocated.
In the embodiment of the invention, JSON files corresponding to each partition to be allocated can be added into a Block Queue (Queue Block), wherein the Block Queue is a virtual concept and can be understood as a Queue. The JSON files corresponding to the partitions to be allocated can be used as elements in the Block Queue, the partitions to be allocated can be numbered, and the JSON files corresponding to the partitions to be allocated are sequentially added to the Block Queue according to the sequence of the numbers. In this way, in the process of partitioning each partition to be allocated, JSON files corresponding to the partition to be allocated can be obtained from the Block Queue sequentially according to the numbers of the partition to be allocated.
In an alternative embodiment of the present invention, in step S102: after the target partition to be allocated is allocated according to the allocation information of the target partition to be allocated, the method may further include: and synchronizing the information stored in the target partition to be allocated to a memory space corresponding to the target partition to be allocated in the target node.
After the target partition to be allocated is deployed on the node with the corresponding relation with the target partition to be allocated, synchronizing the information stored in the target partition to be allocated into the memory space corresponding to the target partition to be allocated in the node with the corresponding relation with the target partition to be allocated.
Correspondingly, after the remaining partitions to be allocated are allocated according to the allocation information of the remaining partitions to be allocated, that is, after the remaining partitions to be allocated are deployed at the nodes corresponding to the remaining partitions to be allocated, the information stored in the remaining partitions to be allocated is synchronized to the memory space corresponding to the remaining partitions to be allocated in the nodes corresponding to the remaining partitions to be allocated.
In an alternative embodiment of the present invention, the target partition to be allocated is one partition to be allocated among the partitions to be allocated; the remaining to-be-allocated partitions are one of the to-be-allocated partitions that are not allocated among the to-be-allocated partitions. That is, the allocation is performed for one partition to be allocated at a time until all the partitions to be allocated are completely allocated, as shown in fig. 2, may include:
s201, distributing a partition to be distributed.
In the embodiment of the invention, the partition to be allocated is a partition storing the message in Kafka.
And obtaining the allocation information of a partition to be allocated, namely obtaining the allocation information of a partition.
In one implementation manner, an operation of generating an allocation file may be invoked, and a JSON file corresponding to a part of the file may be generated, e.g., a JSON file corresponding to a part identified as a: and a.json, performing operation by using the a.json call, and distributing the partition. The operation of generating the allocation file may be understood as an interface or a function, and may be specifically obtained by encapsulation according to a process of obtaining allocation information. An execution operation may also be understood as an interface or function, and may specifically be obtained according to a process encapsulation for allocating a partition to be allocated.
In a specific example, the partition may be added to the Zookeeper/admin/reassessionpoint node according to JSON file.
S202, judging whether the distribution is completed or not; when it is judged that the allocation is not completed, step S203 is executed, and when it is judged that the allocation is completed, step S204 is executed.
S203, wait for allocation to be performed again.
S204, recording the partition allocation completion.
The allocation file allocation completion state can be recorded by calling an operation for verifying whether the allocation file is processed or not and detecting whether the allocation file is processed or not again after waiting a period of time if the allocation is not completed. The verification of the distribution file operation can also be understood as an interface or function.
S205, judging whether an unallocated partition to be allocated exists; if there is an unallocated partition to be allocated, the process returns to step S201, and if there is no unallocated partition to be allocated, step S206 is performed.
S206, completing distribution and notifying.
Detecting whether the remaining part data needs to be processed, if so, re-executing step S201 to obtain the unallocated partition to be allocated, namely obtaining the remaining part data to continue processing; if no part data needs to be processed, the execution is finished. In one implementation, when the allocation is complete, notification information may be sent, such as a notification message to each node in the messaging system indicating that the allocation is complete.
In an optional embodiment, after the allocation of the part is completed, synchronization and migration of the part may be completed through a migration module, and specifically, information stored by the part may be copied from a node where the part is located before the allocation to a node where the part is located after the allocation.
In the embodiment of the invention, the allocation can be carried out for one part each time until all the parts to be allocated are completely allocated, and the single concurrent allocation of a plurality of parts can be realized, so that when one part is allocated, the message service of other parts is not affected, and when one part is allocated, the message service of the other part is not waited for being allocated, and the message service of the one part can be provided, thereby reducing the influence on the message service in the partition allocation process.
In the existing capacity expansion mode, as shown in fig. 3a, the original cluster includes 3 nodes, each node includes 3 partitions, partition 1, partition 2 and partition 3, partition 1 represented by a black filled circle is a Leader in 3 partitions 1, the other 2 partitions 1 represented by a non-filled circle are follow ers, partition 2 represented by a black filled circle is a Leader in 3 partitions 2, and the other 2 partitions 2 represented by a non-filled circle are follow ers; the partition 3 represented by the black filled circle is the Leader among the 3 partitions 1, and the other 3 partitions 1 represented by the unfilled circle are the followers; wherein the Leader is responsible for providing message service, and the Follower synchronizes messages from the Leader. In the existing capacity expansion process, the new cluster after capacity expansion comprises 4 nodes. And simultaneously distributing a plurality of partitions according to a distribution strategy, wherein the newly added capacity expansion nodes in the new cluster comprise a Leader of a partition 3, 1 Follower of a partition 1 and 1 Follower of a partition 2, and 1 node in the 3 existing nodes in the original cluster comprises: 1 Follower for partition 1, 1 Follower for partition 3; 1 node includes a Leader of partition 2, 1 Follower of partition 3; 1 node includes a Leader of partition 1, and 1 Follower of partition 2. And after the reassignment is completed, the Leader of each partition needs to be reselected.
The embodiment of the invention can also support the capacity expansion of the message system, as shown in fig. 3b, the original cluster in fig. 3b is the same as the original cluster in fig. 3a, and in the capacity expansion process of the embodiment of the invention, the new cluster after capacity expansion comprises 4 nodes. The scheduling server allocates each partition to be allocated in sequence according to an allocation policy, and can also be understood as performing allocation in a single concurrency manner, and allocates one partition when allocation of the other partition is completed, until all the partitions to be allocated are completely allocated. In the embodiment of the present invention, the reassignment is performed first for the partition 1, the Leader in the partition 1 is assigned to an existing node in the original cluster, 1 of the 2 follow in the partition 1 is assigned to an existing node in the original cluster, the other is assigned to a newly added expansion node in the new cluster, and the assignment of the partition 1 in the new cluster is shown in fig. 3 b.
In the prior art, a plurality of partitions are allocated at the same time, and after all the partitions are allocated, the Leader corresponding to each partition is elected, so that message service can be provided for each partition. While it takes a certain time to allocate multiple partitions at the same time, so that there is no corresponding Leader for each partition in this period, and the production, consumption, etc. of the message in Kafka are performed based on the Leader, that is, the message service is provided based on the Leader, which results in interruption of the message service in this period.
In the embodiment of the invention, the single concurrent allocation of a plurality of partitions to be allocated can be realized, namely one partition is allocated at the same time, and after the allocation of one partition is completed, the other partition is allocated until all the partitions to be allocated are completely allocated. The problem that message service cannot be provided for a plurality of partitions to be allocated in the occupied time caused by the simultaneous allocation of the partitions is avoided, and the influence on the message service is reduced in the allocation process of the partitions. The partition allocation process which cannot be perceived by the outside can be realized, and the transparent migration of the partition can be realized. The method can reduce the influence on the message service in the capacity expansion process of the message system, and provide powerful guarantee for providing the stability service on line.
The embodiment of the invention can support the capacity expansion of the message system and simultaneously support the capacity reduction of the message system. If one or more nodes in the Kafka cluster are deleted, the partition deployed on the node to be deleted can be redistributed, and then the partition deployed on the node to be deleted can be migrated to the node which is not deleted in the Kafka cluster. Therefore, the influence on the message service can be reduced in the capacity shrinking process of the message system, the hidden danger of the safety of the message is avoided, and the integrity of the message in the message system is ensured.
Corresponding to the method for allocating partitions in the message system provided in the foregoing embodiment, the embodiment of the present invention further provides a device for allocating partitions in a message system, as shown in fig. 4, which may include:
the first obtaining module 401 is configured to obtain allocation information of a target partition to be allocated, where the allocation information of the target partition to be allocated is used to indicate a correspondence between the target partition to be allocated and a node in the message system; the target partition to be allocated is part of the partitions to be allocated;
the allocation module 402 is configured to allocate the target partition to be allocated according to allocation information of the target partition to be allocated;
the first obtaining module 401 is further configured to obtain allocation information of the remaining partitions to be allocated when allocation of the target partition to be allocated is completed;
the allocation module 402 is further configured to allocate the remaining partitions to be allocated according to allocation information of the remaining partitions to be allocated until allocation of all the partitions to be allocated is completed; the remaining partitions to be allocated are unallocated partitions among the partitions to be allocated.
The embodiment of the invention can realize the reassignment of a plurality of partitions under the condition that the Kafka cluster changes. And the allocation can be performed for the partial partition each time until all the partitions to be allocated are completely allocated. Therefore, the situation that message service cannot be provided for all the partitions to be allocated within the occupied time of simultaneously allocating all the partitions to be allocated can be avoided, when the partial partitions are allocated, the message service of other partitions is not affected, when the allocation of the partial partitions is completed, the allocation of other partitions is not waited, the message service of the partial partitions can be provided, and the influence on the message service in the allocation process of the partitions can be reduced.
Optionally, as shown in fig. 5, the apparatus may further include:
a second obtaining module 403, configured to obtain node related information and partition information of a partition to be allocated before obtaining allocation information of a target partition to be allocated; wherein the node-related information is information related to a node in the message system;
and the generating module 404 is configured to generate allocation information of the target partition to be allocated according to the node related information and the partition information.
Optionally, the target partition to be allocated is one partition to be allocated among the partitions to be allocated;
the remaining partition to be allocated is an unallocated one of the partitions to be allocated.
Optionally, the allocation module 402 is specifically configured to deploy the target partition to be allocated to a target node, where the target node is a node indicated in allocation information of the target partition to be allocated and has a corresponding relationship with the target partition to be allocated.
Optionally, as shown in fig. 6, the apparatus may further include:
and the synchronization module 405 is configured to synchronize information stored in the target partition to be allocated to a memory space corresponding to the target partition to be allocated in the target node after allocating the target partition to be allocated according to allocation information of the target partition to be allocated.
Optionally, the partition to be allocated includes a partition deployed on a node to be deleted in the message system, where the node to be deleted is a node deleted when the message system is scaled.
The partition allocation device in the message system provided by the embodiment of the invention is a device applying the partition allocation method in the message system, so that all embodiments of the partition allocation method in the message system are applicable to the device and can achieve the same or similar beneficial effects.
Corresponding to the partition allocation method in the message system provided in the foregoing embodiment, the embodiment of the present invention further provides an electronic device, as shown in fig. 7, including a processor 701, a communication interface 702, a memory 703, and a communication bus 704, where the processor 701, the communication interface 702, and the memory 703 complete communication with each other through the communication bus 704.
A memory 703 for storing a computer program;
the processor 701 is configured to implement the method steps of the partition allocation method in the message system provided in the above embodiment when executing the program stored in the memory 703.
The embodiment of the invention can realize the reassignment of a plurality of partitions under the condition that the Kafka cluster changes. And the allocation can be performed for the partial partition each time until all the partitions to be allocated are completely allocated. Therefore, the situation that message service cannot be provided for all the partitions to be allocated within the occupied time of simultaneously allocating all the partitions to be allocated can be avoided, when the partial partitions are allocated, the message service of other partitions is not affected, when the allocation of the partial partitions is completed, the allocation of other partitions is not waited, the message service of the partial partitions can be provided, and the influence on the message service in the allocation process of the partitions can be reduced.
The communication bus mentioned by the above electronic device may be a peripheral component interconnect standard (Peripheral Component Interconnect, abbreviated as PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, abbreviated as EISA) bus, or the like. The communication bus may be classified as an address bus, a data bus, a control bus, or the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.
The communication interface is used for communication between the electronic device and other devices.
The memory may include random access memory (Random Access Memory, RAM) or non-volatile memory (non-volatile memory), such as at least one disk memory. Optionally, the memory may also be at least one memory device located remotely from the aforementioned processor.
The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), etc.; but also digital signal processors (Digital Signal Processing, DSP for short), application specific integrated circuits (Application Specific Integrated Circuit, ASIC for short), field-programmable gate arrays (Field-Programmable Gate Array, FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
In yet another embodiment of the present invention, there is also provided a computer readable storage medium having stored therein instructions that, when executed on a computer, cause the computer to perform the method steps of the partition allocation method in the message system provided in the above embodiment.
The embodiment of the invention can realize the reassignment of a plurality of partitions under the condition that the Kafka cluster changes. And the allocation can be performed for the partial partition each time until all the partitions to be allocated are completely allocated. Therefore, the situation that message service cannot be provided for all the partitions to be allocated within the occupied time of simultaneously allocating all the partitions to be allocated can be avoided, when the partial partitions are allocated, the message service of other partitions is not affected, when the allocation of the partial partitions is completed, the allocation of other partitions is not waited, the message service of the partial partitions can be provided, and the influence on the message service in the allocation process of the partitions can be reduced.
In a further embodiment of the present invention, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform the method steps of the method of assigning partitions in a messaging system provided in the above embodiments.
The embodiment of the invention can realize the reassignment of a plurality of partitions under the condition that the Kafka cluster changes. And the allocation can be performed for the partial partition each time until all the partitions to be allocated are completely allocated. Therefore, the situation that message service cannot be provided for all the partitions to be allocated within the occupied time of simultaneously allocating all the partitions to be allocated can be avoided, when the partial partitions are allocated, the message service of other partitions is not affected, when the allocation of the partial partitions is completed, the allocation of other partitions is not waited, the message service of the partial partitions can be provided, and the influence on the message service in the allocation process of the partitions can be reduced.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present invention, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, by wired (e.g., coaxial cable, optical fiber, digital Subscriber Line (DSL)), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), etc.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for apparatus, electronic devices, computer readable storage media and computer program product embodiments, the description is relatively simple as it is substantially similar to method embodiments, as relevant points are found in the partial description of method embodiments.
The foregoing description is only of the preferred embodiments of the present invention and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.
Claims (7)
1. A method for partitioning in a messaging system, comprising:
obtaining allocation information of a target partition to be allocated, wherein the allocation information of the target partition to be allocated is used for indicating the corresponding relation between the target partition to be allocated and nodes in the message system; the target partition to be allocated is part of the partitions to be allocated;
distributing the target partition to be distributed according to the distribution information of the target partition to be distributed;
when the allocation of the target partition to be allocated is completed, obtaining allocation information of the remaining partition to be allocated, and allocating the remaining partition to be allocated according to the allocation information of the remaining partition to be allocated until allocation of all the partitions to be allocated is completed; the allocation information of the remaining partitions to be allocated is used for indicating the corresponding relation between the remaining partitions to be allocated and nodes in the message system, and the remaining partitions to be allocated are unallocated partitions to be allocated in the partitions to be allocated;
The allocating the target partition to be allocated according to the allocation information of the target partition to be allocated includes:
the target partition to be allocated is deployed at a target node, wherein the target node is a node which is indicated in allocation information of the target partition to be allocated and has a corresponding relation with the target partition to be allocated;
the allocating the remaining partition to be allocated according to the allocation information of the remaining partition to be allocated includes:
the remaining partitions to be allocated are deployed at nodes with corresponding relations with the remaining partitions to be allocated;
after the target partition to be allocated is allocated according to the allocation information of the target partition to be allocated, the method further includes:
and synchronizing the information stored in the target partition to be allocated to a memory space corresponding to the target partition to be allocated in the target node.
2. The method of claim 1, wherein prior to the obtaining allocation information for the target partition to be allocated, the method further comprises:
acquiring node related information and partition information of the partition to be allocated; wherein the node-related information is information related to a node in the message system;
And generating the allocation information of the target partition to be allocated according to the node related information and the partition information.
3. The method of claim 1, wherein the target partition to be allocated is one of the partitions to be allocated;
the remaining partition to be allocated is an unallocated partition among the partitions to be allocated.
4. A method according to any one of claims 1 to 3, wherein the partition to be allocated comprises a partition deployed on a node to be deleted in the message system, the node to be deleted being a node deleted when the message system is scaled.
5. A partition allocation apparatus in a messaging system, comprising:
the first obtaining module is used for obtaining the allocation information of the target partition to be allocated, wherein the allocation information of the target partition to be allocated is used for indicating the corresponding relation between the target partition to be allocated and the nodes in the message system; the target partition to be allocated is part of the partitions to be allocated;
the allocation module is used for allocating the target partition to be allocated according to the allocation information of the target partition to be allocated;
The first obtaining module is further configured to obtain allocation information of the remaining partitions to be allocated when allocation of the target partition to be allocated is completed;
the distribution module is further configured to distribute the remaining partitions to be distributed according to the distribution information of the remaining partitions to be distributed until all the partitions to be distributed are distributed; the allocation information of the remaining partitions to be allocated is used for indicating the corresponding relation between the remaining partitions to be allocated and nodes in the message system, and the remaining partitions to be allocated are unallocated partitions to be allocated in the partitions to be allocated;
the allocation module is specifically configured to deploy the target partition to be allocated to a target node, where the target node is a node indicated in allocation information of the target partition to be allocated and has a corresponding relationship with the target partition to be allocated;
the allocation module is specifically configured to deploy the remaining partition to be allocated to a node having a corresponding relationship with the remaining partition to be allocated;
the apparatus further comprises: and the synchronization module is used for synchronizing the information stored in the target partition to be allocated to the memory space corresponding to the target partition to be allocated in the target node after the target partition to be allocated is allocated according to the allocation information of the target partition to be allocated.
6. The apparatus of claim 5, wherein the apparatus further comprises:
the second obtaining module is used for obtaining node related information and partition information of the partition to be allocated before obtaining allocation information of the target partition to be allocated; wherein the node-related information is information related to a node in the message system;
and the generation module is used for generating the allocation information of the target partition to be allocated according to the node related information and the partition information.
7. An electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface, the memory complete communication with each other through the communication bus;
the memory is used for storing a computer program;
the processor is configured to implement the method steps of any one of claims 1-4 when executing a program stored on the memory.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911356977.6A CN111142880B (en) | 2019-12-25 | 2019-12-25 | Partition allocation method and device in message system and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911356977.6A CN111142880B (en) | 2019-12-25 | 2019-12-25 | Partition allocation method and device in message system and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111142880A CN111142880A (en) | 2020-05-12 |
| CN111142880B true CN111142880B (en) | 2024-03-12 |
Family
ID=70520003
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911356977.6A Active CN111142880B (en) | 2019-12-25 | 2019-12-25 | Partition allocation method and device in message system and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111142880B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103034540A (en) * | 2012-11-16 | 2013-04-10 | 北京奇虎科技有限公司 | Distributed information system, device and coordinating method thereof |
| CN106095589A (en) * | 2016-06-30 | 2016-11-09 | 浪潮软件集团有限公司 | Partition allocation method, device and system |
| CN109656684A (en) * | 2018-12-11 | 2019-04-19 | 杭州涂鸦信息技术有限公司 | A kind of partition method of Kafka, partition system and relevant apparatus |
| CN110019240A (en) * | 2017-12-29 | 2019-07-16 | ***通信集团北京有限公司 | A kind of service data interaction method, apparatus and system |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4199444B2 (en) * | 2001-08-30 | 2008-12-17 | 日本電気株式会社 | Partition configuration change method, partition configuration change method, and partition configuration change program |
| US20180063055A1 (en) * | 2016-08-26 | 2018-03-01 | Linkedin Corporation | Repartitioning a topic in a publish-subscribe message system |
-
2019
- 2019-12-25 CN CN201911356977.6A patent/CN111142880B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103034540A (en) * | 2012-11-16 | 2013-04-10 | 北京奇虎科技有限公司 | Distributed information system, device and coordinating method thereof |
| CN106095589A (en) * | 2016-06-30 | 2016-11-09 | 浪潮软件集团有限公司 | Partition allocation method, device and system |
| CN110019240A (en) * | 2017-12-29 | 2019-07-16 | ***通信集团北京有限公司 | A kind of service data interaction method, apparatus and system |
| CN109656684A (en) * | 2018-12-11 | 2019-04-19 | 杭州涂鸦信息技术有限公司 | A kind of partition method of Kafka, partition system and relevant apparatus |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111142880A (en) | 2020-05-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11036553B2 (en) | Preempting or traversing allocated resource applications based on job priority and resource demand information | |
| US11586673B2 (en) | Data writing and reading method and apparatus, and cloud storage system | |
| CN107391629B (en) | Method, system, server and computer storage medium for data migration between clusters | |
| JP6658882B2 (en) | Control device, VNF placement destination selection method and program | |
| CN111818112B (en) | Kafka system-based message sending method and device | |
| CN110098946B (en) | Method and device for deploying virtualized network element equipment | |
| US12034608B2 (en) | Network resource management method and system, network equipment and readable storage medium | |
| US8832215B2 (en) | Load-balancing in replication engine of directory server | |
| CN112148468A (en) | Resource scheduling method and device, electronic equipment and storage medium | |
| CN113849260A (en) | Instance processing core allocation method and device | |
| CN104102725A (en) | Method and system for dynamically creating duplicates of hotspot data files in HDFS (Hadoop distributed file system) | |
| CN113986539A (en) | Method, device, electronic equipment and readable storage medium for realizing pod fixed IP | |
| CN113535087A (en) | Data processing method, server and storage system in data migration process | |
| WO2019034084A1 (en) | Method for applying for common service resource, related device, and system | |
| CN111142880B (en) | Partition allocation method and device in message system and electronic equipment | |
| CN112631766A (en) | Dynamic adjustment method and device for project environment resources | |
| CN116187247A (en) | Chip verification system, method, device and storage medium | |
| CN115328608A (en) | Kubernetes container vertical expansion adjusting method and device | |
| US11477267B2 (en) | Operating cloud-managed remote edge sites at reduced disk capacity | |
| EP3648423B1 (en) | Resource allocation method and system | |
| CN112015515B (en) | Instantiation method and device of virtual network function | |
| CN113703930A (en) | Task scheduling method, device and system and computer readable storage medium | |
| CN111459654A (en) | Server cluster deployment method, device, equipment and storage medium | |
| CN112559185B (en) | Chip resource allocation method, device, network equipment and computer storage medium | |
| CN111338752B (en) | Container adjusting method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |