CN112751891A - Kafka-based message processing method, electronic device and storage medium - Google Patents
Kafka-based message processing method, electronic device and storage medium Download PDFInfo
- Publication number
- CN112751891A CN112751891A CN201911044313.6A CN201911044313A CN112751891A CN 112751891 A CN112751891 A CN 112751891A CN 201911044313 A CN201911044313 A CN 201911044313A CN 112751891 A CN112751891 A CN 112751891A
- Authority
- CN
- China
- Prior art keywords
- topic
- monitoring thread
- message
- topic monitoring
- kafka
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
- H04L67/63—Routing a service request depending on the request content or context
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/22—Parsing or analysis of headers
Abstract
The application discloses a message processing method based on kafka, which comprises the following steps: receiving a message request sent by a kafka cluster; acquiring a topic monitoring thread according to the consumer information corresponding to the message request; according to the topic monitoring thread, acquiring metadata corresponding to the message request; the application also discloses a device for processing the messages based on kafka and a storage medium; through the embodiment of the application, in the process that the consumer acquires the messages subscribed by the consumer in the kafka cluster, the messages subscribed by the consumer are processed according to the requirements of the consumer, and the processing pressure of the kafka cluster is dispersed.
Description
Technical Field
The invention relates to the technical field of big data processing, in particular to a kafka-based message processing method, electronic equipment and a storage medium.
Background
The information obtained by the consumer through the kafka cluster comprises redundant information; in the related art, the messages of the consumption of the consumers are processed aiming at the requirements of the consumers in the kafka cluster; the method comprises the following steps: the messages are filtered, analyzed and recombined according to the requirements of the consumers, but the method occupies the computing resources of the kafka cluster, reduces the processing capacity of the kafka cluster, and solves the technical problem of how to disperse the processing pressure of the kafka cluster and process the messages of the consumers according to the requirements of the consumers.
Disclosure of Invention
The embodiment of the application provides a message processing method based on kafka, electronic equipment and a storage medium, wherein the message is processed in a kafka client jar server according to the requirements of consumers, the processing pressure of a kafka cluster is dispersed, and the processing capacity of the kafka cluster is improved.
The embodiment of the application provides a message processing method based on kafka, which comprises the following steps:
receiving a message request sent by a kafka cluster;
acquiring a topic monitoring thread according to the consumer information corresponding to the message request;
and acquiring metadata corresponding to the message request according to the topic monitoring thread.
In the foregoing solution, the obtaining a topic monitoring thread includes:
searching a topic monitoring thread dictionary according to the consumer information;
acquiring the topic monitoring thread from the topic monitoring thread dictionary under the condition that the topic monitoring thread is found in the topic monitoring thread dictionary;
or, under the condition that the topic monitoring thread is not found in the topic monitoring thread dictionary, creating the topic monitoring thread based on the consumer information;
and adding the topic monitoring thread into the topic monitoring thread dictionary.
In the foregoing solution, the obtaining metadata corresponding to the message request includes:
acquiring configuration parameters of the metadata based on the topic monitoring thread;
and acquiring a field of the topic message in the kafka cluster according to the configuration parameters to obtain the metadata.
In the above scheme, the method further comprises:
splicing the metadata into a message string;
and sending the message string to the client.
In the foregoing solution, the configuration parameters include:
at least one of analysis parameters, filtering parameters and analysis parameters.
In the above scheme, the method further comprises:
and receiving at least one topic monitoring thread sent by the kafka cluster, and adding the at least one topic monitoring thread into the topic monitoring thread dictionary.
The embodiment of the present application further provides a message processing apparatus based on kafka, including:
the receiving module is used for receiving the message request sent by the kafka cluster;
the acquisition module is used for acquiring a topic monitoring thread according to the consumer information corresponding to the message request; and acquiring metadata corresponding to the message request according to the topic monitoring thread.
In the above scheme, the apparatus further comprises:
the searching module is used for searching the topic monitoring thread dictionary according to the consumer information;
the obtaining module is further configured to obtain the topic monitoring thread from the topic monitoring thread dictionary under the condition that the topic monitoring thread is found in the topic monitoring thread dictionary;
the creating module is used for creating the topic monitoring thread based on the consumer information under the condition that the topic monitoring thread is not found in the topic monitoring thread dictionary;
and the storage module is used for adding the topic monitoring thread into the topic monitoring thread dictionary.
In the foregoing solution, the obtaining module is further configured to: acquiring configuration parameters of the metadata based on the topic monitoring thread; and acquiring a field of the topic message in the kafka cluster according to the configuration parameters to obtain the metadata.
In the above scheme, the apparatus further comprises:
the splicing module is used for splicing the metadata into a message string;
and the sending module is used for sending the message string to the client.
In the foregoing solution, the configuration parameters include:
at least one of analysis parameters, filtering parameters and analysis parameters.
In the above scheme, the receiving module is further configured to receive at least one topoic monitoring thread sent by the kafka cluster;
the storage module is further configured to add the at least one topic monitoring thread to the topic monitoring thread dictionary.
The embodiment of the present application further provides a kafka-based message processing apparatus, which includes a memory, a processor, and an executable program stored on the memory and capable of being executed by the processor, and when the processor executes the executable program, the steps of the kafka-based message processing method are implemented.
The message processing method based on kafka provided by the application comprises the steps of receiving a message request sent by a kafka cluster; acquiring a topic monitoring thread according to the consumer information corresponding to the message request; and acquiring metadata corresponding to the message request according to the topic monitoring thread. In the process that a consumer acquires a message through a kafka cluster, the operation of processing the topic message in the kafka cluster according to the requirement of the consumer is transferred to a server, so that the server can process the topic message in the kafka cluster uniformly according to the requirement of the consumer, the processing pressure of the kafka cluster is dispersed, and the processing capacity of the kafka cluster is improved. In the embodiment of the application, the server receives at least one topic monitoring thread sent by the kafka cluster, and adds the at least one topic monitoring thread to the topic monitoring thread dictionary to realize the unified management of the topic monitoring threads.
Drawings
Fig. 1 is a schematic overall structure diagram of a message processing method based on kafka according to an embodiment of the present invention;
fig. 2 is a first schematic flowchart of an alternative kafka-based message processing method according to an embodiment of the present application;
FIG. 3 is a schematic diagram of an alternative process for obtaining a topic monitoring thread according to an embodiment of the present application;
fig. 4 is a schematic diagram illustrating an alternative flow of obtaining metadata corresponding to a message request according to an embodiment of the present application;
fig. 5 is a schematic diagram of an optional process of obtaining a field of a topic message in a kafka cluster according to the configuration parameter to obtain the metadata according to the embodiment of the present application;
FIG. 6 is a schematic diagram illustrating an alternative flow of a message processing method based on kafka according to an embodiment of the present application;
fig. 7 is a schematic diagram of an alternative structure of a message processing apparatus based on kafka according to an embodiment of the present application.
Detailed Description
The present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
kafka refers to a distributed message queue system. The main target scene is to construct a data processing framework for processing user behaviors such as massive system logs and web pages and website operation statistical indexes. In a big data message processing demand scenario, a producer-consumer model is generally used to asynchronously process message data, improving system stability and throughput.
For some systems that need to process a large number of messages in real time, such as a telecom operator network management system, a producer produces messages that contain redundant information, and there is no uniform structure, before the messages that contain redundant information and are unstructured are sent to corresponding consumers through a kafka cluster, which needs to process the messages that contain redundant information and are unstructured according to the needs of the consumers. The method comprises at least one of removing redundant information, structuring, filtering, analyzing and recombining. However, the above operations temporarily occupy the computing resources of the kafka cluster, and the processing capacity of the kafka cluster is reduced.
To solve the above problem, a common solution is: the consumer module comprises at least one of a filtering module, a local message queue caching module and a service processing module. The method comprises the steps that a producer submits messages to a kafka cluster, the kafka cluster stores the messages in corresponding servers according to topic types, a filtering module in a consumer module consumes corresponding subject messages to the kafka cluster and filters the messages, when the messages are not in the filtering module, the filtering module adds the messages to a local message queue cache module, and finally a service processing module consumes the messages from the local message queue cache to perform service processing.
In the above scheme, the number of queues in the local message queue cache module may affect the filtering effect, and in the local message queue cache module, the filtering speed may be slowed down due to the excessive number of queues; if the number of queues is too small, incomplete filtering can be caused; at the same time, a system restart or downtime may also result in lost messages. Moreover, the above scheme of transferring the message processing operation to the local client of the consumer is contrary to the original intention of subscribing to process the message in the kafka cluster.
Another common solution is: the mode information is sent with the message as part of the message header information, using the serialization format (Apache Avro) as the message transport format at the time of message transmission. The message mode change notification is realized through a message mode registration service, for a message generated by a producer, before the message is sent to kafka, the message is registered in a Schema register (Schema register) and generates a corresponding mode number, and a number corresponding to the message mode is transmitted along with the message, the number corresponding to the message mode is acquired from the mode registration service, when the consumer acquires the message through kafka, the message metadata information is firstly analyzed and the message mode number is taken, if the application memory has the mode information corresponding to the mode number, the message is analyzed according to the mode information, if the memory does not have the mode information, the application initiates a request to acquire the mode information from the Schema register according to the mode number, then the message is analyzed according to the mode information, and the mode information is cached.
In the above scheme, the message format is limited to a JavaScript Object notation (json) protocol format, and the capabilities of filtering and analyzing the message cannot be embodied.
Based on the problems existing in the current method for processing the message based on the kafka, the application provides a method for processing the message based on the kafka, which can solve the technical problems and the defects which cannot be solved in the prior art.
Fig. 1 shows the overall framework of the kafka-based message processing system of the present application, which will be described in terms of various parts.
In some embodiments, a producer cluster is a cluster of one or more message producers that sends messages to a kafka cluster.
In some embodiments, a kafka cluster consists of one or more servers, each storing messages on a certain topic (topic).
In some embodiments, the kafka cluster includes a topic storage server, and after the producer produces topic, the producer cluster sends various application-generated messages to the corresponding topic storage server for various consumers to use.
In some embodiments, a rule configuration topoc server is included in the kafka cluster for parsing producer production topoc message body format, parsing, filtering, parsing parameters, etc. for use by consumers in consuming messages according to the producer and consumer agreement.
In some embodiments, the topic message processing rule configuration end is configured to generate a topic monitoring thread according to a topic message basic attribute and a topic message processing rule set produced by a consumer, and send the topic monitoring thread to the kafka server.
In some embodiments, a consumer cluster is a cluster of one or more message consumers, subscribing to messages from a kafka cluster.
In some embodiments, the kafka client jar is configured to receive at least one topic monitoring thread sent by a server, and add the at least one topic monitoring thread to a topic monitoring thread dictionary; and the method is also used for processing the topic message in the kafka cluster corresponding to the consumer according to the topic monitoring thread and the message request of the consumer.
In some embodiments, the Kafka client jar refers to Apache Kafka project engineering source code, such as: kafka-0.10.2.0-src.tgz version, custom development, and packaging to obtain kafka-clients-0.10.2.0.jar package. Where 0.10.2.0 is a version number. Usually kafka client jar is introduced by the application layer service in the form of project dependent packages.
Fig. 2 is a schematic flow chart showing an alternative kafka-based message processing method according to the present application, which will be described according to various steps.
And step S101, receiving a message request sent by the kafka cluster.
In some embodiments, the server receives a message request sent by the kafka cluster. The message request is a request sent by a consumer to a kafka cluster and used for subscribing the consumer message; and after receiving the message request, the kafka cluster sends the message request to the server.
In some embodiments, the server is kafka client jar.
And S102, acquiring a topic monitoring thread according to the consumer information corresponding to the message request.
In some embodiments, the server acquires the topic monitoring thread according to the corresponding consumer information requested by the message sent by the kafka cluster.
In some embodiments, the consumer information comprises: at least one of a consumer authentication identifier, a consumer username, a request topic, and a serialization method.
In some embodiments, as shown in fig. 1, after the producer cluster produces topic, the at least one topic configures basic information in a topic message processing rule configuration module, and the rule configuration topic module generates at least one topic monitoring thread related to the at least one topic and sends the at least one topic monitoring thread to the kafka cluster. And the kafka cluster sends at least one topic monitoring thread to the server, and the server adds the at least one topic monitoring thread to the topic monitoring thread dictionary. The at least one topic monitoring thread is used for processing subscription messages of the consumers.
In some embodiments, the name of the topic monitoring thread comprises: at least one of a prefix, a consumer authentication token, a topic name. The topic monitoring thread comprises at least one of basic structure information, configuration parameters and other service processing information of a message body of the topic.
In some embodiments, the obtaining of the topoic monitoring thread includes steps S201 to S204, and fig. 3 shows an optional flowchart of obtaining the topoic monitoring thread provided in this embodiment, which will be described according to each step.
And step S201, searching a topic monitoring thread dictionary according to the consumer information.
In some embodiments, the server looks up a topic monitoring thread associated with the consumer information in a topic monitoring thread dictionary based on at least one of a consumer authentication identifier, a consumer username, a request topic, and a serialization method.
In some embodiments, step S202 is performed when the topic monitoring thread is found in the topic monitoring thread dictionary; and step S203 is performed when the topic monitoring thread is not found in the topic monitoring thread dictionary.
Step S202, the topic monitoring thread is obtained from the topic monitoring thread dictionary.
In some embodiments, the topic monitoring thread dictionary includes a topic monitoring thread related to the consumer information; and the server acquires the topic monitoring thread under the condition that the topic monitoring thread is found in the topic monitoring thread dictionary according to the consumer information.
Step S203, based on the consumer information, the topic monitoring thread is created.
In some embodiments, the server creates a topic monitoring thread associated with the consumer and the topic based on at least one of a consumer authentication identification, a consumer username, a request topic, and a serialization method.
And step S204, adding the topic monitoring thread into the topic monitoring thread dictionary.
In some embodiments, in the event that the topic monitoring thread includes configuration parameters, adding the topic monitoring thread to the topic monitoring thread dictionary; and under the condition that the topic monitoring thread does not comprise configuration parameters, the topic monitoring thread does not need to be added into the topic monitoring thread dictionary.
In some embodiments, the server is kafka client jar.
Thus, the server searches the topic monitoring thread related to the consumer in the topic monitoring thread dictionary according to the consumer information; acquiring the topic monitoring thread under the condition that the topic monitoring thread is found in the topic monitoring thread dictionary; under the condition that the topic monitoring thread is not found in the topic monitoring thread dictionary, the topic monitoring thread is created according to the consumer information, and the created topic monitoring thread is added into the topic monitoring thread dictionary; and ensuring that the server can subsequently acquire the metadata corresponding to the consumption request according to the demand of the consumer.
And step S103, acquiring metadata corresponding to the message request according to the topic monitoring thread.
In some embodiments, the message subscribed by the consumer corresponding to the message request includes redundant information and is unstructured, and the server needs to process the subscription information of the consumer according to the topic monitoring thread to obtain metadata corresponding to the message request. Specifically, the method includes steps S301 to S302.
Fig. 4 is a schematic diagram illustrating an alternative flow of obtaining metadata corresponding to a message request, which will be described according to various steps.
Step S301, based on the topic monitoring thread, obtaining the configuration parameters of the metadata.
In some embodiments, the server obtains configuration parameters of the metadata included in the topic monitoring thread based on the topic monitoring thread.
In some embodiments, the configuration parameters include: at least one of analysis parameters, filtering parameters and analysis parameters.
In some embodiments, the configuration parameters are structured as: taking English symbols' as separators, wherein the first bit of the configuration parameter is a topic message body separator used for separating different messages; the second bit of the configuration parameter is an analytic parameter; the third bit of the configuration parameters is a filtering parameter and an analysis parameter; the fourth to the last bit of the configuration parameter is a specific message processing method corresponding to the filtering parameter and the analysis parameter.
For example, in a topic monitoring thread obtained according to the consumer information, configuration parameters included in the topic monitoring thread are:
`；`1,3,4,5,6,7,8`1,2,3,4`test＝＝$5`1-8$4`$5+$6>>$5`$6/($7+$8)>>$4
the first bit of the configuration parameter is "; ", indicates the topic message body divider, with different topic message bodies passing"; "dividing; the second bit of the configuration parameter is "1, 3,4,5,6,7, 8", which indicates an analysis parameter, that is, a field with a sequence number of 1,3,4,5,6,7,8 in the topic message in the kafka cluster is obtained; the third bit of the configuration parameter is "1, 2,3, 4", which indicates that the filtering parameter is mode 1 and mode 2, and the analyzing parameter is mode 3 and mode 4. In this embodiment, the specific processing method in the mode 1 corresponds to the 4 th bit of the configuration parameter, that is, "test ═ 5", that is, a test (test) field is a field with a sequence number of 5 in a topic message in the kafka cluster; the specific processing method of rule 2 corresponds to the 5 th bit of the configuration parameter, namely "1-8 $ 4", namely, the filtering is performed on the value ranging from 1 to 8 in the field with the sequence number of 4 in the topic message in the kafka cluster; the specific processing method of the mode 3 corresponds to the 6 th bit of the configuration parameter, namely "$ 5+ $6> > $ 5", namely, the fields with sequence numbers of 5 and 6 in the topic message in the kafka cluster are merged to form a field with a sequence number of 5 in the message metadata; the specific processing method of the mode 4 corresponds to the 7 th bit of the configuration parameter, namely "$ 6/($7+ $8) > $ 4", namely, the sum of the field with the sequence number of 7 and the field with the sequence number of 8 is divided by the field with the sequence number of 6 in the topic message in the kafka cluster, and the obtained fields are combined into the field with the sequence number of 4 in the message metadata.
Step S302, obtaining the fields of the topic message in the kafka cluster according to the configuration parameters to obtain the metadata.
In some embodiments, the topic message in the kafka cluster refers to a message subscribed by the consumer to the kafka cluster, and the kafka cluster sends the topic message in the kafka cluster to the server.
In some embodiments, the metadata is: and processing the topic message in the kafka cluster according to the configuration parameters to obtain data.
In some embodiments, the server obtains the field of the topic message in the kafka cluster according to the configuration parameter to obtain the metadata, including step S401 to step S403.
Fig. 5 is a schematic diagram illustrating an alternative flow of the server obtaining the fields of the topic message in the kafka cluster according to the configuration parameters to obtain the metadata, which will be described according to various steps.
Step S401, according to the analysis parameters, obtaining the fields of the topic message in the kafka cluster.
In some embodiments, the topic message in the kafka cluster refers to a message subscribed by the consumer to the kafka cluster, and the kafka cluster sends the topic message in the kafka cluster to the server. The topic message in the kafka cluster contains several fields, each with a unique sequence number.
In some embodiments, obtaining the fields of the topic message in the kafka cluster according to the parsing parameters includes: and the server acquires analysis parameters in the configuration parameters of the metadata according to the topic monitoring thread, and acquires fields of topic messages in the kafka cluster according to the analysis parameters.
For example, the topic message in the kafka cluster corresponding to the consumer includes 15 fields, and the sequence numbers are 1 to 15 in sequence; the server monitors the thread based on the topic, and the obtained configuration parameters are as follows:
`；`1,3,4,5,6,7,8`1,2,3,4`test＝＝$5`1-8$4`$5+$6>>$5`$6/($7+$8)>>$4
the configuration parameters comprise analysis parameters: and 1,3,4,5,6,7, 8', the server obtains fields with sequence numbers of 1,3,4,5,6,7,8 in the topic message in the kafka cluster according to the parsing parameters.
And step S402, filtering the fields of the topic message in the kafka cluster according to the filtering parameters.
In some embodiments, the server filters fields of a topic message in the kafka cluster according to a filtering parameter, including: and the server acquires a filtering parameter in the configuration parameters of the metadata according to the topic monitoring thread, and filters a field of a topic message in the kafka cluster according to the filtering parameter.
For example, the topic message in the kafka cluster corresponding to the consumer includes 15 fields, and the sequence numbers are 1 to 15 in sequence; the server monitors the thread based on the topic, and the obtained configuration parameters are as follows:
`；`1,3,4,5,6,7,8`1,2,3,4`test＝＝$5`1-8$4`$5+$6>>$5`$6/($7+$8)>>$4
the configuration parameters include the following filtering parameters: 1, 2'; the corresponding filtering scheme corresponding to filtering parameter 1 is: test is $5, namely the value of the field with the sequence number of 5 is test; the filtering scheme corresponding to filtering parameter 2 is: 1-8$4, i.e., filtering values ranging from 1 to 8 in the field with sequence number 4.
And step S403, analyzing the fields of the topic message in the kafka cluster according to the analysis parameters.
In some embodiments, the server analyzes the fields of the topic message in the kafka cluster according to the analysis parameters, including: and the server acquires analysis parameters in the configuration parameters of the metadata according to the topic monitoring thread, and analyzes fields of the topic message in the kafka cluster according to the analysis parameters.
For example, the topic message in the kafka cluster corresponding to the consumer includes 15 fields, and the sequence numbers are 1 to 15 in sequence; the server monitors the thread based on the topic, and the obtained configuration parameters are as follows:
`；`1,3,4,5,6,7,8`1,2,3,4`test＝＝$5`1-8$4`$5+$6>>$5`$6/($7+$8)>>$4
the configuration parameters include analysis parameters: 3, 4'; the corresponding analysis protocol corresponding to analysis parameter 3 is: merging the field with the sequence number of 5 and the field with the sequence number of 6, and marking the merged field as a newly formed field with the sequence number of 5; the analysis protocol corresponding to analysis parameter 4 was: the value in the field with sequence number 6 is divided by the sum of the value in the field with sequence number 7 and the value in the field with sequence number 8, and the field obtained by the above operation is marked as a newly composed field with sequence number 4.
Thus, the server processes the first message of the consumer according to the configuration parameters contained in the topic monitoring thread. The first message is analyzed and/or filtered and/or analyzed according to the requirements of different users, and repeated customized development of the first message by a consumer client is avoided.
Therefore, the processing flow of the kafka client jar server is improved, the kafka client jar acquires the topic monitoring thread according to the consumer information, analyzes the first message of the consumer according to the configuration parameters of the topic monitoring thread, and sends the analyzed first message to the consumer, so that the customized processing pressure in the kafka cluster is dispersed to each client cluster, and the sheep swarm effect is avoided. Meanwhile, kafkalclient jar uniformly processes message analysis processing rules of different consumers, and repeated customized development of consumer clients is avoided.
Therefore, the kafka cluster sends the consumption request of the consumer to the server, the server searches a topic monitoring thread dictionary according to the consumer information, and acquires a topic monitoring thread related to the consumer; according to the topic monitoring thread, metadata corresponding to the message request is obtained, the topic message processing process in the kafka cluster according to the requirements of the consumer is carried out in the server, so that the topic message in the kafka cluster corresponding to the consumer can be processed uniformly according to different requirements of the consumer, the processing pressure of the kafka cluster is dispersed, and the processing capacity of the kafka cluster is improved; meanwhile, the scheme provided by the embodiment of the application processes the topic message in the kafka cluster related to the consumer according to the analysis, filtering and analyzing requirements of the consumer, and avoids the kafka cluster side sheep swarm effect caused by excessive consumers.
Fig. 6 shows a kafka-based message processing method provided in an embodiment of the present application, and will be described according to various steps.
And step S501, performing user identity authentication on the consumer and connecting the kafka cluster.
In some embodiments, the server authenticates the user identity card of the consumer, and after the authentication is passed, the consumer connects to the kafka cluster.
In some embodiments, the consumer may be user authenticated using network authentication protocol (kerberos) authentication.
Step S502, initializing initial information of the consumer.
In some embodiments, the server initializes initial information for the consumer.
In some embodiments, the initial information of the consumer includes: at least one of identity authentication subject identification of the consumer, user name of the consumer, topic name request of the consumer and serialization mode.
Step S503, receiving the message request sent by the kafka cluster.
In some embodiments, the server receives a message request sent by the kafka cluster. The message request is a request sent by a consumer to a kafka cluster and used for subscribing the consumer message; and after receiving the message request, the kafka cluster sends the message request to the server.
In some embodiments, the server is kafka client jar.
Step S504, receiving at least one topic monitoring thread sent by the kafka cluster, and adding the at least one topic monitoring thread into the topic monitoring thread dictionary.
In some embodiments, after the producer cluster produces topic, the at least one topic configures basic information in a topic message processing rule configuration module, and the rule configuration topic module generates at least one topic monitoring thread related to the at least one topic and sends the at least one topic monitoring thread to the kafka cluster. And the kafka cluster sends at least one topic monitoring thread to the server, and the server adds the at least one topic monitoring thread to the topic monitoring thread dictionary. The at least one topic monitoring thread is used for processing subscription messages of the consumers.
In some embodiments, the name of the topic monitoring thread comprises: at least one of a prefix, a consumer authentication token, a topic name. The topic monitoring thread comprises at least one of basic structure information, configuration parameters and other service processing information of a message body of the topic.
And step S505, searching a topic monitoring thread dictionary according to the consumer information.
In some embodiments, the server looks up a topic monitoring thread associated with the consumer information in a topic monitoring thread dictionary based on at least one of a consumer authentication identifier, a consumer username, a request topic, and a serialization method.
In some embodiments, step S506 is performed when the topic monitoring thread is found in the topic monitoring thread dictionary; and step S507 is performed when the topic monitoring thread is not found in the topic monitoring thread dictionary.
Step S506, the topic monitoring thread is obtained from the topic monitoring thread dictionary.
In some embodiments, the topic monitoring thread dictionary includes a topic monitoring thread related to the consumer information; and the server acquires the topic monitoring thread under the condition that the topic monitoring thread is found in the topic monitoring thread dictionary according to the consumer information.
In some embodiments, the server performs step S509 after acquiring the topic monitoring thread.
Step S507, creating the topoic monitoring thread based on the consumer information.
In some embodiments, the server creates a topic monitoring thread associated with the consumer and the topic based on at least one of a consumer authentication identification, a consumer username, a request topic, and a serialization method.
Step S508, adding the topic monitoring thread into the topic monitoring thread dictionary.
In some embodiments, in the event that the topic monitoring thread includes configuration parameters, adding the topic monitoring thread to the topic monitoring thread dictionary; and under the condition that the topic monitoring thread does not comprise configuration parameters, the topic monitoring thread does not need to be added into the topic monitoring thread dictionary.
In some embodiments, in the case that the topic monitoring thread does not include configuration parameters, the server obtains the most recent topic monitoring thread from the kafka cluster.
In some embodiments, the server performs step S509 after creating and/or acquiring the topoic monitoring thread.
Step S509, acquiring configuration parameters of the metadata based on the topic monitoring thread.
In some embodiments, the server obtains configuration parameters of the metadata included in the topic monitoring thread based on the topic monitoring thread.
In some embodiments, the configuration parameters include: at least one of analysis parameters, filtering parameters and analysis parameters.
In some embodiments, the configuration parameters are structured as: taking English symbols' as separators, wherein the first bit of the configuration parameter is a topic message body separator used for separating different messages; the second bit of the configuration parameter is an analytic parameter; the third bit of the configuration parameters is a filtering parameter and an analysis parameter; the fourth to the last bit of the configuration parameter is a specific message processing method corresponding to the filtering parameter and the analysis parameter.
For example, in a topic monitoring thread obtained according to the consumer information, configuration parameters included in the topic monitoring thread are:
`；`1,3,4,5,6,7,8`1,2,3,4`test＝＝$5`1-8$4`$5+$6>>$5`$6/($7+$8)>>$4
the first bit of the configuration parameter is "; ", indicates the topic message body divider, with different topic message bodies passing"; "dividing; the second bit of the configuration parameter is "1, 3,4,5,6,7, 8", which indicates an analysis parameter, that is, a field with a sequence number of 1,3,4,5,6,7,8 in the topic message in the kafka cluster is obtained; the third bit of the configuration parameter is "1, 2,3, 4", which indicates that the filtering parameter is mode 1 and mode 2, and the analyzing parameter is mode 3 and mode 4. In this embodiment, the specific processing method in the mode 1 corresponds to the 4 th bit of the configuration parameter, that is, "test ═ 5", that is, test is a field with a sequence number of 5 in a topic message in the kafka cluster; the specific processing method of rule 2 corresponds to the 5 th bit of the configuration parameter, namely "1-8 $ 4", namely, the filtering is performed on the value ranging from 1 to 8 in the field with the sequence number of 4 in the topic message in the kafka cluster; the specific processing method of the mode 3 corresponds to the 6 th bit of the configuration parameter, namely "$ 5+ $6> > $ 5", namely, the fields with sequence numbers of 5 and 6 in the topic message in the kafka cluster are merged to form a field with a sequence number of 5 in the message metadata; the specific processing method of the mode 4 corresponds to the 7 th bit of the configuration parameter, namely "$ 6/($7+ $8) > $ 4", namely, the sum of the field with the sequence number of 7 and the field with the sequence number of 8 is divided by the field with the sequence number of 6 in the topic message in the kafka cluster, and the obtained fields are combined into the field with the sequence number of 4 in the message metadata.
And step S510, acquiring a topic message field in the kafka cluster according to the analysis parameters.
In some embodiments, the topic message in the kafka cluster refers to a message subscribed by the consumer to the kafka cluster, and the kafka cluster sends the topic message in the kafka cluster to the server. The topic message in the kafka cluster contains several fields, each with a unique sequence number.
In some embodiments, obtaining the fields of the topic message in the kafka cluster according to the parsing parameters includes: and the server acquires analysis parameters in the configuration parameters of the metadata according to the topic monitoring thread, and acquires fields of topic messages in the kafka cluster according to the analysis parameters.
For example, the topic message in the kafka cluster corresponding to the consumer includes 15 fields, and the sequence numbers are 1 to 15 in sequence; the server monitors the thread based on the topic, and the obtained configuration parameters are as follows:
`；`1,3,4,5,6,7,8`1,2,3,4`test＝＝$5`1-8$4`$5+$6>>$5`$6/($7+$8)>>$4
the configuration parameters comprise analysis parameters: and 1,3,4,5,6,7, 8', the server obtains fields with sequence numbers of 1,3,4,5,6,7,8 in the topic message in the kafka cluster according to the parsing parameters.
And step S511, filtering the fields of the topic message in the kafka cluster according to the filtering parameters.
In some embodiments, the server filters fields of a topic message in the kafka cluster according to a filtering parameter, including: and the server acquires a filtering parameter in the configuration parameters of the metadata according to the topic monitoring thread, and filters a field of a topic message in the kafka cluster according to the filtering parameter.
For example, the topic message in the kafka cluster corresponding to the consumer includes 15 fields, and the sequence numbers are 1 to 15 in sequence; the server monitors the thread based on the topic, and the obtained configuration parameters are as follows:
`；`1,3,4,5,6,7,8`1,2,3,4`test＝＝$5`1-8$4`$5+$6>>$5`$6/($7+$8)>>$4
the configuration parameters include the following filtering parameters: 1, 2'; the corresponding filtering scheme corresponding to filtering parameter 1 is: test is $5, namely the value of the field with the sequence number of 5 is test; the filtering scheme corresponding to filtering parameter 2 is: 1-8$4, i.e., filtering values ranging from 1 to 8 in the field with sequence number 4.
And step S512, analyzing the fields of the topic message in the kafka cluster according to the analysis parameters.
In some embodiments, the server analyzes the fields of the topic message in the kafka cluster according to the analysis parameters, including: and the server acquires analysis parameters in the configuration parameters of the metadata according to the topic monitoring thread, and analyzes fields of the topic message in the kafka cluster according to the analysis parameters.
For example, the topic message in the kafka cluster corresponding to the consumer includes 15 fields, and the sequence numbers are 1 to 15 in sequence; the server monitors the thread based on the topic, and the obtained configuration parameters are as follows:
`；`1,3,4,5,6,7,8`1,2,3,4`test＝＝$5`1-8$4`$5+$6>>$5`$6/($7+$8)>>$4
the configuration parameters include analysis parameters: 3, 4'; the corresponding analysis protocol corresponding to analysis parameter 3 is: merging the field with the sequence number of 5 and the field with the sequence number of 6, and marking the merged field as a newly formed field with the sequence number of 5; the analysis protocol corresponding to analysis parameter 4 was: the value in the field with sequence number 6 is divided by the sum of the value in the field with sequence number 7 and the value in the field with sequence number 8, and the field obtained by the above operation is marked as a newly composed field with sequence number 4.
Step S513, concatenates the metadata into a message string.
In some embodiments, the server obtains the metadata corresponding to the message request according to the topic monitoring thread, and the metadata exists in a field set manner.
For example, the topic message in the kafka cluster corresponding to the consumer includes 15 fields, and the sequence numbers are 1 to 15 in sequence; the server monitors the thread based on the topic, and the obtained configuration parameters are as follows:
`；`1,3,4,5,6,7,8`
and the server extracts fields with sequence numbers of 1,3,4,5,6,7 and 8 in the topic message in the kafka cluster based on the configuration parameters, wherein the set of fields with sequence numbers of 1,3,4,5,6,7 and 8 is metadata.
In some embodiments, the server concatenates the metadata into a message string, including: the server concatenates the fields included in the metadata into a message string.
For example, the server concatenates fields with sequence numbers 1,3,4,5,6,7,8 in the metadata into a message string.
Step S514, the message string is sent to the client.
In some embodiments, the client is a client of the consumer.
In some embodiments, the server sends the message string to a client of the consumer.
Therefore, the scheme provided by the embodiment of the application can realize that the server uniformly processes topic messages in the kafka cluster corresponding to the consumer according to different requirements of the consumer, disperse the processing pressure of the kafka cluster and improve the processing capacity of the kafka cluster; the scheme provided by the embodiment of the application processes the topic message in the kafka cluster related to the consumer according to the analysis, filtering and analysis requirements of the consumer, and avoids the kafka cluster side flock effect caused by excessive consumers; meanwhile, according to the scheme provided by the embodiment of the application, at least one topoc monitoring thread sent to the server by the kafka cluster is received, and the at least one topoc monitoring thread is added to the topoc monitoring thread dictionary, so that unified management of the topoc monitoring threads is realized.
Fig. 7 is a schematic structural diagram of a message processing apparatus based on kafka according to an embodiment of the present application, including:
a receiving module 701, configured to receive a message request sent by a kafka cluster. The message request is a request sent by a consumer to a kafka cluster and used for subscribing the consumer message; and after receiving the message request, the kafka cluster sends the message request to the server.
An obtaining module 702, configured to obtain a topic monitoring thread according to the consumer information corresponding to the message request; and acquiring metadata corresponding to the message request according to the topic monitoring thread.
In some embodiments, the obtaining the topic monitoring thread comprises: the obtaining module 702 searches a topic monitoring thread related to the consumer information in a topic monitoring thread dictionary according to at least one of a consumer identity authentication identifier, a consumer user name, a request topic and a serialization method. In a case where the topic monitoring thread is found in the topic monitoring thread dictionary, the obtaining module 702 obtains the topic monitoring thread from the topic monitoring thread dictionary.
A creating module 704, configured to create the topic monitoring thread based on the consumer information when the topic monitoring thread is not found in the topic monitoring thread dictionary.
In some embodiments, the name of the topic monitoring thread comprises: at least one of a prefix, a consumer authentication token, a topic name. The topic monitoring thread comprises at least one of basic structure information, configuration parameters and other service processing information of a message body of the topic.
And the searching module 703 is configured to search the topic monitoring thread dictionary according to the consumer information.
An obtaining module 702, further configured to: acquiring configuration parameters of the metadata based on the topic monitoring thread; and acquiring a field of the topic message in the kafka cluster according to the configuration parameters to obtain the metadata.
A concatenation module 706 configured to concatenate the metadata into a message string.
A sending module 707, configured to send the message string to the client.
The receiving module 701 is further configured to receive at least one topic monitoring thread sent by the kafka cluster.
The storage module 705 is further configured to add the at least one topoic monitoring thread to the topoic monitoring thread dictionary.
Those of ordinary skill in the art will understand that: all or part of the steps of implementing the above method embodiments may be accomplished by hardware related to program commands, where the program may be stored in a storage medium, and when the program is executed, when receiving a notification message based on a second application during the running process of a first application, the program responds to the notification message in a first area on a screen of an electronic device; the first area is smaller than a corresponding area of the input method application loaded when the second application is independently run on the screen of the electronic equipment. And the aforementioned storage medium includes: various media capable of storing program codes, such as a removable Memory device, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, and an optical disk.
Alternatively, the integrated unit of the present invention may be stored in a computer-readable storage medium if it is implemented in the form of a software functional module and sold or used as a separate product. Based on such understanding, the technical solutions of the embodiments of the present invention may be essentially or partially implemented in the form of a software product, which is stored in a storage medium and includes several commands for enabling a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the methods described in the embodiments of the present invention. And the aforementioned storage medium includes: a removable storage device, a ROM, a RAM, a magnetic or optical disk, or various other media that can store program code.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.
Claims (14)
1. A method for processing messages based on kafka, the method comprising:
receiving a message request sent by a kafka cluster;
acquiring a topic monitoring thread according to the consumer information corresponding to the message request;
and acquiring metadata corresponding to the message request according to the topic monitoring thread.
2. The method of claim 1, wherein obtaining the topic monitoring thread comprises:
searching a topic monitoring thread dictionary according to the consumer information;
acquiring the topic monitoring thread from the topic monitoring thread dictionary under the condition that the topic monitoring thread is found in the topic monitoring thread dictionary;
or, under the condition that the topic monitoring thread is not found in the topic monitoring thread dictionary, creating the topic monitoring thread based on the consumer information;
and adding the topic monitoring thread into the topic monitoring thread dictionary.
3. The method of claim 1, wherein the obtaining metadata corresponding to the message request comprises:
acquiring configuration parameters of the metadata based on the topic monitoring thread;
and acquiring a field of the topic message in the kafka cluster according to the configuration parameters to obtain the metadata.
4. The method of claim 3, further comprising:
splicing the metadata into a message string;
and sending the message string to the client.
5. The method of claim 3, wherein the configuration parameters comprise:
at least one of analysis parameters, filtering parameters and analysis parameters.
6. The method of claim 2, further comprising:
and receiving at least one topic monitoring thread sent by the kafka cluster, and adding the at least one topic monitoring thread into the topic monitoring thread dictionary.
7. A kafka-based message processing apparatus, comprising:
the receiving module is used for receiving the message request sent by the kafka cluster;
the acquisition module is used for acquiring a topic monitoring thread according to the consumer information corresponding to the message request; and acquiring metadata corresponding to the message request according to the topic monitoring thread.
8. The apparatus of claim 7, further comprising:
the searching module is used for searching the topic monitoring thread dictionary according to the consumer information;
the obtaining module is further configured to obtain the topic monitoring thread from the topic monitoring thread dictionary under the condition that the topic monitoring thread is found in the topic monitoring thread dictionary;
the creating module is used for creating the topic monitoring thread based on the consumer information under the condition that the topic monitoring thread is not found in the topic monitoring thread dictionary;
and the storage module is used for adding the topic monitoring thread into the topic monitoring thread dictionary.
9. The apparatus of claim 7, wherein the obtaining module is further configured to: acquiring configuration parameters of the metadata based on the topic monitoring thread; and acquiring a field of the topic message in the kafka cluster according to the configuration parameters to obtain the metadata.
10. The apparatus of claim 9, further comprising:
the splicing module is used for splicing the metadata into a message string;
and the sending module is used for sending the message string to the client.
11. The apparatus of claim 9, wherein the configuration parameters comprise:
at least one of analysis parameters, filtering parameters and analysis parameters.
12. The apparatus of claim 8,
the receiving module is further configured to receive at least one topoic monitoring thread sent by the kafka cluster;
the storage module is further configured to add the at least one topic monitoring thread to the topic monitoring thread dictionary.
13. A storage medium having stored thereon an executable program, wherein the executable program, when executed by a processor, implements the steps of the kafka-based message processing method of any one of claims 1 to 6.
14. A kafka-based message processing apparatus comprising a memory, a processor, and an executable program stored on the memory and executable by the processor, wherein the processor executes the executable program to perform the steps of the kafka-based message processing method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911044313.6A CN112751891B (en) | 2019-10-30 | 2019-10-30 | Kafka-based message processing method, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911044313.6A CN112751891B (en) | 2019-10-30 | 2019-10-30 | Kafka-based message processing method, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112751891A true CN112751891A (en) | 2021-05-04 |
| CN112751891B CN112751891B (en) | 2022-06-28 |
Family
ID=75640532
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911044313.6A Active CN112751891B (en) | 2019-10-30 | 2019-10-30 | Kafka-based message processing method, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112751891B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115242787A (en) * | 2022-07-22 | 2022-10-25 | 恒生电子股份有限公司 | Message processing system and method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106953901A (en) * | 2017-03-10 | 2017-07-14 | 重庆邮电大学 | A kind of trunked communication system and its method for improving message transmission performance |
| CN107026917A (en) * | 2017-06-16 | 2017-08-08 | 智者四海（北京）技术有限公司 | The method and system pushed for message |
| CN108595483A (en) * | 2018-03-13 | 2018-09-28 | 腾讯科技（深圳）有限公司 | Data processing method and relevant apparatus |
| CN110086636A (en) * | 2019-04-17 | 2019-08-02 | 平安科技（深圳）有限公司 | A kind of message distributing method based on mqtt, server and storage medium |
| CN110278231A (en) * | 2018-03-16 | 2019-09-24 | 中移(苏州)软件技术有限公司 | A kind of data subscription distribution method and system |
-
2019
- 2019-10-30 CN CN201911044313.6A patent/CN112751891B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106953901A (en) * | 2017-03-10 | 2017-07-14 | 重庆邮电大学 | A kind of trunked communication system and its method for improving message transmission performance |
| CN107026917A (en) * | 2017-06-16 | 2017-08-08 | 智者四海（北京）技术有限公司 | The method and system pushed for message |
| CN108595483A (en) * | 2018-03-13 | 2018-09-28 | 腾讯科技（深圳）有限公司 | Data processing method and relevant apparatus |
| CN110278231A (en) * | 2018-03-16 | 2019-09-24 | 中移(苏州)软件技术有限公司 | A kind of data subscription distribution method and system |
| CN110086636A (en) * | 2019-04-17 | 2019-08-02 | 平安科技（深圳）有限公司 | A kind of message distributing method based on mqtt, server and storage medium |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115242787A (en) * | 2022-07-22 | 2022-10-25 | 恒生电子股份有限公司 | Message processing system and method |
| CN115242787B (en) * | 2022-07-22 | 2023-09-05 | 恒生电子股份有限公司 | Message processing system and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112751891B (en) | 2022-06-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108306877B (en) | NODE JS-based user identity information verification method and device and storage medium | |
| US11212358B2 (en) | Techniques for compact data storage of network traffic and efficient search thereof | |
| CN109643358B (en) | Cross-tenant data leakage isolation | |
| US9729557B1 (en) | Dynamic throttling systems and services | |
| CN107231402B (en) | HTTP request processing method, device and system | |
| US9806974B2 (en) | Efficient acquisition of sensor data in an automated manner | |
| US20200210424A1 (en) | Query engine for remote endpoint information retrieval | |
| CN110020062B (en) | Customizable web crawler method and system | |
| US10282366B2 (en) | Multi-dimensional decomposition computing method and system | |
| US9544288B2 (en) | Messaging gateway | |
| US11036764B1 (en) | Document classification filter for search queries | |
| US20140020064A1 (en) | Scalable Fine-Grained Multi-Service Authorization | |
| CN109726004B (en) | Data processing method and device | |
| CN105897888B (en) | Client connection method and system | |
| JP2017016650A (en) | Method and system for detecting and identifying resource on computer network | |
| CN112926061B (en) | Plug-in processing method and device | |
| US9817910B2 (en) | Restful service description language | |
| CN111193633A (en) | Method and device for detecting abnormal network connection | |
| CN110990057A (en) | Extraction method, device, equipment and medium of small program sub-chain information | |
| US10187264B1 (en) | Gateway path variable detection for metric collection | |
| CN112751891B (en) | Kafka-based message processing method, electronic equipment and storage medium | |
| CN106301906B (en) | Object configuration method and device based on distributed system | |
| US10698931B1 (en) | Input prediction for document text search | |
| US20150206075A1 (en) | Efficient Decision Making | |
| CN114595457A (en) | Task processing method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |