CN110019240B

CN110019240B - Service data interaction method, device and system

Info

Publication number: CN110019240B
Application number: CN201711479774.7A
Authority: CN
Inventors: 谷鑫; 谭朝丽; 程娟; 张鹏; 侯兆森; 李云; 陈大庆
Original assignee: China Mobile Communications Group Co Ltd; China Mobile Group Beijing Co Ltd
Current assignee: China Mobile Communications Group Co Ltd; China Mobile Group Beijing Co Ltd
Priority date: 2017-12-29
Filing date: 2017-12-29
Publication date: 2021-11-09
Anticipated expiration: 2037-12-29
Also published as: CN110019240A

Abstract

A service data interaction method, a device and a system are provided, wherein the system comprises a Kafka cluster, a first service system and a second service system, the first service system obtains a key value pair shared between the first service system and the second service system, determines a target partition according to the key value pair and a partition mechanism of the Kafka cluster, and issues service data to the target partition. The Kafka cluster stores the service data to a target partition, receives the key value pair sent by the second service system, determines the target partition according to the key value pair and a partition mechanism of the Kafka cluster, acquires the service data in the target partition, and sends the service data to the second service system. And the second service system receives the service data sent by the Kafka cluster. The invention can reduce the complexity and the expense of the interactive process of the service system.

Description

Service data interaction method, device and system

Technical Field

The present invention relates to the field of communications technologies, and in particular, to a method, an apparatus, and a system for service data interaction.

Background

Kafka is widely applied to a distributed system for interaction of service data as an open-source message middleware.

The Kafka-based business data interaction system mainly comprises a Kafka cluster consisting of a message Producer (Producer) and a server (Broker) and a message Consumer (Consumer). The message producer publishes the message to the Kafka cluster and each message published to the Kafka cluster has a category, which may be referred to as a topic (topic). Each topic contains one or more partitions (partitions), each of which consists of a series of ordered, immutable messages. The Kafka cluster manages messages in topic, and appends messages issued by the message producer to the partition consecutively, and each message in the partition is assigned a consecutive sequence number, which may be referred to as an offset (offset), which may be used to uniquely identify a message in the partition. The Kafka cluster sends the message to the message consumer, who processes the message.

The Kafka-based service data interaction system generally adopts a publish/subscribe mode to perform service data interaction. In publish/subscribe mode, multiple publishers and subscribers need to be established, where both publishers and subscribers include message producers and message consumers, and both have the ability to publish and retrieve messages to and from the Kafka cluster. And the publisher publishes a message to a certain topic, and the subscriber subscribes to the Kafka cluster and acquires the message under the certain topic. Here, topic may be understood as a flow of messages maintained by Kafka clusters that are differentiated according to different classifications. Each publisher needs to maintain a message stream identifier hash table of a mapping relationship between a message stream identifier and an external message stream, each subscriber also needs to maintain a publisher identifier hash table of a mapping relationship between a message stream identifier and a publisher identifier, and a publisher topic hash table of a mapping relationship between a publisher identifier and a subscriber topic, and the publisher, the publisher identifier and the publisher topic have a one-to-one correspondence relationship and are globally unique. When a publisher publishes a message or a subscriber subscribes to acquire the message, the subscription and publication of the message are realized through a plurality of necessary processing processes of full text retrieval of an external configuration file, hash table initialization of a mapping relation, full library retrieval matching of message identification, content splicing of a message body and the like.

However, in the transmission processing of large-scale heterogeneous data by adopting the above method, the complexity of the contents of the mapping relation hash table that the publisher and the subscriber need to maintain is high, and the processes of reading the external configuration file, initializing the mapping relation table, and searching the mapping relation hash table in the whole library require a complex calculation processing process, and the system consumption is high.

Disclosure of Invention

The embodiment of the invention provides a service data interaction method, a device and a system, which are used for reducing the complexity and the expense of a service system interaction process.

In a first aspect, an embodiment of the present invention provides a service data interaction system, where the system includes a Kafka cluster and at least one service system, where the at least one service system includes a first service system that implements a Kafka producer function and a second service system that implements a Kafka consumer function, and each of the first service system and the second service system has a capability of issuing service data to the Kafka cluster in a Kafka message form and receiving service data from the Kafka cluster in a Kafka message form, where:

the first service system is configured to obtain a key value pair shared between the first service system and the second service system, where the key value pair is used to identify service data interacted between the first service system and the second service system, and the first service system determines a target partition according to the key value pair and a partition mechanism of the Kafka cluster, and issues the service data to the target partition.

The Kafka cluster is configured to receive service data issued by the first service system, store the service data to the target partition, receive a key-value pair sent by the second service system, determine the target partition according to the key-value pair and a partition mechanism of the Kafka cluster, acquire the service data in the target partition, and send the service data to the second service system.

And the second service system is used for acquiring the key value pair, sending the key value pair to the Kafka cluster, subscribing the service data interacted between the first service system and the second service system, and receiving the service data sent by the Kafka cluster.

Optionally, the first service system obtains the key-value pair shared between the first service system and the second service system in the following manner:

encrypting the identity information of the first service system to generate a first identity identifier of the first service system; acquiring the identity information of the second service system, encrypting the identity information of the second service system and generating a second identity; and encrypting the first identity identifier and the second identity identifier to generate a message queue identification code, and using the message queue identification code as a key value pair shared between the first service system and the second service system.

The second service system acquires the key-value pair in the following way:

encrypting the identity information of the second service system to generate a second identity; acquiring the identity information of the first service system, and encrypting the identity information of the first service system to generate a first identity mark; and encrypting according to the second identity identifier and the first identity identifier to generate a message queue identification code, and using the message queue identification code as a key value pair shared between the first service system and the second service system.

Optionally, the key-value pair is an identification code that can be resolved into an integer. The first service system and the Kafka cluster determine the target partition according to the key-value pair and the partition mechanism of the Kafka cluster in the following manner:

analyzing the key value pairs, and carrying out remainder operation on integers obtained by analyzing the key value pairs and the partition number included in the theme divided for the service data by the Kafka cluster; and taking the partition with the partition serial number consistent with the numerical value obtained by the remainder operation as the target partition.

Optionally, the first service system is a different service system or a different sub-service system included in the same service system. The Kafka cluster is further configured to: and dividing different topics for the service data issued by different service systems or the service data issued by different sub-service systems in the same service system.

Optionally, the number of the second service systems is at least one, and the at least one second service system shares the same key-value pair with the first service system.

Optionally, the first service system is further configured to: and encapsulating attribute information of the service data, the key value pairs and the service data into a Kafka message body, and issuing the Kafka message body to the target partition, wherein the attribute information of the service data comprises at least one item of a transmission mode of the service data, a storage format of the service data and a type of the service data. The Kafka cluster is further configured to parse a Kafka message body and store attribute information of the service data.

Further, the first service system is further configured to encrypt the service data, and issue the encrypted service data to the target partition. The second service system is further configured to receive the encrypted service data and decrypt the service data.

Specifically, the Kafka cluster stores the service data to the target partition in the following manner:

dividing the target partition by taking a slice as a unit, storing the service data into the slice of the target partition, generating an offset of the service data in the slice, and identifying the storage position of the service data in the target partition by using the offset.

The Kafka cluster acquires the service data in the target partition by adopting the following mode:

and the Kafka cluster determines a fragment for storing the service data in the target partition by identifying the offset of the service data, and acquires the service data in the fragment.

Further, the Kafka cluster is further configured to: and if the capacity of the fragments for storing the service data exceeds a set threshold value or the storage time of the service data exceeds a set time length, deleting the service data.

In a second aspect, an embodiment of the present invention provides a service data interaction method, which is applicable to a first service system that implements a Kafka producer function, where the method includes:

acquiring interactive service data between a first service system and a second service system, and acquiring a key value pair shared between the first service system and the second service system, wherein the key value pair is used for identifying the interactive service data between the first service system and the second service system;

and determining a target partition according to the key value pair and the partition mechanism of the Kafka cluster, and issuing the service data to the target partition.

Specifically, the obtaining of the key-value pair shared between the first service system and the second service system includes:

encrypting the identity information of the first service system to generate a first identity identifier of the first service system;

acquiring the identity information of the second service system, encrypting the identity information of the second service system and generating a second identity;

and encrypting the first identity identifier and the second identity identifier to generate a message queue identification code, and using the message queue identification code as a key value pair shared between the first service system and the second service system.

Further, the key-value pair is an identification code capable of being resolved into an integer;

the determining the target partition according to the key-value pair and the partition mechanism of the Kafka cluster comprises:

analyzing the key value pairs, and carrying out remainder operation on integers obtained by analyzing the key value pairs and the partition number included in the theme divided for the service data by the Kafka cluster;

and taking the partition with the partition serial number consistent with the numerical value obtained by the remainder operation as the target partition.

Specifically, the issuing the service data to the target partition includes:

and encapsulating attribute information of the service data, the key value pairs and the service data into a Kafka message body, and issuing the Kafka message body to the target partition, wherein the attribute information of the service data comprises at least one item of a transmission mode of the service data, a storage format of the service data and a type of the service data.

In a third aspect, an embodiment of the present invention provides a service data interaction method, which is applicable to a Kafka cluster, and the method includes:

receiving service data issued by a first service system, and storing the service data to a target partition, wherein the target partition is determined by the first service system according to a key value pair and a partition mechanism of the Kafka cluster, and the key value pair is used for identifying the service data interacted between the first service system and the second service system;

receiving a key value pair and a subscription request sent by a second service system, wherein the subscription request is used for subscribing interactive service data between the first service system and the second service system;

and determining the target partition according to the key value pair and the partition mechanism of the Kafka cluster, acquiring the service data in the target partition, and sending the service data to the second service system.

Specifically, the key-value pair is an identification code that can be resolved into an integer;

determining the target partition according to the key-value pair and the partition mechanism of the Kafka cluster, including:

Further, before storing the service data in the target partition, the method further includes:

and dividing different topics for the service data issued by different service systems or the service data issued by different sub-service systems in the same service system.

Specifically, storing the service data to a target partition includes:

storing attribute information of service data and the service data to a target partition;

the attribute information of the service data includes at least one of a transmission mode of the service data, a storage format of the service data, and a type of the service data.

Specifically, storing the service data to a target partition includes:

dividing the target partition by taking a slice as a unit, storing the service data into the slice of the target partition, generating an offset of the service data in the slice, and identifying a storage position of the service data in the target partition by using the offset;

acquiring the service data in the target partition, including:

In a fourth aspect, an embodiment of the present invention provides a service data interaction method, which is applicable to a second service system for implementing a Kafka consumer function, and includes:

acquiring a key value pair shared between a first service system and a second service system, wherein the key value pair is used for identifying service data interacted between the first service system and the second service system;

sending the key value pair and a subscription request to a Kafka cluster, and receiving service data sent by the Kafka cluster;

the subscription request is used for subscribing the interactive service data between the first service system and the second service system;

and the Kafka cluster acquires the service data sent by the Kafka cluster in a target partition, and the target partition is determined by the Kafka cluster according to the key value pair and a partition mechanism of the Kafka cluster.

encrypting the identity information of the second service system to generate a second identity;

acquiring the identity information of the first service system, and encrypting the identity information of the first service system to generate a first identity mark;

and encrypting according to the second identity identifier and the first identity identifier to generate a message queue identification code, and using the message queue identification code as a key value pair shared between the first service system and the second service system.

In a fifth aspect, an embodiment of the present invention provides a service data interaction apparatus, which is applicable to a first service system that implements a Kafka producer function, and includes:

the system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring interactive service data between a first service system and a second service system and key value pairs shared between the first service system and the second service system, and the key value pairs are used for identifying the interactive service data between the first service system and the second service system;

the processing unit is used for determining a target partition according to the key value pair acquired by the acquisition unit and a partition mechanism of the Kafka cluster;

and the issuing unit is used for issuing the service data acquired by the acquiring unit to the target partition determined by the processing unit.

Wherein the key-value pair is an identification code that can be resolved into an integer;

the processing unit determines the target partition according to the key-value pair and the partition mechanism of the Kafka cluster by adopting the following modes:

Optionally, the obtaining unit obtains the key-value pair shared between the first service system and the second service system in the following manner:

Optionally, the publishing unit publishes the service data to the target partition in the following manner:

In a sixth aspect, an embodiment of the present invention provides a service data interaction apparatus, where the apparatus is applicable to a Kafka cluster, and the apparatus includes:

the receiving unit is used for receiving the service data issued by the first service system and receiving the key value pair and the subscription request sent by the second service system;

a storage unit, configured to store the service data received by the receiving unit to a target partition, where the target partition is determined by the first service system according to a key-value pair and a partition mechanism of the Kafka cluster, and the key-value pair is used to identify service data interacted between the first service system and the second service system;

the processing unit is used for determining a target partition according to the key value pair received by the receiving unit and the partition mechanism of the Kafka cluster, and acquiring the service data in the target partition;

and the sending unit is used for sending the service data determined by the processing unit to the second service system.

Wherein the key-value pair is an identification code that can be resolved into an integer. The processing unit determines the target partition according to the key-value pair and the partition mechanism of the Kafka cluster in the following manner:

Optionally, the processing unit is further configured to: before the storage unit stores the service data into a target partition, different topics are divided for the service data issued by different service systems or the service data issued by different sub-service systems included in the same service system.

Further, the storage unit stores the service data to a target partition in the following manner:

storing attribute information of service data and the service data to a target partition; the attribute information of the service data includes at least one of a transmission mode of the service data, a storage format of the service data, and a type of the service data.

The processing unit acquires the service data in the target partition by adopting the following mode: and determining the fragments for storing the service data in the target subarea by using the offset for identifying the service data, and acquiring the service data in the fragments.

In a seventh aspect, an embodiment of the present invention provides a service data interaction device, where the device is applicable to a second service system that implements a Kafka consumer function, and the device includes:

the system comprises a processing unit, a first service system and a second service system, wherein the processing unit is used for acquiring a key value pair shared between the first service system and the second service system, and the key value pair is used for identifying service data interacted between the first service system and the second service system;

a sending unit, configured to send the key-value pair and a subscription request to a Kafka cluster, where the subscription request is used to subscribe to service data interacted between the first service system and the second service system;

a receiving unit, configured to receive service data sent by a Kafka cluster, where the service data sent by the Kafka cluster is obtained by the Kafka cluster in a target partition, and the target partition is determined by the Kafka cluster according to the key value pair and a partition mechanism of the Kafka cluster.

Optionally, the processing unit obtains the key-value pair shared between the first service system and the second service system by using the following method:

According to the business data interaction method, the business data interaction device and the business data interaction system, key value pairs are shared among business systems of interactive data, the key value pairs and a partitioning mechanism of a Kafka cluster are used for determining the target partitions, the business data are issued and stored in the determined target partitions, a mapping relation hash table does not need to be maintained, and processes of initializing the mapping relation table and retrieving the mapping relation hash table in a whole library are not needed, so that the complexity and the expense of the business system interaction process can be reduced to a certain extent.

Drawings

FIG. 1 is a diagram of a product operating system architecture to which an embodiment of the present invention is applied;

FIG. 2 is a schematic diagram of a Kafka message system applied in an embodiment of the present invention;

fig. 3 is a schematic diagram illustrating a service data interaction system according to an embodiment of the present invention;

fig. 4 is a schematic diagram of issuing service data to a target partition according to an embodiment of the present invention;

fig. 5 is a schematic diagram of a Partition storage message according to an embodiment of the present invention;

fig. 6 is a schematic diagram of a process of sending a message by a Producer according to an embodiment of the present invention;

fig. 7 is a schematic diagram of a message push networking of a product operation system applying Kafka according to an embodiment of the present invention;

fig. 8 is a schematic diagram of storing service data into a slice of the target partition according to an embodiment of the present invention;

FIG. 9 is a diagram illustrating segment file naming according to an embodiment of the present invention;

fig. 10 is a schematic diagram of a corresponding relationship between a ". index" file and a ". log" file according to an embodiment of the present invention;

fig. 11 is a schematic diagram of a FileMessageSet-like storage file according to an embodiment of the present invention;

fig. 12 is a flowchart of an implementation of a service data interaction method applied to a first service system according to an embodiment of the present invention;

fig. 13 is a schematic flowchart of a first service system obtaining key-value pairs according to an embodiment of the present invention;

fig. 14 is a schematic flowchart of determining a target partition by the first service system according to the embodiment of the present invention;

fig. 15 is a flowchart of an implementation of a service data interaction method applied to a Kafka cluster according to an embodiment of the present invention;

fig. 16 is a flowchart of an implementation of a service data interaction method applied to a second service system according to an embodiment of the present invention;

fig. 17 is a schematic structural diagram of a service data interaction apparatus applied to a first service system according to an embodiment of the present invention;

fig. 18 is a schematic structural diagram of a service data interaction apparatus applied to a Kafka cluster according to an embodiment of the present invention;

fig. 19 is a schematic structural diagram of a service data interaction apparatus applied to a second service system according to an embodiment of the present invention;

fig. 20 is a schematic structural diagram of a service data interaction device according to an embodiment of the present invention.

Detailed Description

The preferred embodiments of the present invention will be described below with reference to the accompanying drawings of the specification, it being understood that the preferred embodiments described herein are merely for illustrating and explaining the present invention, and are not intended to limit the present invention, and that the embodiments and features of the embodiments in the present invention may be combined with each other without conflict.

The embodiment of the invention is mainly applied to a scene of service data interaction among a plurality of service systems, service data can be broadcasted among the service systems in the application scene, and service data can also be unicast, for example, the embodiment of the invention can be applied to a product operation system shown in fig. 1, wherein the product operation system comprises a product operation platform and each service platform cooperating with the product operation platform. In order to realize capability introduction of partners and bearing of business models in an industrial chain, a product operation platform needs to provide various management modes and means including product management, package management, tariff management, share management and charging management for each service platform, and ensure that each service platform can place the optimal product on the product operation platform for promotion and selling. The data formats and transmission modes for transmitting the service data between the product operation platform and each service platform are also different. For example, because the information transmission such as finance, government affairs and finance is involved between the product operation platform and the systems such as the medium-mobile online electronic invoice, the message information mobile government affairs and the small-source science and technology financial intelligent short message, and the like, due to the double consideration of real-time performance and safety, a synchronous data transmission mode is widely adopted, and the acquisition of necessary display information and the circulation of business processes are completed by using data formats such as META, file, XML, JSON, picture and the like. Because the customized processing time needs to be reserved after the products are ordered between the product operation platform and systems such as a scientific and technological flow bank, a basalt scientific and technological xuanxuan, an subsignal scientific and technological neighbor-happy exchange and the like, in consideration of product reliability, an asynchronous data transmission mode is widely adopted, and the acquisition of necessary display information and the circulation of business processes are completed by using data formats such as OFFICE, files, JSON, pictures and the like.

Because the number of the service platforms is huge, the service data volume of interaction between the product operation platform and the service platform is also large, and the data formats used by the service platforms are different, in the embodiment of the invention, the Kafka message system shown in fig. 2 can be embedded in the product operation system shown in fig. 1, and the Kafka message system is used as a data pipeline and a message system, so that data transmission capability support is provided for message driving among the service modules, the sending and receiving of the service data with different data formats are realized, and the synchronous or asynchronous transmission requirements between the product operation platform and the service platforms with different types are met.

In fig. 2, a Broker may be understood as one or more individual servers included in a Kafka cluster, and these servers are generally referred to as a Broker. Producer, which may be understood to be a message Producer, is responsible for issuing messages to Kafka Broker. Consumer, which may be understood as a message Consumer, is responsible for receiving messages from Kafka Broker. One or more consumers may be divided into Consumer groups (Consumer groups), which may generally be divided according to actual traffic needs. Each message issued by Producer to Kafka cluster has a category, which is Topic. The messages of physically different topics are stored separately, and logically, the message of one Topic is stored on one or more brokers, but the user only needs to specify the Topic of the message to produce or consume the data without having to care about where the data is stored. Kafka performs message management by using Topic, and divides each Topic into a plurality of partitions by adopting a Partition mechanism, wherein each Partition corresponds to a logic Log and consists of a plurality of segments. Each Segment stores therein a plurality of messages. Message is the basic unit of communication in the Kafka Message system, and each Producer can issue any number of messages to one Topic. Each Partition has an Index (Index) in memory to record the first message offset in each Segment. The message that the Producer publishes to a certain Topic is evenly distributed to a plurality of partitions, and the Broker adds the message to the last Segment of the corresponding partitions after receiving the published message. When the number of messages on a Segment reaches a configured value or the message publishing time exceeds a threshold value, the messages are automatically written to the disk. After the Segment size reaches a certain size, data cannot be written into the Segment any more, and the breaker can create a new Segment at a proper time. Each Partition consists of a series of ordered, immutable messages that are appended to the Partition in succession. Each message in the Partition has a continuous assigned sequence number called Offset for uniquely identifying a message, and the Kafka cluster can be located to the storage position of the message through the message identification, thereby effectively avoiding additional mapping from the identification to the position. Load balancing between Broker and Consumer can be achieved using Zookeeper. All Broker and Consumer will register in the Zookeeper, which will automatically save some metadata information. If one Broker and Consumer change, the other Broker and Consumer are notified.

The product operation system embedded with the Kafka message system can be understood as a Kafka-based service data interaction system, in which a Kafka cluster and a plurality of service systems (e.g., service platforms in fig. 1) are included. The service system related in the embodiment of the present invention may have a Producer function and may also have a Consumer function, and has the capability of issuing service data in the form of Kafka messages to and receiving service data in the form of Kafka messages from the Kafka cluster. In the embodiment of the present invention, for convenience of description, a service system with a Producer function is referred to as a first service system, and a service system with a Consumer function is referred to as a second service system, where the first service system and the second service system may be the same service system or different service systems.

Fig. 3 is a schematic diagram of a service data interaction system based on Kafka according to an embodiment of the present invention. In fig. 3, a first service system has a Producer function, and is configured to obtain a key value pair shared between the first service system and the second service system, where the key value pair is used to identify service data exchanged between the first service system and the second service system, and the first service system determines a target Partition according to the key value pair and a Partition mechanism of the Kafka cluster, and issues the service data to the target Partition. The Kafka cluster is configured to receive the service data issued by the first service system, and store the service data to the target Partition. The second service system has a Consumer function and is used for acquiring the key value pair, sending the key value pair to the Kafka cluster and subscribing the service data interacted between the first service system and the second service system. The Kafka cluster is used for receiving the key value pair sent by the second service system, determining the target Partition according to the key value pair and a Partition mechanism of the Kafka cluster, acquiring the service data in the target Partition, and sending the service data to the second service system. And the second service system receives the service data sent by the Kafka cluster.

In the embodiment of the invention, the key value pair is shared between the first service system and the second service system, the target Partition is determined by the key value pair and a Partition mechanism of the Kafka cluster, the service data is published and stored in the determined target Partition, a mapping relation hash table is not required to be maintained, and the processes of initializing the mapping relation hash table and searching the mapping relation hash table in a whole library are not required to be executed, so that the complexity and the expense of the interaction process of the service systems can be reduced to a certain extent.

For convenience of description in the embodiments of the present invention, sometimes the first service system and the Producer are often used alternately, and the second service system and the Consumer are often used alternately, and those skilled in the art should understand that the meanings thereof are the same concept.

Specifically, the Consumer received data is performed according to a Group (Customer Group), and therefore, in the embodiment of the present invention, the key value pair referred to above may be understood as a key value shared between the Producer and each Consumer in a Consumer Group, where the number of consumers included in the Consumer Group may be one or multiple. The above-mentioned shared key-value pair may be pre-negotiated between the Producer and the Consumer, or may be generated by using a predetermined generation rule. For example, the first service system may obtain the key-value pair shared between the first service system and the second service system as follows: and encrypting the identity information of the first service system to generate a first identity of the first service system, wherein the first identity can be understood as the unique identity of the first service system in the Kafka-based service data interaction system. And acquiring the identity information of the second service system, encrypting the identity information of the second service system, and generating a second identity, wherein the second identity can be understood as an information pushing identifier unique to the first service system in the Kafka-based service data interaction system, that is, the second identity can determine that the first service system pushes the message to the second service system. And encrypting the first identity identifier and the second identity identifier to generate a message queue identification code, and using the message queue identification code as a key value pair shared between the first service system and the second service system. Similarly, the second service system may obtain the key-value pair in the following manner: and encrypting the identity information of the second service system to generate a second identity, wherein the second identity can be understood as the unique identity of the second service system in the Kafka-based service data interaction system. And acquiring the identity information of the first service system, encrypting the identity information of the first service system, and generating a first identity, wherein the second identity can be understood as a unique information push identity of the first service system in a Kafka-based service data interaction system. And encrypting according to the second identity and the first identity to generate a message queue identification code (Key), and using the message queue identification code as a Key value pair shared between the first service system and the second service system.

In a possible implementation, the first service system and the second service system generate the above-mentioned related message queue identification code (Key) and synchronize to each database included in the Kafka cluster, so that the Kafka cluster can push the message based on the message queue identification code (Key).

In the embodiment of the present invention, the first service system and the second service system may generate the first identity and the second identity using an encryption algorithm such as MD 5. The specific encryption implementation process can be implemented by adopting the prior art, and the embodiment of the invention is not detailed.

In a possible implementation manner, in the embodiment of the present invention, the key-value pair may be set as an identification code that can be resolved into an integer. When determining the target Partition of the business data storage according to the key value pair capable of being analyzed into the integer, the Producer can analyze the key value pair, and carry out a remainder operation on the integer obtained by analyzing the key value pair and the Partition number included in the theme divided for the business data by the Kafka cluster; and taking the Partition with the Partition serial number consistent with the numerical value obtained by the operation of remainder as the target Partition.

Wherein the number of partitions included in the theme of the Kafka cluster for the service data partitioning may be specified by a parameter of part.class of the Producer, which must implement the kafka.producer.partitioner interface. In the embodiment of the invention, Key can be analyzed into an integer, so that the integer obtained by analyzing Key and the total number of partitions included in the theme of business data division can be cancelled, and the message can be sent to the partitions with the serial number corresponding to the numerical value obtained after the cancellation. If the class in the code shown in fig. 4 is regarded as partition.class, and 20 messages (keys are 0, 1, 2, 3, respectively) to Topic3 (including 4 partitions) are sent through the code shown in fig. 4, each Partition has a sequence number, and the sequence numbers start from 0, then the messages with the same Key will be sent and stored in the same Partition, and the sequence number of the Key is exactly the same as the Partition sequence number. The Partition number starts from 0, and the Key in this example also starts from 0.

In the embodiment of the invention, through the first service system and the second service system, the implementation mode of presetting the unique agreement Key according to the service rule and directly executing data transmission can not only save the calculation processes with high complexity and large system consumption, such as reading an external configuration file, initializing a mapping relation table, retrieving a mapping Hash table from a whole database, and the like, but also can linearly improve the system performance by utilizing the inherent distributed characteristic of Kafka.

In the embodiment of the present invention, the first service system may be different service systems or different sub-service systems included in the same service system, and in this case, the messages issued by the first service system may be of multiple types, so that in the embodiment of the present invention, the Kafka cluster may divide different topics for the service data issued by different service systems or the service data issued by different sub-service systems included in the same service system, and the different topics are independent from each other. Since Kafka has an inherent distributed characteristic, in order to achieve high scalability of the system, in the embodiment of the present invention, each Topic with a broad meaning (e.g., client information synchronization, product information presentation, payment order confirmation, etc.) may be distributed to multiple Kafka servers (i.e., brokers) in the Kafka cluster, while multiple different partitions are established for each Topic, and the correspondingly created partitions are specified when Topic is created. Each Partition is an asynchronous ordered queue and is responsible for storing and transmitting an assigned asynchronous Message body (Message), and the same Partition can ensure that push messages in the Partition are sent to the provider in a sequential self-sustaining manner, but the partitions are independent from each other, as shown in fig. 5. Wherein the Partition number starts from 0.

In the embodiment of the invention, the Kafka cluster pairs Topic are divided into one or more partitions, the messages in the partitions are messages with key value pairs, the messages with the same key value pair enter the same Partition, and the messages of the same Partition are sent to the same Consumer. Thus, messages, topics, partitions and consumers having the same key value have a certain mapping relationship, as shown in fig. 6, Producera sends message 0 and message 1 to Consumera through Partition1, and sends message 0, message 1 and message 2 to Consumerb through Partition 2. Producerb sends message 0 and message 1 to Consumerc via Partition 3.

In particular, the Consumer receives data in packets (Customer Group), and the Kafka cluster ensures that each Partition transmits data that is consumed by the same Consumer in the same Customer Group. If repeated consumption is desired, then other Customer groups are needed for consumption. The number of Partitions in each Topic determines the degree to which the Topic can be balanced in the same Customer Group, and more Partitions mean that more consumers can be accommodated, which effectively improves the capability of concurrent consumption. For example, when the service data interaction system based on Kafka provided by the embodiment of the present invention is applied to a production interaction system such as that shown in fig. 2, a product operation platform needs to obtain product classification information from service platforms such as a red cloud video communication platform, a traffic bank platform, and a wechat pervasive application OA platform, and then places the product classification information in the same Customer Group, and similarly, an electronic invoice and financial intelligent communication can be placed in the same Customer Group, and then the same Partition under the same Topic is used to send the electronic invoice and financial intelligent communication to a Customer in the Customer Group, so that broadcasting of one Topic message to a plurality of customers is achieved. If the unique code of the wearable device needs to be acquired from the intelligent police wearable device, the intelligent police wearable device can be independently placed in a Customer Group, so that unicast of a Topic message to a Customer is realized, and a specific message pushing networking schematic diagram can be shown in fig. 7.

Specifically, in the embodiment of the present invention, when the first service system (Producer) issues the service data to the Kafka cluster, the service data may be encapsulated as a Kafka message body and issued to the target Partition determined in the Kafka cluster.

Further, in this embodiment of the present invention, the first service system (Producer) may encapsulate attribute information of service data, the key value pair, and the service data into a Kafka message body, and issue the Kafka message body to the target Partition, where the attribute information of the service data includes at least one of a transmission mode of the service data, a storage format of the service data, and a type of the service data. The Kafka cluster is further configured to parse a Kafka message body, and store attribute information of the service data, so that a subsequent concumer can subscribe related information according to the attribute information of the service data recorded in the Kafka cluster, and the Kafka cluster can flexibly combine a message queue subscribed by the concumer according to the attribute information of the recorded service data, so that the service system can implement interaction of different types of service data between different service systems and synchronous or asynchronous transmission.

The transmission mode of the service data in the embodiment of the invention can be specified by a Producer through a configuration parameter Producer. Specifically, the configuration parameter producer.type may indicate whether the sending manner of the message in the background thread is synchronous or asynchronous, and the default is synchronous, that is, producer.type ═ Sync. If the service data needs to be sent asynchronously, it only needs to be set to an asynchronous mode, that is, product. In the embodiment of the invention, the Producer can also issue the business data in a Batch form by setting the Batch attribute, so that the data processing and transmission performance of the Broker is greatly improved, but the risk of data loss is increased. If the requirements on data transmission accuracy, real-time data response, reliable return result and the like, such as flow bank, electronic invoice and online finance, are high, the parameter must be set to be a synchronous mode Sync.

In the embodiment of the invention, the data is asynchronously pushed in a Batch (Batch) mode, so that the processing efficiency can be improved, and the Producer can accumulate the messages in the memory to a certain amount and then send the asynchronous requests as the Batch. The size of Batch can be controlled by Producer's parameters (Batch. By increasing the size of Batch, the number of network requests and disk IO times can be reduced, and certainly, a tradeoff needs to be made between efficiency and timeliness in setting specific parameters. The number of the released service data can be configured by setting the parameter of batch.

Furthermore, in the embodiment of the present invention, the Producer may also configure the transmission mode of the service data by setting transmission configuration parameters such as a Producer buffering message time, a maximum buffering message number in a Producer buffering queue, a Producer blocking waiting time, and a Producer buffered message number.

In the embodiment of the invention, by the way of dividing the subjects of different service systems or different service subsystems and storing the attribute information of the service data, the horizontal expansion of the message queue can be realized, and the distributed message can be realized.

Further, in the embodiment of the present invention, the first service system is further configured to encrypt the service data, and issue the encrypted service data to the target Partition. The second service system is further configured to receive the encrypted service data and decrypt the service data.

Specifically, according to the product access requirements of different service systems, the method and the device in the embodiment of the invention consider the possibility that the key information of the service process is attacked and stolen in the processes of generation, sending, forwarding and receiving, and can perform encryption and decryption processing on part of key service data so as to ensure the safety and reliability in the data transmission process.

Specifically, in the embodiment of the present invention, the Producer issues the service data to the Kafka Broker through Topic, the Kafka cluster is classified by messages through Topic, Topic can be grouped by partitions in a physical layer, and one Topic can be divided into several partitions. If the Partition is used as the minimum storage unit, when any plurality of production systems continuously send messages to the product operation platform, the Partition file is inevitably expanded unlimitedly, so that the maintenance of the message file and the cleaning of the consumed messages are seriously affected, and therefore the Kafka cluster in the embodiment of the invention stores the service data to the target Partition in the following mode: the target Partition is divided by a slice (Segment) unit, and the service data is stored in the slice of the target Partition, for example, as shown in fig. 8. Each Partition is equivalent to a jumbo file and is evenly distributed into a plurality of Segment data files with the same size, the number of messages in each Segment file is not necessarily equal, and the feature also facilitates the deletion of segments with longer storage time, namely, facilitates the cleaning of consumed messages and improves the utilization rate of a disk.

Specifically, each Partition needs to support sequential reading and writing, and the file life cycle of Segment is determined by a plurality of parameters (log.segment.bytes, log.roll. { Ms, homes } and the like) configured by the server. Segment files are composed of two parts, namely ". Index" files and ". Log" files, which are denoted Segment Index files and data files, respectively. The naming rules for these two files are: the first Segment of the Partition global is started from 0, the name of each subsequent Segment file is the Offset value of the last message of the previous Segment file, the value size is 64 bits, the character length of 20 bits is digit, and no digit is filled with 0, as shown in fig. 9. Wherein, the. Index "file and the". Log "file have corresponding management, and the Segment file shown in FIG. 9 is taken as an example for explanation, and FIG. 10 shows Segment: 00000000000000170410 ". Index" file and ". Log" file.

In the embodiment of the invention, the Index file stores a large amount of metadata, the Log data file stores a large amount of messages, and the metadata in the Index file points to the physical offset address of the Message in the corresponding data file. Taking the metadata [3,348] in the ". Index" Index file as an example, the ". Log" data file represents the 3 rd message, i.e., 170410+3 ═ 170413 messages in the Partition, and the message has a physical offset address of 348.

In the embodiment of the present invention, the Kafka cluster may generate an Offset (Offset) of the service data in the Segment, and identify a storage location of the service data in the target Partition by using the Offset. Specifically, after receiving the service data issued by the Producer, the Kafka cluster may place the service data in a Partition. Each message (traffic data) in the Partition is represented by Offset indicating its storage location in the Partition, which is, of course, not the actual storage location of the message in the Partition data file, but a logical value. In the embodiment of the invention, the service Data can be packaged into a Message class and has attribute information such as Offset, Message size, Data and the like. Wherein Offset is Long type, Messagesize is Int32, which represents Data size, Data is the concrete content of Message, and the format is consistent with the Messageset in Kafka communication protocol. In the embodiment of the invention, the whole data file comprises a plurality of message examples which accord with the format and are arranged together from small to large according to Offset. We package it as a Filemessageset class, whose class diagram is shown in fig. 11.

After the Kafka cluster identifies the service data through Offset in the embodiment of the present invention, the service data can be obtained in the target Partition in the following manner: and determining Segment for storing the service data in the target Partition by utilizing the Offset for identifying the service data, and acquiring the service data in the Segment, wherein compared with an interface programming mode, the mode simply adds the character between the publisher identifier and the message stream identifier and combines the character according to a mode of connecting character strings to form an identifier key for uniquely receiving the Kafka message, thereby being simpler to realize. For example, when reading the message with Offset of 170418 in Segment, first find Segment file, where 00000000000000000000.Index is the first file, the second file is 00000000000000170410.Index (start Offset 170410+1 — 170411), and the third file is 00000000000000239430.Index (start Offset 239430+1 — 239431), so that the Offset 170418 falls into the second file. Other subsequent files can be analogized in turn, named by their actual offsets and arranged, and then can be quickly located to a specific file location according to a binary search method. Secondly, according to the index file: the [8, 1325] in 00000000000000170410.Index is located to the 1325 location in 0000000000170410.Log file for reading.

Furthermore, the Kafka cluster is further configured to delete the service data in the Segment based on the set deletion policy, so as to release the storage space. Specifically, the service data may be deleted when it is determined that the capacity of the fragment storing the service data exceeds a set threshold, or the service data may be deleted when the storage time of the service data exceeds a set duration.

In the embodiment of the invention, the Kafka cluster can also delete the set service data regularly, the relation between the execution of the regular deletion of the consumed files and the performance improvement of the Kafka is not large, namely how to select the deletion strategy is only related to the specific requirement of the disk monitoring size. In addition, the system will keep some metadata information for each Consumer Group as the currently consumed record information (message storage location, also known as Offset). The Offset is controlled by the Consumer, which normally increases the Offset linearly after consuming a message. Because Offset is controlled by the Consumer, Kafka Broker is stateless, it does not need to mark which messages are consumed by which Consumer, it does not need to go through the Broker to ensure that only one Consumer can consume a message in the same Consumer Group, and therefore there is no need to implement a lock mechanism, which provides a strong guarantee for Kafka's high data throughput rate.

Based on the service data interaction system provided above, an embodiment of the present invention further provides a service data interaction method, which may be applied to a first service system that implements a Producer function, and may also be understood as being applied to a Producer. Fig. 12 is a flowchart illustrating an implementation of a service data interaction method applied to a first service system according to an embodiment of the present invention, and referring to fig. 12, the method includes:

s101: and acquiring interactive service data between the first service system and the second service system.

S102: and acquiring the key value pair shared between the first service system and the second service system. Specifically, the number of the second service systems is at least one, and the at least one second service system shares the same key value pair with the first service system, that is, the key value pair is shared by the first service system and a group of second service systems in the embodiment of the present invention, and the number of the second service systems in the group may be one or multiple.

In the embodiment of the present invention, the key value pair shared between the first service system and the second service system may be used to identify service data interacted between the first service system and the second service system.

S103: and determining a target partition according to the obtained key value pair and a partition mechanism of the Kafka cluster, and issuing the obtained service data to the determined target partition.

In a possible implementation manner in this embodiment of the present invention, the first service system may encapsulate, as a Kafka message body, service data attribute information including at least one of a transmission mode of service data, a storage format of service data, and a type of service data, and the key value pair and the service data, and issue the Kafka message body to the target partition, so that the Kafka cluster stores the attribute information of the service data, so as to perform grouping and Topic partitioning according to the attribute information of the service data in the subsequent process.

Specifically, the obtaining, by the first service system, the key value pair shared between the first service system and the second service system may be implemented by using a method flow shown in fig. 13:

s1021: and encrypting the identity information of the first service system to generate a first identity of the first service system.

S1022: and acquiring the identity information of the second service system, and encrypting the identity information of the second service system to generate a second identity.

S1033: and encrypting the first identity identifier and the second identity identifier to generate a message queue identification code, and using the message queue identification code as a key value pair shared between the first service system and the second service system.

Further, the key-value pair referred to above may be an identification code that can be resolved into an integer, and the specific implementation process of the first service system determining the target partition according to the key-value pair and the partition mechanism of the Kafka cluster may adopt a method flow such as that shown in fig. 14, which is shown in fig. 14 and includes:

s1031: and analyzing the acquired key value pair.

S1032: and carrying out remainder operation on the integer obtained by analyzing the key value pair and the partition number included in the theme divided for the service data by the Kafka cluster.

Specifically, the first service system may be a different service system or a different sub-service system included in the same service system. The Kafka cluster can divide different topics for the service data issued by different service systems or the service data issued by different sub-service systems included in the same service system. In the embodiment of the invention, the Topic divided by the Kafka cluster for the service data interacted between the first service system and the second service system can be determined, and the number of the partitions included in the Topic is determined.

S1033: and taking the partition with the partition serial number consistent with the numerical value obtained by the remainder operation as the determined target partition.

According to the business data interaction method provided by the embodiment of the invention, the first business system determines the target partition by using the key value pair and the partition mechanism of the Kafka cluster, the business data is released and stored in the determined target partition, the mapping relation hash table is not required to be maintained, and the processes of initializing the mapping relation hash table and searching the mapping relation hash table in a whole library are not required to be executed, so that the complexity and the expense of the business system interaction process can be reduced to a certain extent.

The embodiment of the invention also provides a service data interaction method, which can be applied to the Kafka cluster and can also be understood as being applied to the Broker. Fig. 15 shows a service data interaction method applied to a Kafka cluster according to an embodiment of the present invention, and referring to fig. 15, the method includes:

s201: and receiving service data issued by a first service system, and storing the service data to a target partition.

In the embodiment of the present invention, the target partition is determined by the first service system according to a key value pair and a partition mechanism of the Kafka cluster, where the key value pair is used to identify service data interacted between the first service system and the second service system.

Optionally, the key-value pair is an identification code that can be resolved into an integer.

Optionally, different topics are divided for service data issued by different service systems or service data issued by different sub-service systems included in the same service system.

Specifically, the Kafka cluster may store the attribute information of the service data and the service data to the target partition. Wherein the attribute information of the service data includes at least one of a transmission mode of the service data, a storage format of the service data, and a type of the service data.

Further, the Kafka cluster may also divide the target partition by taking a slice as a unit, store the service data into the slice of the target partition, generate an offset of the service data in the slice, and identify a storage location of the service data in the target partition by using the offset.

S202: and receiving the key value pair and the subscription request sent by the second service system.

In the embodiment of the present invention, the subscription request is used to subscribe the service data interacted between the first service system and the second service system.

S203: and determining the target partition according to the key value pair and the partition mechanism of the Kafka cluster, and acquiring the service data in the target partition.

In this embodiment of the present invention, determining, by the Kafka cluster, the target partition according to the key-value pair and the partition mechanism of the Kafka cluster may be determined in a manner similar to determining, by the first service system, the target partition, for example: and analyzing the key value pairs, carrying out remainder operation on integers obtained by analyzing the key value pairs and the partition number included in the theme divided for the service data by the Kafka cluster, and taking the partition with the partition serial number consistent with the numerical value obtained by the remainder operation as the target partition.

The Kafka cluster can determine a fragment for storing the service data in the target partition by using the offset for identifying the service data, and acquire the service data in the fragment.

S204: and sending the acquired service data to a second service system.

In the service data interaction method provided by the embodiment of the invention, the Kafka cluster determines the target partition by using the key value pair shared by the first service system and the second service system and the partition mechanism of the Kafka cluster, and issues and stores the service data in the determined target partition without maintaining the mapping relation hash table and executing the processes of initializing the mapping relation table and searching the mapping relation hash table in a whole library, so that the complexity and the expense of the service system interaction process can be reduced to a certain extent.

The embodiment of the invention also provides a service data interaction method, which can be applied to a second service system for realizing the Consumer function, and can also be understood as being applied to the Consumer. Fig. 16 is a flowchart illustrating an implementation of a method applied to a second service system for implementing a provider function according to an embodiment of the present invention, referring to fig. 16, where the method includes:

s301: and acquiring a key value pair shared between the first service system and the second service system.

In the embodiment of the present invention, the key value pair is used to identify service data interacted between the first service system and the second service system.

Specifically, the second service system may acquire the key-value pair in a manner similar to that of the first system, for example: encrypting the identity information of the second service system to generate a second identity; acquiring the identity information of the first service system, and encrypting the identity information of the first service system to generate a first identity mark; and encrypting according to the second identity identifier and the first identity identifier to generate a message queue identification code, and using the message queue identification code as a key value pair shared between the first service system and the second service system.

S302: and sending the acquired key value pair to the Kafka cluster, and sending a subscription request.

The subscription request is used for subscribing the service data interacted between the first service system and the second service system.

S303: and receiving the service data sent by the Kafka cluster.

Specifically, the service data sent by the Kafka cluster is acquired by the Kafka cluster in a target partition, and the target partition is determined by the Kafka cluster according to the key value pair and a partition mechanism of the Kafka cluster.

In the service data interaction method provided by the embodiment of the invention, the second service system sends the key value pair to the Kafka cluster, so that the Kafka cluster determines the target partition by using the key value pair and the partition mechanism of the Kafka cluster, and sends the acquired service data to the second service system, so that the second service system does not need to maintain the mapping relation hash table and does not need to execute the processes of mapping relation table initialization and full-base mapping relation hash table retrieval, and the complexity and the expense of the service system interaction process can be reduced to a certain extent.

Based on the same inventive concept, the embodiment of the present invention provides a service data interaction device, which can be applied to a first service system that implements a Producer function. Fig. 17 is a schematic structural diagram of a service data interaction apparatus applied to a first service system for implementing a Producer function according to an embodiment of the present invention, and referring to fig. 17, the apparatus includes an obtaining unit 101, a processing unit 102, and a publishing unit 103. The acquiring unit 101 is configured to acquire service data interacted between a first service system and a second service system, and a key value pair shared between the first service system and the second service system, where the key value pair is used to identify the service data interacted between the first service system and the second service system. And the processing unit 102 is configured to determine a target partition according to the key-value pair acquired by the acquiring unit 101 and a partition mechanism of the Kafka cluster. The issuing unit 103 is configured to issue the service data acquired by the acquiring unit 101 to the target partition determined by the processing unit 102.

In one possible embodiment, the key-value pairs are identification codes that can be resolved into integers. The processing unit 102 determines the target partition according to the key-value pair and the partition mechanism of the Kafka cluster in the following manner:

Optionally, the obtaining unit 101 obtains the key value pair shared between the first service system and the second service system in the following manner:

Optionally, the issuing unit 103 issues the service data to the target partition in the following manner:

Based on the same inventive concept, the embodiment of the invention provides a service data interaction device, which can be applied to a Kafka cluster. Fig. 18 is a schematic structural diagram of a service data interaction apparatus applied to a Kafka cluster according to an embodiment of the present invention, and referring to fig. 18, the apparatus includes a receiving unit 201, a storage unit 202, a processing unit 203, and a sending unit 204. The receiving unit 201 is configured to receive service data issued by a first service system, and receive a key value pair and a subscription request sent by a second service system. A storing unit 202, configured to store the service data received by the receiving unit 201 in a target partition, where the target partition is determined by the first service system according to a key value pair and a partition mechanism of the Kafka cluster, and the key value pair is used to identify service data interacted between the first service system and the second service system. The processing unit 203 is configured to determine a target partition according to the key-value pair received by the receiving unit 201 and the partition mechanism of the Kafka cluster, and acquire the service data in the target partition. A sending unit 204, configured to send the service data determined by the processing unit 203 to the second service system.

Wherein the key-value pair is an identification code that can be resolved into an integer. The processing unit 203 determines the target partition according to the key-value pair and the partition mechanism of the Kafka cluster in the following manner:

Optionally, the processing unit 203 is further configured to: before the storage unit 202 stores the service data in the target partition, different topics are divided for the service data issued by different service systems or the service data issued by different sub-service systems included in the same service system.

Further, the storage unit 202 stores the service data to the target partition in the following manner:

The processing unit 203 acquires the service data in the target partition by adopting the following method: and determining the fragments for storing the service data in the target subarea by using the offset for identifying the service data, and acquiring the service data in the fragments.

Based on the same inventive concept, the embodiment of the invention further provides a service data interaction device, and the service data interaction device can be applied to a second service system for realizing the Consumer function. Fig. 19 is a schematic structural diagram of a service data interaction apparatus applied to a second service system for implementing a provider function according to an embodiment of the present invention, referring to fig. 19, the apparatus includes a processing unit 301, a transmitting unit 302, and a receiving unit 304. The processing unit 301 is configured to obtain a key value pair shared between a first service system and a second service system, where the key value pair is used to identify service data interacted between the first service system and the second service system. A sending unit 302, configured to send the key-value pair determined by the processing unit 301 to the Kafka cluster, and send a subscription request. The subscription request is used for subscribing the interactive service data between the first service system and the second service system. A receiving unit 303, configured to receive service data sent by a Kafka cluster, where the service data sent by the Kafka cluster is obtained by the Kafka cluster in a target partition, and the target partition is determined by the Kafka cluster according to the key value pair and a partition mechanism of the Kafka cluster.

Optionally, the processing unit 301 obtains the key value pair shared between the first service system and the second service system by the following method:

The embodiment of the invention also provides business data interaction equipment, which can be applied to a Producer, a breaker and a Consumer. Fig. 20 is a schematic structural diagram of a service data interaction device according to an embodiment of the present invention, and referring to fig. 20, the service data interaction device according to the embodiment of the present invention includes a memory 1001, a processor 1002, and a computer program 1003 stored in the memory and capable of running on the processor, and when the processor 1002 executes the computer program, the method for implementing the above-mentioned first service system, second service system, or Kafka cluster service data interaction is implemented.

The Processor may be a Central Processing Unit (CPU), a general purpose Processor, a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a transistor logic device, a hardware component, or any combination thereof. Which may implement or perform the various illustrative logical blocks, modules, and circuits described in connection with the disclosure. A processor may also be a combination of computing functions, e.g., comprising one or more microprocessors, a DSP and a microprocessor, or the like.

Wherein the memory may be integrated in the processor or may be provided separately from the processor.

It can be understood that the drawings of the embodiments of the present application only show simplified designs of the service data interaction device and the service data interaction apparatus. In practical applications, the structure is not limited to the above.

The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the steps in the method for implementing service data interaction by using the first service system, the second service system, or the Kafka cluster as described in the above embodiment.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, embodiments of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

Embodiments of the present invention are described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It will be apparent to those skilled in the art that various changes and modifications may be made in the embodiments of the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to encompass such modifications and variations.

Claims

1. A service data interaction method applied to a first service system implementing Kafka producer function, the method comprising:

acquiring interactive service data between a first service system and a second service system and a key value pair shared between the first service system and the second service system, wherein the key value pair is used for identifying the interactive service data between the first service system and the second service system; wherein the obtaining of the key-value pair shared between the first service system and the second service system includes: encrypting the identity information of the first service system to generate a first identity identifier of the first service system; acquiring the identity information of the second service system, encrypting the identity information of the second service system and generating a second identity; encrypting the first identity identifier and the second identity identifier to generate a message queue identification code, and using the message queue identification code as a key value pair shared between the first service system and the second service system;

determining a target partition according to the key value pair and a partition mechanism of the Kafka cluster, and issuing the service data to the target partition; wherein the issuing the service data to the target partition includes: and encapsulating attribute information of the service data, the key value pairs and the service data into a Kafka message body, and issuing the Kafka message body to the target partition, wherein the attribute information of the service data comprises at least one item of a transmission mode of the service data, a storage format of the service data and a type of the service data.

2. The method of claim 1, wherein the key-value pair is an identification code that can be resolved as an integer;

analyzing the key value pairs, and carrying out remainder operation on integers obtained by analyzing the key value pairs and the partition number included in the theme divided by the Kafka cluster for the service data;

3. The method of claim 1, wherein the number of the second business systems is at least one, and the at least one second business system shares the same key-value pair with the first business system.

4. A business data interaction method is applied to a Kafka cluster, and comprises the following steps:

receiving service data issued by a first service system, and storing the service data to a target partition, wherein the storing the service data to the target partition includes: storing attribute information of service data and the service data to a target partition; the attribute information of the service data comprises at least one item of a transmission mode of the service data, a storage format of the service data and a type of the service data; the target partition is determined by the first service system according to a key value pair and a partition mechanism of the Kafka cluster, wherein the key value pair is used for identifying service data interacted between the first service system and the second service system; the key value pair is a message queue identification code generated by encrypting a first identity identifier and a second identity identifier by the first service system, the first identity identifier is obtained by encrypting identity information of the first service system by the first service system, and the second identity identifier is obtained by encrypting identity information of the second service system by the first service system;

and determining a target partition according to the key value pair and the partition mechanism of the Kafka cluster, acquiring the service data in the target partition, and sending the service data to the second service system.

5. The method of claim 4, wherein the key-value pairs are identification codes that can be resolved into integers;

6. The method of claim 5, wherein prior to storing the traffic data to the target partition, the method further comprises:

7. The method of claim 4, wherein storing the traffic data to a target partition comprises:

acquiring the service data in the target partition, including:

and determining the fragments for storing the service data in the target subarea by using the offset for identifying the service data, and acquiring the service data in the fragments.

8. A service data interaction method is applied to a second service system for realizing Kafka consumer function, and comprises the following steps:

acquiring a key value pair shared between a first service system and a second service system, wherein the key value pair is used for identifying service data interacted between the first service system and the second service system; wherein the obtaining of the key-value pair shared between the first service system and the second service system includes: encrypting the identity information of the second service system to generate a second identity; acquiring the identity information of the first service system, and encrypting the identity information of the first service system to generate a first identity mark; encrypting according to the second identity identifier and the first identity identifier to generate a message queue identification code, and using the message queue identification code as a key value pair shared between the first service system and the second service system;

the Kafka cluster acquires the service data sent by the Kafka cluster in a target partition, the target partition is determined by the Kafka cluster according to the key value pair and a partition mechanism of the Kafka cluster, and the attribute information of the service data and the service data are stored in the target partition; the attribute information of the service data includes at least one of a transmission mode of the service data, a storage format of the service data, and a type of the service data.

9. A service data interaction device, applied to a first service system implementing Kafka producer function, comprising:

the system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring interactive service data between a first service system and a second service system and key value pairs shared between the first service system and the second service system, and the key value pairs are used for identifying the interactive service data between the first service system and the second service system; the obtaining unit, when obtaining the key value pair shared between the first service system and the second service system, is specifically configured to: encrypting the identity information of the first service system to generate a first identity identifier of the first service system; acquiring the identity information of the second service system, encrypting the identity information of the second service system and generating a second identity; encrypting the first identity identifier and the second identity identifier to generate a message queue identification code, and using the message queue identification code as a key value pair shared between the first service system and the second service system;

the issuing unit is used for issuing the service data acquired by the acquiring unit to the target partition determined by the processing unit; when the issuing unit is configured to issue the service data acquired by the acquiring unit to the target partition determined by the processing unit, the issuing unit is specifically configured to: and encapsulating attribute information of the service data, the key value pairs and the service data into a Kafka message body, and issuing the Kafka message body to the target partition, wherein the attribute information of the service data comprises at least one item of a transmission mode of the service data, a storage format of the service data and a type of the service data.

10. The apparatus of claim 9, wherein the key-value pair is an identification code that can be resolved as an integer;

11. A service data interaction device is applied to a Kafka cluster, and comprises:

the receiving unit is used for receiving the service data issued by the first service system and receiving the key value pair and the subscription request sent by the second service system; the key value pair is a message queue identification code generated by encrypting a first identity identifier and a second identity identifier by the first service system, the first identity identifier is obtained by encrypting identity information of the first service system by the first service system, and the second identity identifier is obtained by encrypting identity information of the second service system by the first service system;

a storage unit, configured to store the service data received by the receiving unit to a target partition, where the target partition is determined by the first service system according to a key-value pair and a partition mechanism of the Kafka cluster, and the key-value pair is used to identify service data interacted between the first service system and the second service system; when the storage unit stores the service data received by the receiving unit into the target partition, the storage unit is specifically configured to: storing attribute information of service data and the service data to the target subarea; the attribute information of the service data comprises at least one item of a transmission mode of the service data, a storage format of the service data and a type of the service data;

the processing unit is further configured to determine a target partition according to the key value pair received by the receiving unit and the partition mechanism of the Kafka cluster, and acquire the service data in the target partition;

12. A service data interaction device, which is applied to a second service system for implementing Kafka consumer function, and comprises:

the system comprises a processing unit, a first service system and a second service system, wherein the processing unit is used for acquiring a key value pair shared between the first service system and the second service system, and the key value pair is used for identifying service data interacted between the first service system and the second service system; the processing unit, when obtaining the key value pair shared between the first service system and the second service system, is specifically configured to: encrypting the identity information of the second service system to generate a second identity of the second service system; acquiring the identity information of the first service system, and encrypting the identity information of the first service system to generate a first identity mark; encrypting the first identity identifier and the second identity identifier to generate a message queue identification code, and using the message queue identification code as a key value pair shared between the first service system and the second service system;

a receiving unit, configured to receive service data sent by a Kafka cluster, where the service data sent by the Kafka cluster is obtained by the Kafka cluster in a target partition, the target partition is determined by the Kafka cluster according to the key value pair and a partition mechanism of the Kafka cluster, and attribute information of the service data and the service data are stored in the target partition; the attribute information of the service data includes at least one of a transmission mode of the service data, a storage format of the service data, and a type of the service data.

13. A service data interaction system is characterized by comprising a Kafka cluster and at least one service system, wherein the at least one service system comprises a first service system for realizing the Kafka producer function and a second service system for realizing the Kafka consumer function;

the first business system has a function of implementing the method of any one of claims 1 to 3;

the Kafka cluster having a function of implementing the method of any one of claims 4 to 7;

the second service system having functionality to implement the method of claim 8.

14. A service data interaction device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the service data interaction method according to any one of claims 1 to 3, or implements the service data interaction method according to any one of claims 4 to 7, or implements the service data interaction method according to claim 8 when executing the program.

15. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the service data interaction method according to any one of claims 1 to 3, or implements the service data interaction method according to any one of claims 4 to 7, or implements the service data interaction method according to claim 8.