CN115623071A - Method and system for distributing publish-subscribe messages of single machine and multiple clients - Google Patents
Method and system for distributing publish-subscribe messages of single machine and multiple clients Download PDFInfo
- Publication number
- CN115623071A CN115623071A CN202211077361.7A CN202211077361A CN115623071A CN 115623071 A CN115623071 A CN 115623071A CN 202211077361 A CN202211077361 A CN 202211077361A CN 115623071 A CN115623071 A CN 115623071A
- Authority
- CN
- China
- Prior art keywords
- message
- client
- cache
- topic
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 28
- 238000009826 distribution Methods 0.000 claims description 19
- 230000007246 mechanism Effects 0.000 claims description 12
- 238000003860 storage Methods 0.000 claims description 10
- 230000005540 biological transmission Effects 0.000 abstract description 2
- 238000004364 calculation method Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 238000007726 management method Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Transfer Between Computers (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method and a system for distributing publish-subscribe messages of a single machine and multiple clients, belongs to the technical field of database data transmission, and aims to solve the technical problems of reducing the pressure of a server and reducing the occupation of a memory. The method comprises the following steps: creating a pull request thread facing to multiple clients, wherein the pull request thread is used for pulling a theme message from a server based on a pull request; creating a global cache facing to multiple clients; for each topic message published to a server by a user, storing data of each topic message into a global cache based on a pull request, and storing a read-only reference of each topic message into a cache of each client subscribing the topic message; when each client side obtains the subscribed topic messages, the data of the topic messages are copied from the global cache to the cache of the client side based on the read-only references of the topic messages.
Description
Technical Field
The invention relates to the technical field of database data transmission, in particular to a method and a system for distributing publish-subscribe messages of a single machine and multiple clients.
Background
Along with people's demand to the energy is bigger and bigger, the collection control of the energy is more and more towards the control management that becomes more meticulous, consequently, production management requirement to the energy is more and more accurate, thereby the field of digital energy has been produced, a large amount of real-time collection system produce a large amount of chronogenesis data, thereby produced to the time sequence data and saved, demand such as calculation, especially convection current calculation, feedback control system need know flow calculation result in order to control under certain scene, operations such as notice, the demand that the while large-screen demonstrates also is one of more demand, alarm system also needs to know the change simultaneously.
For this scenario, a publish/subscribe model is generally adopted to build a solution, publish/subscribe messaging is an asynchronous inter-service communication mode, and in the publish/subscribe mode, any message published to a topic is received immediately by all subscribers to the topic. Publish/subscribe messaging may be used to enable event-driven architectures, or separate applications, to improve performance, reliability, and extensibility. In a time-series data flow calculation scenario, data is generated continuously, and in a large-scale application scenario, a user has a high requirement on data acquisition performance, and when a pull-mode design is adopted, the following problems exist:
(1) Each client continuously requests data from the server, and after a large number of clients are accessed, the server can bear a large number of continuous requests, and particularly under a single-machine multi-client architecture, each client continuously requests data from the server, so that the load of the system is increased, and the performance of the system is reduced;
(2) When multiple clients subscribe to the same theme at the same time, the data is often copied for multiple times, so that the occupation of the memory is greatly increased.
Based on the above analysis, how to reduce the pressure of the server and reduce the memory usage is a technical problem to be solved.
Disclosure of Invention
The technical task of the invention is to provide a method and a system for distributing publish-subscribe messages of a single machine and multiple clients to solve the problems of reducing the pressure of a server and reducing the memory occupation.
In a first aspect, the present invention provides a method for distributing publish-subscribe messages of a single machine and multiple clients, which is applied to a publish-subscribe message system including a server and multiple clients, where the multiple clients are configured in the same single machine, and the method includes the following steps:
creating a pull request thread facing to multiple clients, wherein the pull request thread is used for pulling a theme message from a server based on a pull request;
creating a global cache facing to a plurality of clients, wherein the global cache is used for storing data of theme messages pulled from the server based on pull requests;
for each topic message published to a server by a user, storing data of each topic message into a global cache based on a pull request, and storing a read-only reference of each topic message into a cache of each client subscribing the topic message;
when each client acquires the subscribed theme message, copying the data of the theme message from the global cache to the cache of the client based on the read-only reference of the theme message.
Preferably, the method further comprises the following steps: creating a shared cache facing a server;
the server side pushes the data of each topic message issued by the user to a shared cache for storage;
based on the pull request, the server side sends the data of each topic message in the shared cache to the global cache for storage, and stores the read-only reference of each topic message to the cache of each client side subscribing the topic message.
Preferably, the server stores the read-only references of each topic message in the cache of each client subscribing to the topic message through a copy-on-write mechanism.
Preferably, the read-only reference of the subject message is an address index of the subject message in the global cache.
Preferably, a queue is configured in the cache of each client;
for each client, the read-only references of the client subscription topic messages are stored in its in-cache queue.
In a second aspect, the invention relates to a single-machine multi-client publish-subscribe message distribution system, which comprises a server and a plurality of clients, wherein the plurality of clients are configured on the same single machine, the plurality of clients are configured with a pull request thread, and the pull request thread is used for pulling a subject message from the server based on a pull request;
the client sides are provided with a global cache, and the global cache is used for storing data of the theme message pulled from the server side based on the pull request;
for each topic message issued by the user to the server, the server is configured to: storing data of each topic message into a global cache based on a pull request, and storing a read-only reference of each topic message into a cache of each client subscribing the topic message;
when each client acquires the subscribed topic message, the client is used for copying the data of the topic message from the global cache to the cache of the client based on the read-only reference of the topic message.
Preferably, the server is configured with a shared cache;
the server is used for pushing the data of each theme message issued by the user to a shared cache for storage;
based on the pull request, the server is used for sending the data of each topic message in the shared cache to the global cache for storage, and storing the read-only reference of each topic message to the cache of each client subscribing the topic message.
Preferably, the server is configured to store the read-only reference of each topic message in a cache of each client subscribing to the topic message through a copy-on-write mechanism.
Preferably, the read-only reference of the subject message is an address index of the subject message in the global cache.
Preferably, a queue is configured in the cache of each client;
for each client, the read-only references of the client subscription topic messages are stored in its in-cache queue.
The method and the system for distributing the publish-subscribe messages of the single machine and the plurality of clients have the following advantages that: each client does not independently perform pull request but creates a global cache, only a pull request thread is independently created, data is always read through the pull request and the read data of each topic message is stored in the global cache, the read-only reference of each topic message is stored in the client cache subscribing the topic message by using a distribution mechanism, and the data is copied from the global cache to the cache by the read-only reference when the client acquires the data, so that the pressure of a server is reduced, the utilization rate of a memory is improved, the operation efficiency of a publish-subscribe system is integrally improved, and the utilization rate of system resources and the system stability are greatly saved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed for the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on the drawings without creative efforts.
The invention is further described below with reference to the accompanying drawings.
FIG. 1 is a block diagram of a conventional publish-subscribe multi-client message distribution process;
fig. 2 is a block diagram of a single pull thread work flow in a publish-subscribe message distribution method for a single machine and multiple clients according to embodiment 1;
fig. 3 is a block diagram of a work flow of a copy-on-write mechanism in the publish-subscribe message distribution method of the single-machine multi-client in embodiment 1.
Fig. 4 is a block diagram of a process of acquiring data by a client in a Copy-on-Write mode in the publish-subscribe message distribution method of a single machine and multiple clients according to embodiment 1.
Detailed Description
The present invention is further described in the following with reference to the drawings and the specific embodiments so that those skilled in the art can better understand the present invention and can implement the present invention, but the embodiments are not to be construed as limiting the present invention, and the embodiments and the technical features of the embodiments can be combined with each other without conflict.
The embodiment of the invention provides a method and a system for distributing publish-subscribe messages of a single machine and multiple clients, which are used for solving the technical problems of reducing the pressure of a server and reducing the occupation of a memory.
Example 1:
as shown in fig. 1, in a conventional publish-subscribe system design with a single machine and multiple clients, each client sends a pull request to a server through a pull thread thereof, and pulls data of a topic message subscribed by the client based on the pull request, and if multiple clients request the same topic message, the server needs to copy and send the data of the topic message to each client in multiple copies, which increases the pressure of the server and occupies a large amount of memory.
The invention relates to a single machine multi-client publish-subscribe message distribution method, which is applied to a publish-subscribe message system comprising a server and a plurality of clients, wherein the plurality of clients are configured on the same single machine, and the method comprises the following steps:
s100, creating a pull request thread facing to multiple clients, wherein the pull request thread is used for pulling a theme message from a server based on a pull request;
and creating a global cache facing to multiple clients, wherein the global cache is used for storing data of the theme message pulled from the server based on the pull request.
Compared with the conventional design, in step S100 of this embodiment, each client does not separately perform pull request, but creates a global cache and separately creates a pull request thread at the client. The single pull request thread faces to the multiple clients, sends pull requests to the service all the time, pulls data of all theme messages of the service to a client degree based on the pull requests, and stores the data in the global cache.
S200, for each topic message published to the server by the user, storing data of each topic message in a global cache based on a pull request, and storing a read-only reference of each topic message in a cache of each client subscribing to the topic message, as specifically shown in fig. 2.
As a specific implementation of this embodiment, as shown in fig. 3, the server stores the read-only reference of each topic message into the cache of each client subscribing to the topic message through a copy-on-write mechanism, where the read-only reference of the topic message in this embodiment is an address index of the topic message in the global cache.
That is, in this embodiment, the server stores the data of the same topic message in the global cache, and stores the address index of the global cache for subscribing to the topic message in the cache of each client subscribing to the topic message.
Compared with the conventional design, a plurality of clients subscribe to the same theme instantly, and in the embodiment, the server sends the data of the same theme message to the global cache once without copying the data for sending for many times, so that the pressure of the server is reduced. Meanwhile, the server side sends the address index of the subject message in the global cache to the client side cache subscribing the subject message by using a distribution mechanism, namely the current client side has read operation authority to the subject message subscribed by the current client side and cannot perform write operation, thereby ensuring the data integrity and the irreparable modification of each subject message in the global cache.
S300, when each client side obtains the subscribed theme message, copying the data of the theme message from the global cache to the cache of the client side based on the read-only reference of the theme message.
After step S200 is executed, the index address of the subscription subject message in the global cache is stored in the client cache, and if the client needs to obtain the data of the subscription subject message and perform a write operation, the client needs to copy the data of the corresponding subject message from the global cache to the cache based on the index address of the subscription subject message in the global cache, and then perform a write operation on the data stored in the cache, as shown in fig. 4.
As a modification of the method of the present embodiment, the following operations are further included in step S100: and creating a shared cache facing the server, wherein the shared cache is used for storing data of each topic message issued to the server by the user.
Correspondingly, in step S200, based on the pull request, the server sends the data of each topic message in the shared cache to the global cache for storage, and stores the read-only reference of each topic message to the cache of each client subscribing to the topic message.
In the method of the embodiment, each client does not independently perform pull request but creates a cache, only a pull request thread is created independently, data is read all the time, a distribution mechanism is utilized to store the received data in the client cache appointed by the data storage tape, a Copy-on-Write mechanism is utilized, only read reference of the data is stored in the client cache for the same issuing message every time, and the data is copied to the client only when a user really acquires the data, so that the occupation of a large amount of instantaneous memory is greatly reduced.
Example 2:
the invention relates to a single-machine multi-client publishing and subscribing message distribution system which comprises a server and a plurality of clients, wherein the clients are configured on the same single machine, and a pull request thread is configured on the clients and used for pulling a theme message from the server based on a pull request; a global cache is configured on a plurality of client sides and is used for storing data of the theme messages pulled from the server sides based on pull requests. The system can execute the method disclosed in embodiment 1 to realize message distribution.
For each topic message issued by the user to the server, the server is configured to: and storing the data of each topic message into a global cache based on the pull request, and storing the read-only reference of each topic message into a cache of each client subscribing the topic message.
As a specific implementation, the server is configured to store, through a copy-on-write mechanism, the read-only reference of each topic message into a cache of each client subscribing to the topic message, where the read-only reference of the topic message in this embodiment is an address index of the topic message in the global cache.
That is, in this embodiment, the server stores the data of the same topic message in the global cache, and stores the address index of the global cache for subscribing to the topic message in the cache of each client subscribing to the topic message.
Compared with the conventional design, a plurality of clients subscribe to the same theme instantly, and in the embodiment, the server sends the data of the same theme message to the global cache once without copying the data for sending for many times, so that the pressure of the server is reduced. Meanwhile, the server side sends the address index of the subject message in the global cache to the client side cache subscribing the subject message by using a distribution mechanism, namely the current client side has read operation authority to the subject message subscribed by the current client side and cannot perform write operation, thereby ensuring the data integrity and the irreparable modification of each subject message in the global cache.
When each client acquires the subscribed theme message, the client is used for copying the data of the theme message from the global cache to the cache of the client based on the read-only reference of the theme message.
The client cache stores the index address of the subscription subject message in the global cache, if the client needs to acquire the data of the subscription subject message and perform write operation, the client needs to copy the data of the corresponding subject message from the global cache to the cache based on the index address of the subject message in the global cache, and then perform write operation on the data stored in the cache.
As an improvement of the system of this embodiment, a shared cache is configured at the server, and the shared cache is used for storing data of each topic message issued by the user to the server.
Correspondingly, based on the pull request, the server is used for sending the data of each topic message in the shared cache to the global cache for storage, and storing the read-only reference of each topic message to the cache of each client subscribing the topic message.
While the invention has been shown and described in detail in the drawings and in the preferred embodiments, it is not intended to limit the invention to the embodiments disclosed, and it will be apparent to those skilled in the art that many more embodiments of the invention are possible that combine the features of the different embodiments described above and still fall within the scope of the invention.
Claims (10)
1. A single machine multi-client publish-subscribe message distribution method is characterized in that the method is applied to a publish-subscribe message system comprising a server and a plurality of clients, wherein the plurality of clients are configured on the same single machine, and the method comprises the following steps:
creating a pull request thread facing to multiple clients, wherein the pull request thread is used for pulling a theme message from a server based on a pull request;
creating a global cache facing to multiple clients, wherein the global cache is used for storing data of a theme message pulled from a server based on a pull request;
for each topic message published to a server by a user, storing data of each topic message into a global cache based on a pull request, and storing a read-only reference of each topic message into a cache of each client subscribing the topic message;
when each client side obtains the subscribed topic messages, the data of the topic messages are copied from the global cache to the cache of the client side based on the read-only references of the topic messages.
2. The method for distributing publish-subscribe messages of a stand-alone multi-client according to claim 1, further comprising the steps of: creating a shared cache facing a server;
the server side pushes the data of each topic message issued by the user to a shared cache for storage;
based on the pull request, the server side sends the data of each topic message in the shared cache to the global cache for storage, and stores the read-only reference of each topic message to the cache of each client side subscribing the topic message.
3. The stand-alone multi-client publish-subscribe message distribution method according to claim 1 or 2, wherein the server stores the read-only references of each topic message into the cache of each client subscribing to the topic message through a copy-on-write mechanism.
4. The method for distributing publish-subscribe messages of a single machine multiple clients according to claim 1 or 2, wherein the read-only reference of the subject message is an address index of the subject message in the global cache.
5. The method for distributing publish-subscribe messages of a single machine multi-client according to claim 1 or 2, wherein a queue is configured in the cache of each client;
for each client, the read-only references of the client subscription topic messages are stored in its in-cache queue.
6. A single-machine multi-client publish-subscribe message distribution system comprises a server and a plurality of clients, wherein the clients are configured on the same single machine,
the plurality of client sides are configured with a pull request thread, and the pull request thread is used for pulling the theme message from the server side based on a pull request;
the client sides are provided with a global cache, and the global cache is used for storing data of the theme messages pulled from the server side based on pull requests;
for each topic message issued by the user to the server, the server is configured to: storing data of each topic message into a global cache based on a pull request, and storing a read-only reference of each topic message into a cache of each client subscribing the topic message;
when each client acquires the subscribed topic message, the client is used for copying the data of the topic message from the global cache to the cache of the client based on the read-only reference of the topic message.
7. The standalone multi-client publish-subscribe message distribution system as claimed in claim 6, wherein said server is configured with a shared cache;
the server is used for pushing the data of each theme message issued by the user to a shared cache for storage;
based on the pull request, the server is used for sending the data of each topic message in the shared cache to the global cache for storage, and storing the read-only reference of each topic message to the cache of each client subscribing the topic message.
8. The standalone multi-client publish-subscribe message distribution system according to claim 6 or 7, wherein said server is configured to store the read-only reference of each topic message into the cache of each client subscribing to said topic message through a copy-on-write mechanism.
9. The standalone multi-client publish-subscribe message distribution system as claimed in claim 6 or 7, wherein the read-only reference of the subject message is an address index of the subject message in the global cache.
10. The standalone multi-client publish-subscribe message distribution system according to claim 6 or 7, wherein each client has a queue configured in its cache;
for each client, the read-only references of the client subscription topic messages are stored in its in-cache queue.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211077361.7A CN115623071A (en) | 2022-09-05 | 2022-09-05 | Method and system for distributing publish-subscribe messages of single machine and multiple clients |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211077361.7A CN115623071A (en) | 2022-09-05 | 2022-09-05 | Method and system for distributing publish-subscribe messages of single machine and multiple clients |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115623071A true CN115623071A (en) | 2023-01-17 |
Family
ID=84858684
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211077361.7A Pending CN115623071A (en) | 2022-09-05 | 2022-09-05 | Method and system for distributing publish-subscribe messages of single machine and multiple clients |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115623071A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117914943A (en) * | 2024-03-20 | 2024-04-19 | 深圳华锐分布式技术股份有限公司 | Data subscription and pushing method, device, equipment and medium |
-
2022
- 2022-09-05 CN CN202211077361.7A patent/CN115623071A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117914943A (en) * | 2024-03-20 | 2024-04-19 | 深圳华锐分布式技术股份有限公司 | Data subscription and pushing method, device, equipment and medium |
| CN117914943B (en) * | 2024-03-20 | 2024-06-14 | 深圳华锐分布式技术股份有限公司 | Data subscription and pushing method, device, equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Fu et al. | A fair comparison of message queuing systems | |
| CN106874424B (en) | A kind of collecting webpage data processing method and system based on MongoDB and Redis | |
| US20060047666A1 (en) | Control of publish/subscribe messaging | |
| CN112579148B (en) | Service message processing method and device based on service agent and electronic equipment | |
| CN114363407B (en) | Message service method and device, readable storage medium and electronic equipment | |
| CN113032419B (en) | Multi-source data aggregation search method, device, equipment and storage medium | |
| CN110648178A (en) | Method for increasing kafka consumption capacity | |
| CN114064211B (en) | Video stream analysis system and method based on end-side-cloud computing architecture | |
| CN115623071A (en) | Method and system for distributing publish-subscribe messages of single machine and multiple clients | |
| CN112055061A (en) | Distributed message processing method and device | |
| CN114827171B (en) | Information synchronization method, apparatus, computer device and storage medium | |
| CN115328664A (en) | Message consumption method, device, equipment and medium | |
| CN111526188A (en) | System and method for ensuring zero data loss based on Spark Streaming in combination with Kafka | |
| CN111917814A (en) | Data publishing method, data subscribing method, data publishing device, data subscribing system and readable storage medium | |
| CN110704212B (en) | Message processing method and device | |
| CN116842090A (en) | Accounting system, method, equipment and storage medium | |
| CN111475315A (en) | Server and subscription notification push control and execution method | |
| CN110955461A (en) | Processing method, device and system of computing task, server and storage medium | |
| CN116107710A (en) | Method, apparatus, device and medium for processing offline rendering tasks | |
| Aahlad et al. | Asynchonrous Notifications Among Distributed Objects. | |
| CN117692418A (en) | Message processing method, device, computer equipment and storage medium | |
| CN115883639A (en) | Web real-time message pushing method and device, equipment and storage medium | |
| CN116166451B (en) | Dynamic adjustment method, system and device for topic number and storage medium | |
| US9613150B2 (en) | Remote viewing of documents via the web in real-time | |
| CN112965796B (en) | Task scheduling system, method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |