WO2022120806A1 - 面向高性能计算多云的分布式消息传递方法及*** - Google Patents

面向高性能计算多云的分布式消息传递方法及*** Download PDF

Info

Publication number
WO2022120806A1
WO2022120806A1 PCT/CN2020/135768 CN2020135768W WO2022120806A1 WO 2022120806 A1 WO2022120806 A1 WO 2022120806A1 CN 2020135768 W CN2020135768 W CN 2020135768W WO 2022120806 A1 WO2022120806 A1 WO 2022120806A1
Authority
WO
WIPO (PCT)
Prior art keywords
message
queue
cloud
service
model
Prior art date
Application number
PCT/CN2020/135768
Other languages
English (en)
French (fr)
Inventor
林帅康
马健
刘阳
朱和胜
温书豪
赖力鹏
Original Assignee
深圳晶泰科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳晶泰科技有限公司 filed Critical 深圳晶泰科技有限公司
Priority to PCT/CN2020/135768 priority Critical patent/WO2022120806A1/zh
Publication of WO2022120806A1 publication Critical patent/WO2022120806A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication

Definitions

  • the invention relates to the field of communications, in particular to a distributed message delivery method and system for high-performance computing and multi-cloud.
  • High Performance Computing refers to the use of aggregated computing power to handle data-intensive computing tasks that cannot be accomplished by standard workstations, including modeling, simulation, and rendering. It plays an increasingly important role in modern scientific research and industrial production.
  • the multi-cloud high-performance computing platform provides a powerful computing cluster that can rapidly expand and contract.
  • the multi-cloud computing cluster resources break through geographical restrictions, enabling computing resources to be distributed globally across regions.
  • the task also has the distributed characteristics. This feature is very different from traditional high-performance computing clusters built in the same region and network. For this reason, in order to improve the programming and development efficiency of enterprise developers and reduce the difficulty of multi-cloud and high-performance programming for enterprise developers, rich cloud middleware services are essential.
  • the multi-cloud distributed messaging method is the information delivery bridge between tasks and tasks and between tasks and services in the cloud computing environment.
  • a distributed message delivery system for high-performance computing and multi-cloud is provided, which can improve the convenience of communication.
  • a distributed messaging approach for HPC multi-cloud including:
  • Production message According to the usage scenario, create a message model, obtain the information to be delivered, fill in the corresponding information according to the information to be delivered and the message model according to the requirements of the message template format, and generate a message instance.
  • the message template includes: message header, message carrier, and message trailer ;
  • the message producer creates a network connection to the message service through the SDK provided by the multi-cloud distributed message queue service, and pushes the generated message instance to the specified message queue in the multi-cloud distributed message queue service through the connection;
  • the multi-cloud distributed message queue service interprets the pushed messages, performs message routing according to the relevant field information in the message header, and pushes the relevant messages to message consumers on different message models.
  • the message header includes: message queue name, message authentication information, whether the message is persistent, message sender, message receiver, message type, creation time, and message version number; different information is distinguished according to the message header information.
  • message model a message model
  • the message interpretation route includes: the multi-cloud distributed message queue service disassembles the pushed message, obtains the message header information of the message, searches the currently created message queue according to the message queue name in the message header, and finds the current queue, Compare whether the consumption type of the message header is consistent with the attributes of the message queue where it is located, confirm whether the current message producer or message consumer has the right to operate the message queue according to the message authentication information in the message header, and determine one-to-one or one-to-one according to the message model To-many correlation, establish a one-to-one or one-to-many message consumption model.
  • the message entry queue further includes message confirmation: if a message confirmation instruction or an enabled message confirmation mechanism is received, the multi-cloud distributed message service receives the message fed back by the message consumer and the message is normally consumed. The message service feeds back the received message and feeds back to the message producer that the message has been received normally.
  • the message entry queue further includes message persistence: if message persistence is enabled or a message persistence instruction is received when the message queue is created, the multi-cloud distributed message queue service will send each message on the queue to the service. The received information is persisted.
  • the message model includes: one or more of a point-to-point model, a publish-subscribe model, a remote call model, a broadcast model, and a work queue model;
  • the step of entering the message into the queue further includes: operating according to different message models;
  • the multi-cloud distributed message queue service will generate a unique ID for the message producer.
  • the message will be consumed according to the associated ID.
  • the producer replies to the message and sends it to the correct message producer;
  • the message model is a publish-subscribe model, receive subscription instructions, perform multi-queue delivery according to the number of subscribed message consumers, subscribe topic messages according to different message queue names, and control the sending of messages of the corresponding subscribed topics to message consumers according to the message queue name. ;
  • the consumption model is the work queue model, different messages are taken out from the consumption queue one by one and assigned to the message consumers, and the same message will not be consumed by different consumers twice.
  • it also includes interconnection: setting up message service intermediaries in different areas, deploying message service intermediaries in each area, interconnecting message service intermediaries, and message producers by directly accessing message service intermediaries in their regions Communicate with consumers in other regions.
  • it also includes nearby access: deploying a message queue cluster in each public cloud, the message queue service is accessed nearby, and deploying a message queue cluster in different clouds uses the same API interface for access;
  • the message tail includes: message digest algorithm, digest value, check value;
  • Message queue attributes include: one or more of queue name and queue persistence.
  • a distributed messaging system for high-performance computing multi-cloud including:
  • Node server apply for multiple node servers in each cloud, deploy clusters, and create access portals for cluster services respectively;
  • Message queue center controller deploy the message queue center controller, and configure the access entry of each cluster service to the message queue center controller;
  • the message queue center controller includes:
  • Production message module According to the usage scenario, create a message model, obtain the information to be delivered, fill in the corresponding information according to the information to be delivered and the message model according to the requirements of the message template format, and generate a message instance.
  • the message template includes: message header, message carrier, message tail;
  • the message producer creates a network connection to the message service through the SDK provided by the multi-cloud distributed message queue service, and pushes the generated message instance to the specified message queue in the multi-cloud distributed message queue service through the connection;
  • the multi-cloud distributed message queue service interprets the pushed messages, performs message routing according to the relevant field information in the message header, and pushes the relevant messages to message consumers on different message models.
  • the message header includes: message queue name, message authentication information, whether the message is persistent, message sender, message receiver, message type, creation time, and message version number; different information is distinguished according to the message header information.
  • message model a message model
  • the message interpretation and routing module also includes: the multi-cloud distributed message queue service disassembles the pushed message, obtains the message header information of the message, searches the currently created message queue according to the message queue name in the message header, and finds the current message queue. Queue, compare whether the consumption type of the message header is consistent with the attribute of the message queue, confirm whether the current message producer or message consumer has permission to operate the message queue according to the message authentication information in the message header, and determine one-to-one according to the message model Or one-to-many correlation, establish a one-to-one or one-to-many message consumption model;
  • the message entry queue module further includes a message confirmation unit: if a message confirmation instruction or an enabled message confirmation mechanism is received, the multi-cloud distributed message service receives an instruction that the message fed back by the message consumer is normally consumed, and the multi-cloud distributed message service responds to the received message. The received message is fed back, and it is fed back to the message producer that the message has been received normally;
  • message persistence unit If message persistence is enabled when the message queue is created or a message persistence command is received, the multi-cloud distributed message queue service performs data persistence for each received message on the queue.
  • an interconnection module setting up message service intermediate stations in different areas, deploying message service intermediate stations in each area, and interconnecting message service intermediate stations, and message producers directly access the message service in the area where they are located.
  • Intermediate stations communicate with consumers in other areas;
  • the nearest access module Deploy message queue clusters in each public cloud, the message queue service is accessed nearby, and the same API interface is used to deploy message queue clusters in different clouds.
  • the above-mentioned distributed message delivery method and system for high-performance computing and multi-cloud, the message queue definition of message transmission format for multi-cloud high-performance computing, and the support of multi-message models make the development and integration of high-performance computing on the cloud more efficient and convenient.
  • Distributed messaging services on multiple clouds, running on computing resources and executing programs with high performance can achieve cross-regional message delivery without caring about the actual network location where the program is currently located.
  • the distributed message service shields the complexity of cross-regional and heterogeneous networks. Developers only need to access the interface through a unified distributed message service and focus on the realization of computing workflow and upper-layer business.
  • the present invention realizes the nearby access of message queue services by deploying message queue clusters in major public cloud vendors.
  • the message queue cluster access deployed in different cloud vendors uses the same API interface, which reduces the difficulty of service integration, and developers do not need to integrate according to different cloud vendors.
  • the multi-cloud distributed message queue service is naturally distributed globally and has strong service high availability.
  • FIG. 1 is a partial flowchart of a distributed message delivery method for high-performance computing and multi-cloud according to an embodiment of the present invention
  • FIG. 2 is a schematic diagram of a message model according to an embodiment of the present invention.
  • the present invention can realize cross-regional message transmission through the distributed message service built on the multi-cloud, running on the computing resources and executing the program with high performance without caring about the current real network location of the program.
  • the distributed message service shields the complexity of cross-regional and heterogeneous networks. Developers only need to access the interface through a unified distributed message service and focus on the realization of computing workflow and upper-layer business.
  • the present invention provides a distributed multi-message transfer method suitable for multi-cloud high-performance computing scenarios, establishes an information and data transfer mechanism between tasks and service systems in a multi-cloud high-performance computing platform, shields the complexity of cross-regional network communication, and is oriented towards It provides a reference implementation for building a communication model on a multi-cloud high-performance computing environment.
  • a distributed message delivery method for high-performance computing and multiple clouds includes:
  • Step S101 producing a message: according to the usage scenario, create a message model, obtain the information to be delivered, fill in the corresponding information according to the information to be delivered and the message model according to the requirements of the message template format, and generate a message instance, the message template includes: a message header, a message carrier , message tail;
  • Step S103 the message enters the queue: the message producer creates a network connection to the message service through the SDK provided by the multi-cloud distributed message queue service, and pushes the generated message instance to the designated message queue in the multi-cloud distributed message queue service through the connection ;
  • Step S105 message interpretation and routing: the multi-cloud distributed message queue service interprets the pushed message, performs message routing according to the relevant field information in the message header, and pushes the relevant message to message consumers on different message models.
  • a multi-cloud distributed message queue service which accepts and forwards messages. Think of it as a post office: when you put mail to be published in a mailbox, you can ensure that the mail is eventually delivered to the recipient.
  • the Distributed Message Queuing Service is a PO Box, a Post Office, and a Postman.
  • the main difference between the Distributed Message Queuing Service and the Post Office is that it does not handle paper, but instead receives, stores and forwards binary data messages.
  • the message model of this embodiment includes one or more of a point-to-point model, a publish-subscribe model, a remote calling model, a broadcast model, and a work queue model.
  • the message producer can be a producer, a sender, a client, etc. according to different message models, as long as the function of the message producer of the present invention is implemented.
  • the message consumers can be consumers, receivers, servers, etc. according to different message models or different usage scenarios, as long as the functions of the message consumers of the present invention are implemented.
  • the message model is a point-to-point model: the producer produces the message and sends it to the message queue, and the consumer consumes the message from the message queue.
  • the message model is a publish-subscribe model, receive subscription instructions, perform multi-queue delivery according to the number of subscribed message consumers, subscribe topic messages according to different message queue names, and control the sending of messages of the corresponding subscribed topics to message consumers according to the message queue name.
  • consumer A and consumer B subscribe to messages of different topic types through message queues, such as messages with a topic of log type and messages with a topic of monitoring type.
  • message consumers Through the publish-subscribe model, message consumers only subscribe to the message topics they care about, and messages on other topics are ignored.
  • the multi-cloud distributed message queue service will generate a unique ID for the message producer.
  • the message will be consumed according to the associated ID.
  • the producer replies to the message and sends it to the correct message producer.
  • the difference between the remote call model and other message models is that it is a response model.
  • the remote call model is suitable for scenarios where the message loss rate is extremely low and the message delivery reliability is extremely high. Blocking waiting, waiting for the server to return a signal after consuming the message to tell the sender that the message has been delivered safely.
  • the message queue under the broadcast model supports the same message to be consumed multiple times
  • the sender sends the message to the broadcast message queue, and both receiver A and receiver B can receive the message, so as to simulate Mechanisms for consuming broadcasts.
  • the consumption model is the work queue model, different messages are taken out from the consumption queue one by one and assigned to the message consumers, and the same message will not be consumed by different consumers twice.
  • the implementation mechanism of the work queue model is similar to that of the broadcast model, the difference is that the same message cannot be received by different consumers, and different messagers will fetch different messages from the message queue one by one.
  • the network connection created by the message producer to the message service through the SDK (Software Development Kit) provided by the multi-cloud distributed message service can be performed in or before the message production step, or in the message entry queue step, as long as the generated message is processed.
  • the message instance can be pushed to the specified message queue in the multi-cloud distributed message queue service through this connection.
  • the message producer creates a network connection to the message service through the SDK (Software Development Kit) provided by the multi-cloud distributed message queue service, fills in the corresponding fields according to the requirements of the message template format, and generates a message instance.
  • SDK Software Development Kit
  • the message producer creates a network connection to the message service through the SDK provided by the multi-cloud distributed message queue service, and pushes the generated message instance to the specified message queue in the message queue service through the connection. message model to operate.
  • the multi-cloud distributed message service interprets the pushed messages, performs message routing according to the relevant field information in the message header, and pushes the relevant messages to message consumers on different message models.
  • Message queue attributes include: one or more of queue name and queue persistence.
  • multi-cloud distributed message queue service acts as an intermediate depositor of messages, it is necessary to receive and forward messages to consumers for message confirmation to ensure message delivery.
  • the message entering the queue also includes message confirmation: if the message confirmation command or the enabled message confirmation mechanism is received, the multi-cloud distributed message service receives the message fed back by the message consumer and is instructed to be consumed normally, and the multi-cloud distributed message service processes the received message. Feedback, feedback to the message producer that the message has been received normally.
  • the message entry queue also includes message persistence: if message persistence is enabled when the message queue is created or a message persistence command is received, the multi-cloud distributed message queue service performs data persistence for each received message on the queue.
  • the multi-cloud distributed message service refers to data persistence for each received message on the queue to prevent the loss of messages in the memory of the storage computer due to service interruption.
  • the multi-cloud distributed message service can restore the original message data in the message queue from the persistent data service during the interruption and restart process.
  • the public network transmission is used by default.
  • the distributed message queue service can also transmit messages through the dedicated line.
  • the transmission stability and privacy of private line messages will be better, but it is also necessary to pay attention to whether the bandwidth of the private line is sufficient.
  • the message header is distinguished by the message header.
  • the fields contained in the message header are: message queue name, message authentication information, whether the message is persistent, message sender, message receiver, message type, creation time, and message version number.
  • the message carrier which stores the actual data content of the message.
  • the message tail is mainly to ensure the integrity of the data during the message transmission. It contains fields: message digest algorithm, digest value, and check value.
  • the message interpretation route includes: the multi-cloud distributed message queue service disassembles the pushed message, obtains the message header information of the message, searches the currently created message queue according to the message queue name in the message header, finds the current queue, and compares Whether the consumption type of the message header is consistent with the attributes of the message queue where it is located, confirm whether the current message producer or message consumer has permission to operate the message queue according to the message authentication information in the message header, and determine one-to-one or one-to-many according to the message model The correlation, establish a one-to-one or one-to-many message consumption model.
  • the distributed message delivery method for high-performance computing and multi-cloud in this embodiment further includes interconnection: setting up message service intermediate stations in different regions, deploying message service intermediate stations in each region, interconnecting message service intermediate stations, and producing messages. Consumers communicate with consumers in other regions by directly accessing the message service intermediate station in their region.
  • the multi-cloud-oriented distributed message delivery method for high-performance computing in this embodiment further includes nearby access: deploying a message queue cluster in each public cloud, the message queue service is accessed nearby, and deploying a message queue cluster in different clouds uses the same access method. API interface.
  • the present invention realizes the nearby access of message queue services by deploying message queue clusters in major public cloud vendors.
  • the message queue cluster access deployed in different cloud vendors uses the same API interface, which reduces the difficulty of service integration, and developers do not need to integrate according to different cloud vendors.
  • the multi-cloud distributed message queue service is naturally distributed globally and has strong service high availability.
  • a distributed message delivery system for high-performance computing and multiple clouds of the present invention includes:
  • Node server apply for multiple node servers in each cloud, deploy clusters, and create access portals for cluster services respectively;
  • Message Queue Center Controller Deploy the message queue center controller, and configure the access entry of each cluster service to the message queue center controller.
  • the message queue center controller of the present invention includes:
  • Production message module According to the usage scenario, create a message model, obtain the information to be delivered, fill in the corresponding information according to the information to be delivered and the message model according to the requirements of the message template format, and generate a message instance.
  • the message template includes: message header, message carrier, message tail;
  • the message producer creates a network connection to the message service through the SDK provided by the multi-cloud distributed message queue service, and pushes the generated message instance to the specified message queue in the multi-cloud distributed message queue service through the connection;
  • the multi-cloud distributed message queue service interprets the pushed messages, performs message routing according to the relevant field information in the message header, and pushes the relevant messages to message consumers on different message models.
  • High Performance Computing refers to the use of aggregated computing power to handle data-intensive computing tasks that cannot be accomplished by standard workstations, including modeling, simulation, and rendering.
  • High-performance computing on the cloud It refers to moving traditional high-performance computing clusters to the cloud, and endows them with the characteristics of on-demand expansion charges, diverse heterogeneous computing resources, and the ability to build clusters across the world.
  • Multi-cloud refers to the hybrid integration of multiple public clouds to expand the scale of computing resources.
  • a distributed system is a loosely coupled system composed of multiple processors interconnected through communication lines. From a processor in the system, other processors and corresponding resources are remote, and only its own resources are local.
  • a "message” is a unit of data sent between two computers. Messages can be very simple, such as containing only text strings, or more complex, possibly containing embedded objects. The message is sent to the queue.
  • a "message queue” is a container that holds messages during their transmission.
  • a message queue manager acts as a middleman when relaying messages from its source to its destination. The main purpose of queues is to provide routing and guarantee delivery of messages; if a receiver is unavailable when a message is sent, message queues hold the message until it can be successfully delivered.
  • Leader Node The message queue deployed by a single node has the risk of a single point of failure, that is to say, if the message queue service deployed on a single node fails, the entire message queue service will be unavailable, directly affecting the message producer. Messaging with the messager.
  • the message queue deployed by multiple nodes solves the problem of single-point failure in a large program. In order to maintain the message delivery management under multiple nodes, it is necessary to elect a leader node, and the producer and the messager communicate with the leader node. .
  • Kubernetes K8s for short, is an abbreviation formed by replacing the 8-character “ubernete” with 8. It is an open source, used to manage containerized applications on multiple hosts in the cloud platform. The goal of Kubernetes is to make the deployment of containerized applications simple and efficient (powerful). Kubernetes provides application deployment, planning, updating, and maintenance. a mechanism.
  • RabbitMQ is an open source message broker software (also known as message-oriented middleware) that implements the Advanced Message Queuing Protocol (AMQP).
  • AMQP Advanced Message Queuing Protocol
  • the system of the invention includes: distributed deployment design in multi-cloud environment, message model design, message template format design, multi-cloud distributed message transmission process, and message persistence.
  • the invention is oriented to the field of high-performance computing multi-cloud distributed message queue service.
  • the design goal of the distributed multi-message queue service is to meet the data interaction requirements of tasks and services in the high-performance computing environment built by multi-cloud, and to solve the complex cross-regional network arising from the multi-cloud environment. Improve the reliability of message transmission and reduce the complexity of cross-regional communication programming.
  • the message header in this embodiment includes: message queue name, message authentication information, whether the message is persistent, message sender, message receiver, message type, creation time, and message version number. Different message models are distinguished according to different message header information.
  • the message interpretation routing module of the present embodiment also includes: the multi-cloud distributed message queue service disassembles the pushed message, obtains the message header information of the message, and searches the currently created message according to the message queue name in the message header Queue, find the current queue, compare whether the consumption type of the message header is consistent with the attributes of the message queue, and confirm whether the current message producer or message consumer has permission to operate the message queue according to the message authentication information in the message header.
  • the message model Determine one-to-one or one-to-many dependencies, and establish a one-to-one or one-to-many message consumption model.
  • the message entry queue module of this embodiment further includes a message confirmation unit: if a message confirmation instruction or an enabled message confirmation mechanism is received, the multi-cloud distributed message service receives an instruction that the message fed back by the message consumer is normally consumed, and the multi-cloud distributed message service The message service feeds back the received message and feeds back to the message producer that the message has been received normally.
  • the message entry queue module of this embodiment further includes a message persistence unit: if message persistence is enabled or a message persistence instruction is received when the message queue is created, the multi-cloud distributed message queue service will send each message on the queue. The received information is persisted.
  • Persistence writes the data in the message service memory (which is easy to lose after power failure) to reliable storage that is not lost after power failure, such as disks and databases.
  • Multi-cloud distributed messaging services may be cross-regional or cross-border, and private lines generally refer to private networks built by enterprises and operators. The biggest difference between them and the public network is better security and stability. There is not much difference in the deployment of message service, except that the message producer and messager use the private network address instead of the public network address when establishing a connection with the message service.
  • the private network addresses will be forwarded by the internal gateway route of the enterprise on the local area network or private line network.
  • the high-performance computing multi-cloud-oriented distributed messaging system in this embodiment further includes: an interconnection module: setting up message service intermediate stations in different regions, deploying message service intermediate stations in each region, and interconnecting message service intermediate stations, Message producers communicate with consumers in other regions by directly accessing the message service intermediate station in their region.
  • the multi-cloud-oriented distributed messaging system for high-performance computing in this embodiment further includes: a nearby access module: deploying a message queue cluster in each public cloud, the message queue service is accessed nearby, and the message queue cluster is deployed in different clouds to access and use The same API interface.
  • the message model of this embodiment includes one or more of a point-to-point model, a publish-subscribe model, a remote calling model, a broadcast model, and a work queue model.
  • the message producer can be a producer, a sender, a client, etc. according to different message models, as long as the function of the message producer of the present invention is implemented.
  • the message consumers can be consumers, receivers, servers, etc. according to different message models or different usage scenarios, as long as the functions of the message consumers of the present invention are implemented.
  • the message model is a point-to-point model: the producer produces the message and sends it to the message queue, and the consumer consumes the message from the message queue.
  • the message model is a publish-subscribe model, receive subscription instructions, perform multi-queue delivery according to the number of subscribed message consumers, subscribe topic messages according to different message queue names, and control the sending of messages of the corresponding subscribed topics to message consumers according to the message queue name.
  • consumer A and consumer B subscribe to messages of different topic types through message queues, such as messages with a topic of log type and messages with a topic of monitoring type.
  • message queues such as messages with a topic of log type and messages with a topic of monitoring type.
  • message consumers only subscribe to the message topics they care about, and messages on other topics are ignored.
  • message consumers subscribe to topic messages according to different message queue names. For example, if the queue name is: /log, all messages prefixed with /log are received by consumers. If the queue name is: /log/warning, then Only warning level logs are received by consumers.
  • the multi-cloud distributed message queue service will generate a unique ID for the message producer.
  • the message will be consumed according to the associated ID.
  • the producer replies to the message and sends it to the correct message producer.
  • the difference between the remote call model and other message models is that it is a response model.
  • the remote call model is suitable for scenarios where the message loss rate is extremely low and the message delivery reliability is extremely high. Blocking waiting, waiting for the server to return a signal after consuming the message to tell the sender that the message has been delivered safely.
  • the message queue under the broadcast model supports the same message to be consumed multiple times
  • the sender sends the message to the broadcast message queue, and both receiver A and receiver B can receive the message, so as to simulate Mechanisms for consuming broadcasts.
  • the consumption model is the work queue model, different messages are taken out from the consumption queue one by one and assigned to the message consumers, and the same message will not be consumed by different consumers twice.
  • the implementation mechanism of the work queue model is similar to that of the broadcast model, the difference is that the same message cannot be received by different consumers, and different messagers will fetch different messages from the message queue one by one.
  • a message service intermediate station is deployed nearby in each area, and the message service intermediate stations are interconnected using a public network or a dedicated line. For message producers and consumers, it looks like a direct connection to the same message service.
  • the reliability of transmission mainly depends on distributed message service deployment, and each message service intermediate station implements message flow control, retransmission and other mechanisms.
  • three node servers are applied for in each cloud, and Kubernetes can be selected as the cluster management software. Deploy the RabbitMQ cluster through Kubernetes, and create access entries for the services respectively.
  • the message queue center controller the message queue center controller is deployed in the AWS Cloud center, and the access portals of each RabbitMQ cluster service are configured in the message queue center controller.
  • Name is the unique name of the message queue.
  • Model types include: point-to-point model, publish-subscribe model, remote call model, broadcast model, and work queue model.
  • Durable specifies whether the message queue starts the persistence mechanism.
  • the message header consists of the following:
  • Name is the name of the message queue
  • Auth is the message authentication information
  • Durable is the message persistence
  • Sender is the sender ID
  • Receiver is the receiver ID
  • Time is the message creation time
  • Version is the message version number.
  • Multi-cloud distributed messaging boils down to the following basic operations. Each operation takes a message as a parameter.
  • the multi-cloud distributed messaging approach includes the following steps:
  • Model types include: point-to-point model, publish-subscribe model, remote call model, broadcast model, work queue model).
  • the message producer calls the SendMessage or SendBatchMessages operation to add the message instance to the queue in the distributed message server.
  • ReceiveMessage or ReceiveBatchMessages to get message instances from the specified message queue.
  • the following uses the Golang programming language as an example to illustrate the message transfer process between the message sender and the receiver.
  • the current basic component of the distributed message queue takes the RabbitMQ open source message middleware as an example, but the implementation of the SDK and the message queue center controller in the present invention does not depend on this system.
  • Msg CloudMessage.CreateMessage(Message-Header,Message-Body,Message-Tail)()
  • a message service intermediate station is deployed nearby in each area, and the message service intermediate stations are interconnected by public network or special line. For message producers and consumers, it looks like a direct connection to the same message service. The reliability of transmission mainly depends on the deployment of distributed message services, and each message service intermediate station implements mechanisms such as message flow control and retransmission.
  • the intermediate stations are interconnected by public network or dedicated line. For a message producer to directly access the message service in its region, it can communicate with consumers in other regions.
  • the present invention realizes the nearby access of message queue services by deploying message queue clusters in major public cloud vendors.
  • the message queue cluster access deployed in different cloud vendors uses the same API interface, which reduces the difficulty of service integration, and developers do not need to integrate according to different cloud vendors.
  • the multi-cloud distributed message queue service is naturally distributed globally and has strong service high availability.
  • the present invention solves that the message service provided by the existing public cloud cannot satisfy the multiple message models simultaneously integrated by the present invention, and the API interfaces used by various clouds are inconsistent, which is not friendly to multi-cloud integration development. And the message service provided by a single public cloud is more inclined to serve services in a single cloud, and the reliability of cross-cloud and cross-region communication delays can be guaranteed.
  • the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
  • computer-usable storage media including, but not limited to, disk storage, CD-ROM, optical storage, etc.
  • These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions
  • the apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

一种面向高性能计算多云的分布式消息传递方法及***包括：根据使用场景，创建消息模型，获取待传递信息，根据待传递信息及消息模型依据消息模板格式的要求填充信息，生成消息实例，消息模板包括消息头、消息载体、消息尾；消息生产者通过多云分布式消息队列服务提供的SDK创建到消息服务的网络连接，将生成的消息实例通过该连接推送到多云分布式消息队列服务中的指定消息队列中；多云分布式消息队列服务对推送过来的消息进行解释，根据消息头信息进行消息路由，将相关的消息推送给不同消息模型上的消息消费者；上述方法及***，消息队列面向多云的高性能计算的消息传输格式定义、多消息模型的支持，使云上的高性能计算开发集成更加高效便捷。

Description

面向高性能计算多云的分布式消息传递方法及*** 技术领域
本发明涉及通信领域，特别涉及一种面向高性能计算多云的分布式消息传递方法及***。
背景技术
高性能计算(High Performance Computing,简称HPC)，是指利用聚集起来的计算能力来处理标准工作站无法完成的数据密集型计算任务，包括建模、仿真和渲染等。它在现代科学研究、工业生产中发挥着越来越重要的作用。
随着云计算的普及，很多传统的服务也搬上了云，其中就包括HPC。云上的高性能计算相比传统的超算来说被赋予了更多的灵活性，如按需使用的计算集群、快速弹性扩缩容的计算资源、跨多区域或多云厂商构建的超大集群、更多样性的异构计算资源等。
多云的高性能计算平台提供了强大的可快速扩缩容的计算集群，多云计算集群资源突破了地域的限制，使计算资源可跨地域实现全球的分布，而运行在这些计算资源上的计算任务也因此具备了分布式的特性。这一特性跟传统的高性能计算同地域、同网络构建的集群是很不一样的。也正因此，为提高企业开发者的编程开发效率，降低企业开发者多云高性能的编程难度，丰富的云中间件服务是必不可少的。其中多云的分布式消息传递方法是云计算环境中任务与任务之间、任务与服务之间的信息专递桥梁。
发明内容
基于此，有必要提供一种可提高通信便捷性的面向高性能计算多云的分布式消息传递方法。
同时，提供一种可提高通信便捷性的面向高性能计算多云的分布式消息传递***。
一种面向高性能计算多云的分布式消息传递方法，包括：
生产消息：根据使用场景，创建消息模型，获取待传递信息，根据待传递信息及消息模型依据消息模板格式的要求填充相应的信息，生成消息实例，消 息模板包括：消息头、消息载体、消息尾；
消息进入队列：消息生产者通过多云分布式消息队列服务提供的SDK创建到消息服务的网络连接，将生成的消息实例通过该连接推送到多云分布式消息队列服务中的指定消息队列中；
消息解释路由：多云分布式消息队列服务对推送过来的消息进行解释，并根据消息头中的相关字段信息进行消息路由，将相关的消息推送给不同消息模型上的消息消费者。
在优选实施例中，所述消息头包括：消息队列名称、消息认证信息、消息是否持久化、消息发送方、消息接收方、消息类型、创建时间、消息版本号；根据消息头信息不同区分不同消息模型；
所述消息解释路由包括:多云分布式消息队列服务对推送过来的消息进行拆解，得到该消息的消息头信息，根据消息头中的消息队列名称搜索当前已经创建的消息队列，找到当前队列，比对消息头的消费类型是否与所在消息队列属性一致，根据消息头的消息认证信息确认当前消息生产者或消息消费者是否有权限对该消息队列进行操作，根据消息模型确定一对一或一对多的相关性，建立一对一或一对多的消息消费模型。
在优选实施例中，所述消息进入队列还包括消息确认：若接收到消息确认指令或启用的消息确认机制，多云分布式消息服务接收到消息消费者反馈的消息被正常消费指令，多云分布式消息服务对接收到的消息进行反馈，反馈给消息生产者该消息已被正常接收。
在优选实施例中，所述消息进入队列还包括消息持久化：若消息队列创建时启用了消息持久化或接收到消息持久化指令，则多云分布式消息队列服务对该队列上的每一条收到的信息进行数据持久化。
在优选实施例中，所述消息模型包括：点对点模型、发布订阅模型、远程调用模型、广播模型、工作队列模型的一种或多种；
所述消息进入队列步骤还包括：根据不同消息模型进行操作；
若消息模型为远程调用模型，则多云分布式消息队列服务会为消息生产者生成一个唯一的ID，在多个关联的消息生产者同时跟消息消费者进行消息交换 时，根据关联ID将消息消费者回复消息投送到正确的消息生产者；
若消息模型为发布订阅模型，接收订阅指令，根据订阅的消息消费者数量进行多队列投放，根据不同消息队列名称来订阅主题消息，根据消息队列名称控制将相应订阅主题的消息发送给消息消费者；
若消费模型为工作列队模型，则在消费列队中逐条取出不同的消息分配给消息消费者，同一条消息不被不同的消费者二次消费。
在优选实施例中，还包括互联：分区域设置消息服务中间站，在每个区域部署消息服务中间站，消息服务中间站之间互联，消息生产者通过直接访问所处区域的消息服务中间站与其他区域的消费者通信。
在优选实施例中，还包括就近访问：在各公有云中部署消息队列集群，消息队列服务就近访问，在不同云中部署消息队列集群访问使用相同的API接口；
消息尾包括：消息摘要算法、摘要值、校验值；
消息队列属性包括：队列名称、队列持久化的一种或多种。
一种面向高性能计算多云的分布式消息传递***，包括：
节点服务器：在每个云中申请多台节点服务器，部署集群，并分别创建集群服务的访问入口；
消息队列中心控制器：部署消息队列中心控制器，将各个集群服务的访问入口配置到消息队列中心控制器；
所述消息队列中心控制器包括：
生产消息模块：根据使用场景，创建消息模型，获取待传递信息，根据待传递信息及消息模型依据消息模板格式的要求填充相应的信息，生成消息实例，消息模板包括：消息头、消息载体、消息尾；
消息进入队列模块：消息生产者通过多云分布式消息队列服务提供的SDK创建到消息服务的网络连接，将生成的消息实例通过该连接推送到多云分布式消息队列服务中的指定消息队列中；
消息解释路由模块：多云分布式消息队列服务对推送过来的消息进行解释，并根据消息头中的相关字段信息进行消息路由，将相关的消息推送给不同消息模型上的消息消费者。
在优选实施例中，所述消息头包括：消息队列名称、消息认证信息、消息是否持久化、消息发送方、消息接收方、消息类型、创建时间、消息版本号；根据消息头信息不同区分不同消息模型；
所述消息解释路由模块还包括:多云分布式消息队列服务对推送过来的消息进行拆解，得到该消息的消息头信息，根据消息头中的消息队列名称搜索当前已经创建的消息队列，找到当前队列，比对消息头的消费类型是否与所在消息队列属性一致，根据消息头的消息认证信息确认当前消息生产者或消息消费者是否有权限对该消息队列进行操作，根据消息模型确定一对一或一对多的相关性，建立一对一或一对多的消息消费模型；
所述消息进入队列模块还包括消息确认单元：若接收到消息确认指令或启用的消息确认机制，多云分布式消息服务接收到消息消费者反馈的消息被正常消费指令，多云分布式消息服务对接收到的消息进行反馈，反馈给消息生产者该消息已被正常接收；
及消息持久化单元：若消息队列创建时启用了消息持久化或接收到消息持久化指令，则多云分布式消息队列服务对该队列上的每一条收到的信息进行数据持久化。
在优选实施例中，还包括：互联模块：分区域设置消息服务中间站，在每个区域部署消息服务中间站，消息服务中间站之间互联，消息生产者通过直接访问所处区域的消息服务中间站与其他区域的消费者通信；
及就近访问模块：在各公有云中部署消息队列集群，消息队列服务就近访问，在不同云中部署消息队列集群访问使用相同的API接口。
上述面向高性能计算多云的分布式消息传递方法及***，消息队列面向多云的高性能计算的消息传输格式定义、多消息模型的支持，使云上的高性能计算开发集成更加高效便捷，构建在多云之上的分布式消息服务，运行在计算资源之上在高性能执行程序可实现跨地域的消息传递而无需关心程序当前所在的真实网络位置。分布式消息服务屏蔽掉了跨地域及异构网络的复杂性，开发者只需要通过统一的分布式消息服务访问接口，专注于计算工作流及上层业务的实现。
本发明通过在各大公有云厂商中部署消息队列集群实现消息队列服务的就近访问。不同云厂商中部署的消息队列集群访问使用的是相同的API接口，降低了服务集成的难度，开发者无需根据不同云厂商进行集成。多云分布式消息队列服务天然的全球分布式部署，具备极强的服务高可用性。
附图说明
图1为本发明一实施例的面向高性能计算多云的分布式消息传递方法的部分流程图；
图2为本发明一实施例的消息模型的示意图。
具体实施方式
本发明通过构建在多云之上的分布式消息服务，运行在计算资源之上在高性能执行程序可实现跨地域的消息传递而无需关心程序当前所在的真实网络位置。分布式消息服务屏蔽掉了跨地域及异构网络的复杂性，开发者只需要通过统一的分布式消息服务访问接口，专注于计算工作流及上层业务的实现。
多云环境下的分布式消息传递方法的设计需要考虑如下因素：
1、跨地域的真实网络环境下的传输时延及稳定性；
2、分布式消息传输的协议以及其格式定义，格式的定义应该简洁又要满足计算任务运行的需要。
3、分布式消息多模型的支持，消息模型的支持依赖于当前多云计算环境下的使用需求。通常消息队列是一个存放消息的容器，当需要消息的时候就从消息队列只取出消息使用。消息队列是分布式***中重要的组件，面向多云的高性能计算的消息传输格式定义、多消息模型的支持使云上的高性能计算开发集成更加高效便捷。
本发明提供一种适用于多云高性能计算场景的分布多消息传递方法，建立多云高性能计算平台中任务以及服务***之间的信息数据传递机制，屏蔽掉跨地域网络通信的复杂性，为面向多云的高性能计算环境之上构建通信模型提供参考实现。
如图1所示，本发明一实施例的面向高性能计算多云的分布式消息传递方法，包括：
步骤S101，生产消息：根据使用场景，创建消息模型，获取待传递信息，根据待传递信息及消息模型依据消息模板格式的要求填充相应的信息，生成消息实例，消息模板包括：消息头、消息载体、消息尾；
步骤S103，消息进入队列：消息生产者通过多云分布式消息队列服务提供 的SDK创建到消息服务的网络连接，将生成的消息实例通过该连接推送到多云分布式消息队列服务中的指定消息队列中；
步骤S105，消息解释路由：多云分布式消息队列服务对推送过来的消息进行解释，并根据消息头中的相关字段信息进行消息路由，将相关的消息推送给不同消息模型上的消息消费者。
多云分布式消息队列服务它接受并转发消息。可以将其视为邮局：将要发布的邮件放在邮箱中时，可以确保最终将邮件传递给收件人。以此类推，分布式消息队列服务是一个邮政信箱，一个邮局和一个邮递员。分布式消息队列服务与邮局之间的主要区别在于，它不处理纸张，而是接收，存储和转发二进制数据消息。
如图2所示，本实施例的消息模型包括：点对点模型、发布订阅模型、远程调用模型、广播模型、工作队列模型的一种或多种。
消息生产者可以根据消息模型不同，为生产者、发送者、Client等，只要实现本发明的消息生产者的功能即可。
消息消费者可以根据消息模型不同或使用场景不同为消费者、接收者、Server等，只要实现本发明的消息消费者的功能即可。
若消息模型为点对点模型：生产者生产消息，并发送到消息队列中，而消费者则从消息队列中消费消息。
若消息模型为发布订阅模型，接收订阅指令，根据订阅的消息消费者数量进行多队列投放，根据不同消息队列名称来订阅主题消息，根据消息队列名称控制将相应订阅主题的消息发送给消息消费者。
在发布订阅模型中，消费者A跟消费者B通过消息队列分别订阅了不同主题类型的消息，如主题为日志类型的消息、主题为监控类型的消息。通过发布订阅模型，消息消费者只订阅其关心的消息主题，对于其它主题消息则会被忽略。
若消息模型为远程调用模型，则多云分布式消息队列服务会为消息生产者生成一个唯一的ID，在多个关联的消息生产者同时跟消息消费者进行消息交换时，根据关联ID将消息消费者回复消息投送到正确的消息生产者。
远程调用模型与其它消息模型不同之处在于它是一种应答模型，远程调用模型适用于消息丢失率容忍性极低、消息传递可靠性极高的场景，因此消息发送端发送了消息之后还需要阻塞等待，等待服务端在消费完消息后回传一个信号告诉发送端，该消息已安全传达。
若消息模型为广播模型，广播模型下的消息队列支持同样的一条消息被多 次消费，发送者把消息发送到广播消息队列，接收者A以及接收者B都可以收到该消息，以此模拟消费广播的机制。
若消费模型为工作列队模型，则在消费列队中逐条取出不同的消息分配给消息消费者，同一条消息不被不同的消费者二次消费。
工作队列模型与广播模型实现机制类似，所不同的在于同一条消息无法被不同的消费者所接收，不同的消息者将会从消息队列中逐条取出不同的消息。
消息生产者通过多云分布式消息服务提供的SDK(Software Development Kit)创建到消息服务的网络连接可以在生产消息步骤中或步骤前进行，也可在消息进入队列步骤中进行，只要在将生成的消息实例通过该连接推送到多云分布式消息队列服务中的指定消息队列中前即可。
进一步，生产消息，消息生产者通过多云分布式消息队列服务提供的SDK(Software Development Kit)创建到消息服务的网络连接，根据消息模板格式的要求填充相应的字段，生成消息实例。
进一步，消息进入队列，消息生产者通过多云分布式消息队列服务提供的SDK创建到消息服务的网络连接后，将生成的消息实例通过该连接推送到消息队列服务中的指定消息队列中，根据不同消息模型进行操作。
进一步，消息解释路由，多云分布式消息服务对推送过来的消息进行解释，并根据消息头中的相关字段信息进行消息路由，将相关的消息推送给不同消息模型上的消息消费者。
消息队列属性包括：队列名称、队列持久化的一种或多种。
由于多云分布式消息队列服务作为消息的中间存放者，需要对消息的推送接收以及转发到消费者进行消息确认，以确保消息的传达。
消息进入队列还包括消息确认：若接收到消息确认指令或启用的消息确认机制，多云分布式消息服务接收到消息消费者反馈的消息被正常消费指令，多云分布式消息服务对接收到的消息进行反馈，反馈给消息生产者该消息已被正常接收。
消息进入队列还包括消息持久化：若消息队列创建时启用了消息持久化或接收到消息持久化指令，则多云分布式消息队列服务对该队列上的每一条收到的信息进行数据持久化。
在分布式环境中由于跨地域网络的特性，网络的中断，服务的故障等都有可能造成消息在传输过程中的丢失，因此对于可靠性要求较高的业务来说持久化是很有必要的，但消息持久化由于涉及到第三方存储或者本地硬盘存储的I/O(Input/Output)操作，因此也必定会一定的传递效率下降，因此消息持久化特 性在消息模型中是一个可选项。当消息队列创建时启用了持久化选项，则多云分布式消息服务会指对该队列上的每一条收到的消息进行数据持久化，以防止服务中断导致存储计算机内存中的消息丢失。通过持久化机制，多云分布式消息服务在中断重启过程中可以从持久化的数据服务中恢复消息队列中原有的消息数据。
在跨国消息传递的过程中默认使用的是公网传输，如企业已经构建了跨国的专线通道，分布式消息队列服务也可以通过专线进行消息传递。专线消息的传递稳定性、私密性会更好，但也需要注意专线的带宽是否充足。
消息头，消息模型是使用消息头进行区分的。消息头包含的字段有：消息队列名称、消息认证信息、消息是否持久化、消息发送方、消息接收方、消息类型、创建时间、消息版本号。
消息载体，消息载体存储着消息的实际数据内容。
消息尾，主要是确保消息传输过程中数据的完整性，它包含的字段有：消息摘要算法、摘要值、校验值。
消息解释路由包括:多云分布式消息队列服务对推送过来的消息进行拆解，得到该消息的消息头信息，根据消息头中的消息队列名称搜索当前已经创建的消息队列，找到当前队列，比对消息头的消费类型是否与所在消息队列属性一致，根据消息头的消息认证信息确认当前消息生产者或消息消费者是否有权限对该消息队列进行操作，根据消息模型确定一对一或一对多的相关性，建立一对一或一对多的消息消费模型。
进一步，本实施例的面向高性能计算多云的分布式消息传递方法，还包括互联：分区域设置消息服务中间站，在每个区域部署消息服务中间站，消息服务中间站之间互联，消息生产者通过直接访问所处区域的消息服务中间站与其他区域的消费者通信。
进一步，本实施例的面向高性能计算多云的分布式消息传递方法，还包括就近访问：在各公有云中部署消息队列集群，消息队列服务就近访问，在不同云中部署消息队列集群访问使用相同的API接口。
本发明通过在各大公有云厂商中部署消息队列集群实现消息队列服务的就近访问。不同云厂商中部署的消息队列集群访问使用的是相同的API接口，降低了服务集成的难度，开发者无需根据不同云厂商进行集成。多云分布式消息队列服务天然的全球分布式部署，具备极强的服务高可用性。
本发明的一种面向高性能计算多云的分布式消息传递***，包括：
节点服务器：在每个云中申请多台节点服务器，部署集群，并分别创建集 群服务的访问入口；
消息队列中心控制器：部署消息队列中心控制器，将各个集群服务的访问入口配置到消息队列中心控制器。
本发明的消息队列中心控制器包括：
生产消息模块：根据使用场景，创建消息模型，获取待传递信息，根据待传递信息及消息模型依据消息模板格式的要求填充相应的信息，生成消息实例，消息模板包括：消息头、消息载体、消息尾；
消息进入队列模块：消息生产者通过多云分布式消息队列服务提供的SDK创建到消息服务的网络连接，将生成的消息实例通过该连接推送到多云分布式消息队列服务中的指定消息队列中；
消息解释路由模块：多云分布式消息队列服务对推送过来的消息进行解释，并根据消息头中的相关字段信息进行消息路由，将相关的消息推送给不同消息模型上的消息消费者。
高性能计算(High Performance Computing,简称HPC)：是指利用聚集起来的计算能力来处理标准工作站无法完成的数据密集型计算任务，包括建模、仿真和渲染等。
云上的高性能计算：是指把传统的高性能计算集群搬到云上，赋予其云上按需扩展收费、多样性异构计算资源、可跨全球构建集群的特性。
多云：是指将多个公有云混合集成，以扩大计算资源的规模。
分布式***：分布式***是多个处理机通过通信线路互联而构成的松散耦合的***。从***中某台处理机来看，其余的处理机和相应的资源都是远程的，只有它自己的资源才是本地的。
消息队列：“消息”是在两台计算机间传送的数据单位。消息可以非常简单，例如只包含文本字符串；也可以更复杂，可能包含嵌入对象。消息被发送到队列中。“消息队列”是在消息的传输过程中保存消息的容器。消息队列管理器在将消息从它的源中继到它的目标时充当中间人。队列的主要目的是提供路由并保证消息的传递；如果发送消息时接收者不可用，消息队列会保留消息，直到可以成功地传递它。
Leader Node(领导节点)：单个节点部署的消息队列具有单点失效的风险，也就是说单个节点上部署的消息队列服务如果出现故障，那么整个消息队列服务将不可用，直接影响消息的生产者与消息者的消息收发。而多个节点部署的消息队列则在很大程序上解决了单点失效的问题，为了维护多个节点下的消息传递管理因此需要选举出一个领导节点，生产者及消息者与领导节点进行通信。
Kubernetes:简称K8s，是用8代替8个字符“ubernete”而成的缩写。是一个开源的，用于管理云平台中多个主机上的容器化的应用，Kubernetes的目标是让部署容器化的应用简单并且高效(powerful),Kubernetes提供了应用部署，规划，更新，维护的一种机制。
Rabbitmq:RabbitMQ是实现了高级消息队列协议(AMQP)的开源消息代理软件(亦称面向消息的中间件)。
本发明的***包括：多云环境下的分布式部署设计、消息模型的设计、消息模板格式的设计、多云分布式消息传递过程、消息的持久化。
本发明面向高性能计算多云分布式消息队列服务领域。分布多消息队列服务的设计目标是满足多云构建而成的高性能计算环境中任务及服务的数据交互需求，解决因多云环境而出现的跨地域复杂网络。提升消息传延的可靠性，降低跨地域通信编程的复杂性。
进一步，本实施例的消息头包括：消息队列名称、消息认证信息、消息是否持久化、消息发送方、消息接收方、消息类型、创建时间、消息版本号。根据消息头信息不同区分不同消息模型。
进一步，本实施例的消息解释路由模块还包括:多云分布式消息队列服务对推送过来的消息进行拆解，得到该消息的消息头信息，根据消息头中的消息队列名称搜索当前已经创建的消息队列，找到当前队列，比对消息头的消费类型是否与所在消息队列属性一致，根据消息头的消息认证信息确认当前消息生产者或消息消费者是否有权限对该消息队列进行操作，根据消息模型确定一对一或一对多的相关性，建立一对一或一对多的消息消费模型。
进一步，本实施例的消息进入队列模块还包括消息确认单元：若接收到消息确认指令或启用的消息确认机制，多云分布式消息服务接收到消息消费者反馈的消息被正常消费指令，多云分布式消息服务对接收到的消息进行反馈，反馈给消息生产者该消息已被正常接收。
进一步，本实施例的消息进入队列模块还包括消息持久化单元：若消息队列创建时启用了消息持久化或接收到消息持久化指令，则多云分布式消息队列服务对该队列上的每一条收到的信息进行数据持久化。
持久化将消息服务内存(断电易丢失)中的数据写入到可靠的断电不丢失的存储中，如确盘、数据库等。
多云的分布式消息服务存在着跨地区或者跨国的可能，而专线一般是指企业跟运营商构建的专用网络，它跟公网的最大区别是安全性、稳定性更好。对于消息服务部署来说区别不大，只是消息的生产者与消息者在跟消息服务建立 连接时使用的是私有网络地址而不是公网地址。私有网络地址均会被企业内部的网关路由进行局域网或专线网转发。
进一步，本实施例的面向高性能计算多云的分布式消息传递***，还包括：互联模块：分区域设置消息服务中间站，在每个区域部署消息服务中间站，消息服务中间站之间互联，消息生产者通过直接访问所处区域的消息服务中间站与其他区域的消费者通信。
进一步，本实施例的面向高性能计算多云的分布式消息传递***还包括：就近访问模块：在各公有云中部署消息队列集群，消息队列服务就近访问，在不同云中部署消息队列集群访问使用相同的API接口。
如图2所示，本实施例的消息模型包括：点对点模型、发布订阅模型、远程调用模型、广播模型、工作队列模型的一种或多种。
消息生产者可以根据消息模型不同，为生产者、发送者、Client等，只要实现本发明的消息生产者的功能即可。
消息消费者可以根据消息模型不同或使用场景不同为消费者、接收者、Server等，只要实现本发明的消息消费者的功能即可。
若消息模型为点对点模型：生产者生产消息，并发送到消息队列中，而消费者则从消息队列中消费消息。
若消息模型为发布订阅模型，接收订阅指令，根据订阅的消息消费者数量进行多队列投放，根据不同消息队列名称来订阅主题消息，根据消息队列名称控制将相应订阅主题的消息发送给消息消费者。
在发布订阅模型中，消费者A跟消费者B通过消息队列分别订阅了不同主题类型的消息，如主题为日志类型的消息、主题为监控类型的消息。通过发布订阅模型，消息消费者只订阅其关心的消息主题，对于其它主题消息则会被忽略。发布订阅模型中消息消费者根据不同的消息队列名称来订阅主题消息，如队列名称为:/log则所有以/log前缀的消息均被消费者接收，若队列名称为:/log/warning,则只有warning(警告)级别的日志被消费者接收。
若消息模型为远程调用模型，则多云分布式消息队列服务会为消息生产者生成一个唯一的ID，在多个关联的消息生产者同时跟消息消费者进行消息交换时，根据关联ID将消息消费者回复消息投送到正确的消息生产者。
远程调用模型与其它消息模型不同之处在于它是一种应答模型，远程调用模型适用于消息丢失率容忍性极低、消息传递可靠性极高的场景，因此消息发送端发送了消息之后还需要阻塞等待，等待服务端在消费完消息后回传一个信号告诉发送端，该消息已安全传达。
若消息模型为广播模型，广播模型下的消息队列支持同样的一条消息被多次消费，发送者把消息发送到广播消息队列，接收者A以及接收者B都可以收到该消息，以此模拟消费广播的机制。
若消费模型为工作列队模型，则在消费列队中逐条取出不同的消息分配给消息消费者，同一条消息不被不同的消费者二次消费。
工作队列模型与广播模型实现机制类似，所不同的在于同一条消息无法被不同的消费者所接收，不同的消息者将会从消息队列中逐条取出不同的消息。
本实施例通过在每个区域都就近部署一个消息服务中间站，消息服务中间站之间使用公网或者专线互联。对于消息生产者与消费者来说形似直接连接同一个消息服务。传输的可靠性主要依靠分布式的消息服务部署，同时各个消息服务中间站实现了消息流控，重传等机制。
进一步，具体的，在本实施例中，在每个云中申请3台节点服务器，集群管理软件可以选用Kubernetes。通过Kubernetes部署RabbitMQ集群，并分别创建服务的访问入口。
进一步，本实施例中，消息队列中心控制器：在AWS Cloud中心部署消息队列中心控制器，并将各个RabbitMQ集群服务的访问入口配置到消息队列中心控制器中。
不同消息模型的消息队列：
面向高性能计算的多云分布式消息队列的形式化表示为Queue,消息模型为Model则：
Queue＝Create of{Name,Model，Durable}
其中Name是消息队列的唯一名称。
Model类型包括：点对点模型、发布订阅模型、远程调用模型、广播模型、工作队列模型。
Durable指定该消息队列是否启动持久化机制。
消息模板格式：
面向高性能计算多云分布式消息形式化表示如下：设Message为消息，则
Message＝consist of{Message-Header,Message-Body,Message-Tail}
消息头 消息载体 消息尾
其中消息头组成如下:
Message-Header＝consist of{Name,Auth,Durable,Sender,Receiver,Time,Version}
其中，Name为消息队列名称，Auth为消息认证信息，Durable标识消息持 久化，Sender为发送方标识，Receiver为接收方标识，Time为消息创建时间,Version为消息版本号。
Figure PCTCN2020135768-appb-000001
多云分布式消息传递
多云分布式消息传递归结为以下几个基本操作。各个操作均以消息为参数。
CreateQueue(Name,Model,Durable)；消息队列创建操作。
CreateMessage(Message-Header,Message-Body,Message-Tail)；消息创建操作。
SendMessage(Message)；消息发送到队列操作。
ReceiveMessage(Queue-Name)；从消息队列中获取消息操作。
SendBatchMessages([]Message)；批量发送消息操作。
ReceiveBatchMessages(Queue-Name)；从消息队列中批量获取消息操作。
多云分布式消息传递方法包括以下步骤：
根据不同的使用场景，通过Mode参数调用CreateQueue创建不同的消息队列模型Model类型包括：点对点模型、发布订阅模型、远程调用模型、广播模型、工作队列模型)。
准备好Message-Header,Message-Body,Message-Tail，调用CreateMessage创建消息实例。
消息生产者调用SendMessage或者SendBatchMessages操作，将消息实例添加到分布式消息服务器中的队列。
消息消费者调用ReceiveMessage或者ReceiveBatchMessages从指定的消息队列中获取消息实例。
下面结合一具体实施例进行详细说明：
下面以Golang编程语言为例，说明消息发送者与接收者双方之间的消息传递过程。当前的分布式消息队列基础组件以RabbitMQ开源消息中间件为例，但本发明中的SDK及消息队列中心控制器实现并不依赖于该***。
消息头的定义
字段名称 类型 长度(Byte) 字段说明
消息队列各称 String <＝128B 消息队列的唯一标识
消息认证信息 String <＝512B Authorization:Basic
持久化 Bool 4B 是否消息持久标识
消息发送方 String 32B IP+端口号
消息接收方 String 32B IP+端口号
消息类型 String 8B  
创建时间 String 128B 格式:YYYY/MM/DD-hhmmss
消息版本号 String 8B  
消息体的定义
Figure PCTCN2020135768-appb-000002
消息类定义
type CloudMessage struct{
messageHeader MessageHeader
messageBody MessageBody
messageTail MessageTail
}
func(msg*CloudMessage)CreateQueue(Name,Model,Durable){}
func(msg*CloudMessage)CreateMessage(Message-Header,Message-Body,Message-Tail){}
func(msg*CloudMessage)SendMessage(Message){}
func(msg*CloudMessage)ReceiveMessage(Queue-Name){}
func(msg*CloudMessage)SendBatchMessages([]Message){}
func(msg*CloudMessage)ReceiveBatchMessages(Queue-Name){}
消息传递步骤：
创建指定消息模型的消息队列，以点对点模型为例
Queue＝CloudMessage.CreateQueue(“hello”,”点对点”,true)
根据消息模板定义生成消息实例
Msg＝CloudMessage.CreateMessage(Message-Header,Message-Body,Message-Tail)()
发送消息
CloudMessage.SendMessage(Msg)
接收消息
CloudMessage.ReceiveMessage(Queue.Name)
本发明通过在每个区域都就近部署一个消息服务中间站，消息服务中间站之间使用公网或者专线互联。对于消息生产者与消费者来说形似直接连接同一个消息服务。传输的可靠性主要依靠分布式的消息服务部署，同时各个消息服务中间站实现了消息流控，重传等机制。通过在每个区域都就近部署一个消息服务中间站，中间站之间使用公网或者专线互联。对于消息生产者来说直接访问它所处区域的消息服务便可实现与其它区域的消费者通信。
本发明通过在各大公有云厂商中部署消息队列集群实现消息队列服务的就近访问。不同云厂商中部署的消息队列集群访问使用的是相同的API接口，降低了服务集成的难度，开发者无需根据不同云厂商进行集成。多云分布式消息队列服务天然的全球分布式部署，具备极强的服务高可用性。
本发明解决现有公有云所提供的消息服务无法同时满足本发明所同时集成的多种消息模型，并且各种云使用的API接口不一致，对于多云集成开发不友好。并且单公有云所提供的消息服务更倾向于服务于单云中的服务，对于跨云跨区域的通信延时，可靠性很能保证。
以上述依据本申请的理想实施例为启示，通过上述的说明内容，相关工作人员完全可以在不偏离本项申请技术思想的范围内，进行多样的变更以及修改。本项申请的技术性范围并不局限于说明书上的内容，必须要根据权利要求范围来确定其技术性范围。
本领域内的技术人员应明白，本申请的实施例可提供为方法、***、或计算机程序产品。因此，本申请可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本申请可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。
本申请是参照根据本申请实施例的方法、设备(***)、和计算机程序产品的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的装置。
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备 以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。

Claims (10)

  1. 一种面向高性能计算多云的分布式消息传递方法，其特征在于，包括：
    生产消息：根据使用场景，创建消息模型，获取待传递信息，根据待传递信息及消息模型依据消息模板格式的要求填充相应的信息，生成消息实例，消息模板包括：消息头、消息载体、消息尾；
    消息进入队列：消息生产者通过多云分布式消息队列服务提供的SDK创建到消息服务的网络连接，将生成的消息实例通过该连接推送到多云分布式消息队列服务中的指定消息队列中；
    消息解释路由：多云分布式消息队列服务对推送过来的消息进行解释，并根据消息头中的相关字段信息进行消息路由，将相关的消息推送给不同消息模型上的消息消费者。
  2. 根据权利要求1所述的面向高性能计算多云的分布式消息传递方法，其特征在于，所述消息头包括：消息队列名称、消息认证信息、消息是否持久化、消息发送方、消息接收方、消息类型、创建时间、消息版本号；根据消息头信息不同区分不同消息模型；
    所述消息解释路由包括:多云分布式消息队列服务对推送过来的消息进行拆解，得到该消息的消息头信息，根据消息头中的消息队列名称搜索当前已经创建的消息队列，找到当前队列，比对消息头的消费类型是否与所在消息队列属性一致，根据消息头的消息认证信息确认当前消息生产者或消息消费者是否有权限对该消息队列进行操作，根据消息模型确定一对一或一对多的相关性，建立一对一或一对多的消息消费模型。
  3. 根据权利要求1所述的面向高性能计算多云的分布式消息传递方法，其特征在于，所述消息进入队列还包括消息确认：若接收到消息确认指令或启用的消息确认机制，多云分布式消息服务接收到消息消费者反馈的消息被正常消费指令，多云分布式消息服务对接收到的消息进行反馈，反馈给消息生产者该消息已被正常接收。
  4. 根据权利要求1所述的面向高性能计算多云的分布式消息传递方法，其特征在于，所述消息进入队列还包括消息持久化：若消息队列创建时启用了消息持久化或接收到消息持久化指令，则多云分布式消息队列服务对该队列上的每一条收到的信息进行数据持久化。
  5. 根据权利要求1至4任意一项所述的面向高性能计算多云的分布式消息传递 方法，其特征在于，所述消息模型包括：点对点模型、发布订阅模型、远程调用模型、广播模型、工作队列模型的一种或多种；
    所述消息进入队列步骤还包括：根据不同消息模型进行操作；
    若消息模型为远程调用模型，则多云分布式消息队列服务会为消息生产者生成一个唯一的ID，在多个关联的消息生产者同时跟消息消费者进行消息交换时，根据关联ID将消息消费者回复消息投送到正确的消息生产者；
    若消息模型为发布订阅模型，接收订阅指令，根据订阅的消息消费者数量进行多队列投放，根据不同消息队列名称来订阅主题消息，根据消息队列名称控制将相应订阅主题的消息发送给消息消费者；
    若消费模型为工作列队模型，则在消费列队中逐条取出不同的消息分配给消息消费者，同一条消息不被不同的消费者二次消费。
  6. 根据权利要求1至4任意一项所述的面向高性能计算多云的分布式消息传递方法，其特征在于，还包括互联：分区域设置消息服务中间站，在每个区域部署消息服务中间站，消息服务中间站之间互联，消息生产者通过直接访问所处区域的消息服务中间站与其他区域的消费者通信。
  7. 根据权利要求2所述的面向高性能计算多云的分布式消息传递方法，其特征在于，还包括就近访问：在各公有云中部署消息队列集群，消息队列服务就近访问，在不同云中部署消息队列集群访问使用相同的API接口；
    消息尾包括：消息摘要算法、摘要值、校验值；
    消息队列属性包括：队列名称、队列持久化的一种或多种。
  8. 一种面向高性能计算多云的分布式消息传递***，其特征在于，包括：
    节点服务器：在每个云中申请多台节点服务器，部署集群，并分别创建集群服务的访问入口；
    消息队列中心控制器：部署消息队列中心控制器，将各个集群服务的访问入口配置到消息队列中心控制器；
    所述消息队列中心控制器包括：
    生产消息模块：根据使用场景，创建消息模型，获取待传递信息，根据待传递信息及消息模型依据消息模板格式的要求填充相应的信息，生成消息实 例，消息模板包括：消息头、消息载体、消息尾；
    消息进入队列模块：消息生产者通过多云分布式消息队列服务提供的SDK创建到消息服务的网络连接，将生成的消息实例通过该连接推送到多云分布式消息队列服务中的指定消息队列中；
    消息解释路由模块：多云分布式消息队列服务对推送过来的消息进行解释，并根据消息头中的相关字段信息进行消息路由，将相关的消息推送给不同消息模型上的消息消费者。
  9. 根据权利要求8所述的面向高性能计算多云的分布式消息传递***，其特征在于，所述消息头包括：消息队列名称、消息认证信息、消息是否持久化、消息发送方、消息接收方、消息类型、创建时间、消息版本号；根据消息头信息不同区分不同消息模型；
    所述消息解释路由模块还包括:多云分布式消息队列服务对推送过来的消息进行拆解，得到该消息的消息头信息，根据消息头中的消息队列名称搜索当前已经创建的消息队列，找到当前队列，比对消息头的消费类型是否与所在消息队列属性一致，根据消息头的消息认证信息确认当前消息生产者或消息消费者是否有权限对该消息队列进行操作，根据消息模型确定一对一或一对多的相关性，建立一对一或一对多的消息消费模型；
    所述消息进入队列模块还包括消息确认单元：若接收到消息确认指令或启用的消息确认机制，多云分布式消息服务接收到消息消费者反馈的消息被正常消费指令，多云分布式消息服务对接收到的消息进行反馈，反馈给消息生产者该消息已被正常接收；
    及消息持久化单元：若消息队列创建时启用了消息持久化或接收到消息持久化指令，则多云分布式消息队列服务对该队列上的每一条收到的信息进行数据持久化。
  10. 根据权利要求8或9所述的面向高性能计算多云的分布式消息传递***，其特征在于，还包括：互联模块：分区域设置消息服务中间站，在每个区域部署消息服务中间站，消息服务中间站之间互联，消息生产者通过直接访问所处区域的消息服务中间站与其他区域的消费者通信；
    及就近访问模块：在各公有云中部署消息队列集群，消息队列服务就近访问，在不同云中部署消息队列集群访问使用相同的API接口。
PCT/CN2020/135768 2020-12-11 2020-12-11 面向高性能计算多云的分布式消息传递方法及*** WO2022120806A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/CN2020/135768 WO2022120806A1 (zh) 2020-12-11 2020-12-11 面向高性能计算多云的分布式消息传递方法及***

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2020/135768 WO2022120806A1 (zh) 2020-12-11 2020-12-11 面向高性能计算多云的分布式消息传递方法及***

Publications (1)

Publication Number Publication Date
WO2022120806A1 true WO2022120806A1 (zh) 2022-06-16

Family

ID=81974127

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/135768 WO2022120806A1 (zh) 2020-12-11 2020-12-11 面向高性能计算多云的分布式消息传递方法及***

Country Status (1)

Country Link
WO (1) WO2022120806A1 (zh)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115225630A (zh) * 2022-07-19 2022-10-21 浪潮云信息技术股份公司 一种边缘计算场景下的云边消息通信方法
CN115576714A (zh) * 2022-10-19 2023-01-06 深圳市中兴新云服务有限公司 基于mq框架确保消息队列消费顺序准确性的方法
CN116016655A (zh) * 2022-12-28 2023-04-25 唯品会(广州)软件有限公司 基于RabbitMQ的消息处理方法、***及相关设备
CN116646061A (zh) * 2023-04-28 2023-08-25 西安交通大学 分布式ct成像和智能诊疗***及方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101977165A (zh) * 2010-11-08 2011-02-16 北京中科院软件中心有限公司 云模式下的消息传输方法及消息总线***
CN103685538A (zh) * 2013-12-20 2014-03-26 中电长城网际***应用有限公司 一种分布式网络架构
US20170344754A1 (en) * 2016-05-31 2017-11-30 Genesys Telecommunications Laboratories, Inc. System and Method for Data Management and Task Routing Based on Data Tagging
US20190243690A1 (en) * 2018-02-07 2019-08-08 HT Research Inc. Workgroup Hierarchical Core Structures for Building Real-time Workgroup Systems
CN111432025A (zh) * 2020-04-10 2020-07-17 中国人民解放军国防科技大学 一种面向云边协同的分布式服务目录管理方法及***

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101977165A (zh) * 2010-11-08 2011-02-16 北京中科院软件中心有限公司 云模式下的消息传输方法及消息总线***
CN103685538A (zh) * 2013-12-20 2014-03-26 中电长城网际***应用有限公司 一种分布式网络架构
US20170344754A1 (en) * 2016-05-31 2017-11-30 Genesys Telecommunications Laboratories, Inc. System and Method for Data Management and Task Routing Based on Data Tagging
US20190243690A1 (en) * 2018-02-07 2019-08-08 HT Research Inc. Workgroup Hierarchical Core Structures for Building Real-time Workgroup Systems
CN111432025A (zh) * 2020-04-10 2020-07-17 中国人民解放军国防科技大学 一种面向云边协同的分布式服务目录管理方法及***

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115225630A (zh) * 2022-07-19 2022-10-21 浪潮云信息技术股份公司 一种边缘计算场景下的云边消息通信方法
CN115576714A (zh) * 2022-10-19 2023-01-06 深圳市中兴新云服务有限公司 基于mq框架确保消息队列消费顺序准确性的方法
CN116016655A (zh) * 2022-12-28 2023-04-25 唯品会(广州)软件有限公司 基于RabbitMQ的消息处理方法、***及相关设备
CN116646061A (zh) * 2023-04-28 2023-08-25 西安交通大学 分布式ct成像和智能诊疗***及方法
CN116646061B (zh) * 2023-04-28 2024-01-26 西安交通大学 分布式ct成像和智能诊疗***及方法

Similar Documents

Publication Publication Date Title
WO2022120806A1 (zh) 面向高性能计算多云的分布式消息传递方法及***
US10958723B2 (en) Cloud-end data multicast method and system, and computer device
US8638789B1 (en) Optimal multicast forwarding in OpenFlow based networks
CN112527523A (zh) 面向高性能计算多云的分布式消息传递方法及***
CN111600936B (zh) 一种适用于泛在电力物联网边缘终端的基于多容器的非对称处理***
US10148565B2 (en) OPENFLOW communication method and system, controller, and service gateway
WO2021088641A1 (zh) 数据发送方法、处理方法、接收方法及其设备、存储介质
WO2015024368A1 (zh) 一种面向构件的混合型云操作***体系结构及其通信方法
CN110113406B (zh) 基于分布式的计算服务集群***
CN102546839B (zh) 面向大规模网络的高效、可靠的软件分发方法
CN115379010B (zh) 一种容器网络构建方法、装置、设备及存储介质
CN110635932B (zh) 一种基于OpenStack控制平面的虚拟网络性能的优化方法
CN114710571B (zh) 数据包处理***
WO2023045363A1 (zh) 一种会议消息推送方法、会议服务端及电子设备
CN114024910B (zh) 一种用于金融交易***的极低延时可靠通讯***及方法
CN114244768A (zh) 二层未知组播的转发方法、装置、设备及存储介质
JP2009123202A (ja) データを処理するためのプロセッサ‐サーバ・ハイブリッド・システムおよび方法
US10938591B2 (en) Multicast system
US20160212083A9 (en) Connection sharing across entities in a distributed messaging system
CN115225482A (zh) 一种基于Kubernetes进行Pod容器网络配置的方法及装置
CN107360588B (zh) 一种小基站oam的消息处理方法
CN115296952B (zh) 一种设备调度方法、装置、设备及存储介质
WO2023035777A1 (zh) 网络配置方法、代理组件、控制器、电子设备和存储介质
US10728155B2 (en) Inter-datacenter multicast system
US20110078233A1 (en) Apparatus, system, and method for improved performance of real time applications in intermittent connection environments

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20964742

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 18.10.2023)

122 Ep: pct application non-entry in european phase

Ref document number: 20964742

Country of ref document: EP

Kind code of ref document: A1