CN110688228A - Method, system and electronic equipment for guaranteeing distributed system message time sequence - Google Patents
Method, system and electronic equipment for guaranteeing distributed system message time sequence Download PDFInfo
- Publication number
- CN110688228A CN110688228A CN201910954949.8A CN201910954949A CN110688228A CN 110688228 A CN110688228 A CN 110688228A CN 201910954949 A CN201910954949 A CN 201910954949A CN 110688228 A CN110688228 A CN 110688228A
- Authority
- CN
- China
- Prior art keywords
- message
- generated
- enabling
- service processing
- thread
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 33
- 239000000872 buffer Substances 0.000 claims description 34
- 238000004519 manufacturing process Methods 0.000 abstract description 16
- 238000013467 fragmentation Methods 0.000 abstract description 10
- 238000006062 fragmentation reaction Methods 0.000 abstract description 10
- 230000009286 beneficial effect Effects 0.000 description 9
- 239000000463 material Substances 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/52—Program synchronisation; Mutual exclusion, e.g. by means of semaphores
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention relates to a method, a system and electronic equipment for guaranteeing message time sequence of a distributed system, wherein M message producers are preset, each message producer respectively obtains a message list to be generated from a message database, namely, the message database can be processed in a fragmentation way to generate M corresponding message lists to be generated, the processing capacity of the messages is expanded, and the M message producers sequentially write the messages into M corresponding message themes, further expanding the processing capacity of the messages and effectively avoiding the occurrence of single-point faults, because the messages in the message list to be generated are sequentially ordered according to time and are sequentially written into the corresponding message themes, the ordered production and consumption of the message time sequence are ensured, and the M message producers preset in the message middleware are of the same type, namely only one type of message middleware is needed without depending on multiple types of message middleware, the applicability is strong.
Description
Technical Field
The present invention relates to the field of communications technologies, and in particular, to a method, a system, and an electronic device for guaranteeing a distributed system message timing.
Background
When the architecture of the related online service acceptance system of an operator is designed, in order to improve the flexible expansibility of the system, the system deployment architecture generally considers the deployment in a distributed or clustered manner, i.e., a distributed system or a clustered system, and meanwhile, the integration between the systems is generally realized in an asynchronous message manner based on the purposes of system decoupling, flow peak clipping and the like.
Taking telecommunication operation business acceptance as an example, in the message synchronization process, it is important to consider that the customer creation message, the order message and the payment message can be performed according to a strict time sequence, otherwise, system abnormality occurs, such as the absence of the customer who falls on the floor of the order message, the absence of the order when the payment message falls on the floor or the absence of the customer, and in order to solve the above problems, the following method is usually adopted in a distributed system of the system at present:
1) in the case of small traffic, one message subject, one message producer, and one message consumer can be used to guarantee the time-series production and writing. However, the scheme has the problems of weak processing capability, single point of failure and the like.
2) When the traffic is large, the local time sequence characteristic of part of message middleware is adopted to ensure that the local messages in the message theme can be orderly produced and consumed, but the message theme depends on various types of message middleware, so that the message theme is inconvenient to popularize and use in a large area and has poor applicability.
Therefore, how to enhance the message processing capability and ensure ordered message sequence production and consumption, no single point failure and strong applicability is a technical problem to be solved urgently in the industry.
Disclosure of Invention
The invention provides a method, a system and electronic equipment for guaranteeing a distributed system message time sequence, aiming at the defects of the prior art.
The invention discloses a method for guaranteeing the message time sequence of a distributed system, which adopts the technical scheme as follows:
s1, presetting M message producers, and enabling each message producer to respectively obtain a to-be-generated message list in time sequence from a message database, wherein M is a positive integer and is more than or equal to 2;
s2, presetting M message themes in the message middleware, and enabling each message producer to write the messages in the message list to be generated into the corresponding message themes in sequence;
and S3, enabling each message theme to respectively send the received message to each corresponding message consumer, and enabling each message consumer to call the corresponding application service according to the received message.
The method for guaranteeing the message time sequence of the distributed system has the beneficial effects that: the method comprises the steps that M message producers are preset, each message producer respectively obtains a message list to be generated from a message database, namely the message database can be subjected to fragmentation processing to generate M corresponding message lists to be generated, the processing capacity of the messages is expanded, the M message producers sequentially write the messages into M corresponding message themes, the processing capacity of the messages is further expanded, and single-point faults are effectively avoided. Therefore, the method for guaranteeing the message time sequence of the distributed system can enhance the message processing capacity, ensure the ordered production and consumption of the message time sequence, has no single-point fault and has strong applicability.
On the basis of the above scheme, the method for guaranteeing the message timing of the distributed system of the present invention can be further improved as follows.
Further, the message consumer includes a message consumption main thread and N message service processing threads, where N is a positive integer and N is greater than or equal to 2, and S3 specifically includes the following steps:
s30, after each message topic sends the received message to the corresponding message consumption main thread, the message consumption main thread sends the received message to the corresponding message service processing thread;
and S31, each message service processing thread calls the corresponding application service according to the received message.
The beneficial effect of adopting the further scheme is that: the processing capacity of the message is further expanded through the N message service processing threads.
Further, S30 includes the following steps: and S301, enabling the message consumption main thread to judge whether the service processing thread corresponding to the received message is in an idle state, if so, continuing to execute S30, and if not, continuing to execute S301.
The beneficial effect of adopting the further scheme is that: and the message consumption main thread resends the received message to a corresponding message service processing thread, firstly judges whether the service processing thread is in an idle state, and repeatedly judges whether the message is normally written or not when the service processing thread is not in the idle state so as to prevent the message from being lost and missed.
Further, S301 includes the following steps: s3011, when the message consumption main thread judges that the service processing thread corresponding to the received message is not in an idle state, the message consumption main thread continuously judges whether a message buffer area of the service processing thread corresponding to the received message is in an idle state, if so, the message is buffered in the message buffer area, and if not, S301 is continuously executed.
The beneficial effect of adopting the further scheme is that: when the message consumption main thread judges that the service processing thread corresponding to the received message is not in an idle state, the message buffer area of the service processing thread is continuously judged whether to be in the idle state, if the message buffer area is in the idle state, the message is written into the message buffer area, the message processing speed is increased, and if the message buffer area is not in the idle state, the service processing thread and the message buffer area are repeatedly judged whether to be in the idle state, so that whether the message is normally written is determined, and the phenomena of loss and missing of the message are further prevented.
Further, S1 specifically is: taking M as a divisor to carry out remainder on the unique identifier of the business entity object to obtain a first remainder, dividing each message in the message database into M message lists to be generated according to the first remainder, and enabling each message producer to respectively obtain the message lists to be generated from the message database;
the business entity object is a client, a user or an order, and the corresponding unique identifier is a client number, a user number or an order number.
The beneficial effect of adopting the further scheme is that: and obtaining a first remainder after taking the M as a divisor for the unique identifier of the business entity object, such as the customer number, the user number and the order number, and carrying out fragmentation processing on the message database according to the first remainder to generate M corresponding to-be-generated message lists, thereby ensuring that the same business entity object corresponds to the same message producer.
Further, S30 specifically is: and obtaining a second remainder after the unique identifier of the business entity object is remainded again by taking N as a divisor, and enabling the message consumption main thread to resend the received message to the corresponding message business processing thread according to the second remainder.
The beneficial effect of adopting the further scheme is that: and obtaining a second remainder after carrying out remainting on the unique identifier of the business entity object, such as the customer number, the user number and the order number, by taking N as a divisor, and resending the received message to the corresponding message business processing thread according to the second remainder so as to ensure that the same business entity object corresponds to the same message business processing thread, namely, ensure that the same business entity object corresponds to the same message consumer.
Further, S3 includes the following steps: and finishing the data processing of the message through the application service, and sending and storing the message to an application database.
The beneficial effect of adopting the further scheme is that: the processing data of the message, such as the called application service, the specific information of the message and the like can be checked at any time through the application database.
Further, S2 includes the following steps: and after each message producer writes any message in the message list to be generated into the corresponding message theme, after the message middleware feeds back 'write success', writing the next message.
The beneficial effect of adopting the further scheme is that: and after the successful writing of the previous message is determined, the writing of the next message is performed, so that on one hand, the phenomena of loss and missing of the message are further prevented, and on the other hand, the generation of the time sequence of the message is ensured.
The technical scheme of the system for guaranteeing the message time sequence of the distributed system comprises the following steps:
the message generation system comprises an acquisition module, a writing module and a calling module, wherein the acquisition module is used for enabling preset M message producers to respectively acquire a to-be-generated message list in time sequence from a message database, wherein M is a positive integer and is more than or equal to 2;
the writing module is used for enabling each message producer to sequentially write the messages in the message list to be generated into corresponding message themes, wherein the number of the message themes is M, and the M message themes are all preset in a message middleware;
the calling module is used for enabling each message theme to respectively send the received message to each corresponding message consumer, and enabling each message consumer to call the corresponding application service according to the received message.
The system for guaranteeing the message time sequence of the distributed system has the advantages that: the acquisition module enables M preset message producers to respectively acquire a to-be-generated message list from the message database, namely the message database can be subjected to fragmentation processing to generate M corresponding to-be-generated message lists, the processing capacity of the messages is expanded, the writing module enables the M message producers to sequentially write the messages into M corresponding message themes, the processing capacity of the messages is further expanded, and single-point faults are effectively avoided, because the messages in the to-be-generated message list are sequentially sequenced according to time and sequentially written into the corresponding message themes, the calling module enables the message consumers to consume the corresponding messages, the time sequence production and the sequential consumption of the messages are ensured, and the M preset message producers in the message middleware are of the same type, namely only one type of message middleware is needed without depending on multiple types of message middleware, the applicability is strong. Therefore, the system for guaranteeing the message time sequence of the distributed system can enhance the message processing capacity, ensure the ordered production and consumption of the message time sequence, has no single-point fault and has strong applicability.
The technical scheme of the electronic equipment is as follows:
the system comprises a memory, a processor and a program stored in the memory and running on the processor, wherein the processor executes the program to realize the steps of the method for guaranteeing the message timing of the distributed system.
The electronic equipment has the following beneficial effects: the processor enables M preset message producers to respectively obtain the message list to be generated from the message database, namely the message database can be subjected to fragmentation processing to generate M corresponding message lists to be generated, the processing capacity of the messages is expanded, the M message producers write the messages into M corresponding message themes in sequence, the processing capacity of the messages is further expanded, and single-point faults are effectively avoided, because the messages in the message list to be generated are sequenced according to time and are written into the corresponding message themes in sequence, and the message consumers consume the corresponding messages, the ordered production and consumption of the message time sequence are ensured, and the M preset message producers in the message middleware are of the same type, namely only one type of message middleware is needed without depending on multiple types of message middleware, the applicability is strong. Therefore, the electronic equipment for guaranteeing the message time sequence of the distributed system can enhance the message processing capacity, ensure the ordered production and consumption of the message time sequence, has no single-point fault and has strong applicability.
Drawings
Fig. 1 is a schematic flowchart of a method for guaranteeing a message timing sequence of a distributed system according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart illustrating message production and message consumption in a distributed system message timing guarantee method according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a system for guaranteeing a message timing of a distributed system according to an embodiment of the present invention.
Detailed Description
The detailed explanation of the embodiment of the present invention is made with reference to fig. 1 and 2. As shown in fig. 1, a method for guaranteeing message timing in a distributed system according to an embodiment of the present invention includes the following steps,
s1, presetting M message producers, and enabling each message producer to respectively obtain a to-be-generated message list in time sequence from a message database, wherein M is a positive integer and is more than or equal to 2;
s2, presetting M message themes in the message middleware, and enabling each message producer to write the messages in the message list to be generated into the corresponding message themes in sequence;
and S3, enabling each message theme to respectively send the received message to each corresponding message consumer, and enabling each message consumer to call the corresponding application service according to the received message.
The method comprises the steps that M message producers are preset, each message producer respectively obtains a message list to be generated from a message database, namely the message database can be subjected to fragmentation processing to generate M corresponding message lists to be generated, the processing capacity of the messages is expanded, the M message producers sequentially write the messages into M corresponding message themes, the processing capacity of the messages is further expanded, and single-point faults are effectively avoided. Therefore, the method for guaranteeing the message time sequence of the distributed system can enhance the message processing capacity, ensure the ordered production and consumption of the message time sequence, has no single-point fault and has strong applicability.
As shown in fig. 2, for convenience of description, M message producers are respectively labeled as a first message producer and a second message producer … … mth message producer, M message topics are respectively labeled as a first message topic and a second message topic … … mth message topic, and the message database sharding process generates corresponding M to-be-generated message lists, which are respectively labeled as a first to-be-generated message list and a second to-be-generated message list … … mth to-be-generated message list.
The first message producer may obtain each message in the first to-be-generated message list, the second message producer may obtain each message in the second to-be-generated message list, and so on, the mth message producer may obtain each message in the mth to-be-generated message list, or may adopt a cross obtaining manner, for example, the first message producer may obtain each message in the mth to-be-generated message list, and so on.
In fig. 2, a first message producer corresponds to a first message topic, that is, the first message producer generates each message in a first to-be-generated message list or other to-be-generated message lists to the first message topic, a second message producer corresponds to the second message topic, and so on, the mth message producer corresponds to the mth message topic; and a cross-sending mode can also be adopted, for example, the first message producer corresponds to the M message subject, namely, the first message producer sends each message in the first message list to be generated or other message lists to be generated to the M message subject, and the like.
Preferably, in the above technical solution, the message consumer includes a message consumption main thread and N message service processing threads, where N is a positive integer and N is greater than or equal to 2, and S3 specifically includes the following steps:
s30, after each message topic sends the received message to the corresponding message consumption main thread, the message consumption main thread sends the received message to the corresponding message service processing thread;
and S31, each message service processing thread calls the corresponding application service according to the received message.
The processing capacity of the message is further expanded through the N message service processing threads.
For convenience of description, as shown in fig. 2, M message consumers are respectively labeled as a first message consumer and a second message consumer … … mth message consumer, wherein the first message consumer includes a first message consumption main thread and N message service processing threads, and the first message consumer is configured toThe N message service processing threads in the system are respectively marked as P11Message service processing thread, P12Message service processing thread … … P1NThe message service processing thread is analogized in turn, the Mth message consumer comprises an Mth message consumption main thread and N message service processing threads, and the N message service processing threads in the Mth message consumer are respectively marked as PM1Message service processing thread, PM2Message service processing thread … … PMNThe first message subject sends each message to the first message consumption main thread, and then the first message consumption main thread sends the message to the P11Message service processing thread, P12Message service processing thread … … P1NThe message service processing thread and the like, the Mth message subject sends each message to the Mth message consumption main thread, and the Mth message consumption main thread forwards the message to the Pth messageM1Message service processing thread, PM2Message service processing thread … … PMNCompared with the prior art, the message service processing thread, namely the method for guaranteeing the message time sequence of the distributed system, disclosed by the invention, has the advantages that the message processing capacity is improved by M multiplied by N times, the numerical values of M and N can be increased according to the actual situation, the message processing capacity is further expanded, and single-point faults can be effectively avoided. And the application service cluster stores a plurality of application service, and different application service is called for different messages to complete message consumption. And the message topics and the message consumption main threads can adopt a cross correspondence mode, for example, the first message topic sends each message to the Mth message consumption main thread and the like.
Preferably, in the above technical solution, S30 further includes the following steps: and S301, enabling the message consumption main thread to judge whether the service processing thread corresponding to the received message is in an idle state, if so, continuing to execute S30, and if not, continuing to execute S301. And the message consumption main thread resends the received message to a corresponding message service processing thread, firstly judges whether the service processing thread is in an idle state, and repeatedly judges whether the message is normally written or not when the service processing thread is not in the idle state so as to prevent the message from being lost and missed.
For example, there is a message on the first message consuming main thread that needs to be sent to P11A message service processing thread, before sending, a first message consumption main thread firstly judges P11Whether the message service processing thread is in an idle state or not, if P is the idle state11When the message service processing thread is in idle state, sending, if P is11When the message service processing thread is uploading a message, i.e. is not in idle state, the first message consumes the main thread pair P11Repeatedly judging whether the message service processing thread is in an idle state or not until P11When the message service processing thread is in idle state, the first message consumption main thread sends the message to P11The message service processing thread invokes the corresponding application service.
Preferably, in the above technical solution, S301 further includes the following steps: s3011, when the message consumption main thread judges that the service processing thread corresponding to the received message is not in an idle state, the message consumption main thread continuously judges whether a message buffer area of the service processing thread corresponding to the received message is in an idle state, if so, the message is buffered in the message buffer area, and if not, S301 is continuously executed.
When the message consumption main thread judges that the service processing thread corresponding to the received message is not in an idle state, the message buffer area of the service processing thread is continuously judged whether to be in the idle state, if the message buffer area is in the idle state, the message is written into the message buffer area, the message processing speed is increased, and if the message buffer area is not in the idle state, the service processing thread and the message buffer area are repeatedly judged whether to be in the idle state, so that whether the message is normally written is determined, and the phenomena of loss and missing of the message are further prevented.
For example, there is a message on the first message consuming main thread that needs to be sent to P11The message service processing thread, and the first message consumption main thread judges P11If the message service processing thread is not in idle state, the first message consumption main thread continues to judge P11Whether a message buffer area in the message service processing thread is in an idle state or not, if so, the message is written into the P11Message buffers in message service processing threads, when P11After the message service processing thread is in idle state, the message is transmitted to P11The message service processing thread invokes the corresponding application service. If not, repeatedly judging P11Whether a message service processing thread is in an idle state and P11Whether a message buffer in the message service processing thread is in an idle state.
Wherein, a first judgment frequency and a second judgment frequency can be set, and P is judged according to the first judgment frequency11Judging whether the message service processing thread is in an idle state or not according to the second frequency11Whether a message buffer in a message service processing thread is in an idle state, for example, if the first judgment frequency is 10 times/second, P is checked in one second11Judging whether the message service processing thread is in an idle state for 10 times, and if the second judgment frequency is 5 times/second, then P is processed within one second11Whether a message buffer in the message service processing thread is in an idle state is judged for 5 times. Furthermore, a buffer threshold may be set and P may be selected based on the buffer threshold11The message buffer in the message service processing thread is expanded, for example, the buffer threshold is set to 4, which indicates that the message service processing thread can be operated at P114 messages are buffered in a message buffer in the message service processing thread.
Preferably, in the above technical solution, S1 specifically is: taking M as a divisor to carry out remainder on the unique identifier of the business entity object to obtain a first remainder, dividing each message in the message database into M message lists to be generated according to the first remainder, and enabling each message producer to respectively obtain the message lists to be generated from the message database; the business entity object is a client, a user or an order, and the corresponding unique identifier is a client number, a user number or an order number. And obtaining a first remainder after taking the M as a divisor for the unique identifier of the business entity object, such as the customer number, the user number and the order number, and carrying out fragmentation processing on the message database according to the first remainder to generate M corresponding to-be-generated message lists, thereby ensuring that the same business entity object corresponds to the same message producer. In the telecommunication industry, a natural person or an enterprise and public institution has a plurality of devices such as a mobile phone number and a set top box, wherein the natural person is called a client, the devices such as the mobile phone number and the set top box are called a user, and an order represents the generation of message production and consumption; the customer number, the user number or the order number can be represented by 18-bit positive integers, or can be represented by 8-bit positive integers, 10-bit positive integers, 11-bit positive integers, 12-bit positive integers, 14-bit positive integers and 16-bit positive integers according to actual conditions.
The following is performed by using the client numbers expressed by 18-bit positive integers, and assuming that there are 6 clients, which are respectively marked as a first client, a second client, a third client, a fourth client, a fifth client and a sixth client, the client numbers are respectively: 000000000000000001, 000000000000000002, 000000000000000003, 000000000000000004, 000000000000000005 and 000000000000000006, wherein M is 3, that is, there are 3 message producers, that is, a first message producer, a second message producer and a third message producer, each customer number is complemented by taking M is 3 as a divisor, the first remainder corresponding to the customer number of the first customer is 1, the first remainder corresponding to the customer number of the second customer is 2, the first remainder corresponding to the customer number of the third customer is 0, the first remainder corresponding to the customer number of the fourth customer is 1, the first remainder corresponding to the customer number of the fifth customer is 2, the first remainder corresponding to the customer number of the sixth customer is 0, that is, when the first remainder is 0, the third customer and the sixth customer are corresponded to the first customer and the fourth customer, when the first remainder is 2, the second customer and the fifth customer are corresponded to the second customer, and when the first remainder is 0, a list of the customer messages and the third message in the sixth customer message database is divided into a list of messages to be generated, dividing the messages of the first client and the fourth client into a second message list to be generated, and dividing the messages of the second client and the fifth client into a third message list to be generated, that is, the message database is fragmented to generate 3 corresponding message lists to be generated.
The method comprises the steps that a first remainder is preset to be 0 and corresponds to a first message producer, the first remainder is preset to be 1 and corresponds to a second message producer, the first remainder is 2 and corresponds to a third message producer, the first message producer acquires a first to-be-generated message list generated by messages of a third client and a sixth client, the second message producer acquires a second to-be-generated message list generated by messages of the first client and a fourth client, and the third message producer acquires a second to-be-generated message list generated by messages of the second client and a fifth client, so that the same business entity object corresponds to the same message producer. Preferably, in the above technical solution, S30 specifically is: and obtaining a second remainder after the unique identifier of the business entity object is remainded again by taking N as a divisor, and enabling the message consumption main thread to resend the received message to the corresponding message business processing thread according to the second remainder. And obtaining a second remainder after carrying out remainting on the unique identifier of the business entity object, such as the customer number, the user number and the order number, by taking N as a divisor, and resending the received message to the corresponding message business processing thread according to the second remainder so as to ensure that the same business entity object corresponds to the same message business processing thread, namely, ensure that the same business entity object corresponds to the same message consumer.
The above is still further described by way of example: suppose that the first message producer acquires the third client and the sixth client and forwards them to the first message consumption main thread, and sets N to 2, that is, there are 2 message service processing threads, P respectively11Message service processing thread and P12Message traffic processing thread, then:
continuing to use the client numbers of the third client and the sixth client, and taking N-2 as a divisor to carry out remainder on the client numbers of the third client and the sixth client, wherein a second remainder after the remainder of the client number corresponding to the third client is 1; the second remainder after the balance of the customer number corresponding to the sixth customer is 0; when the preset second remainder is 0, the corresponding P is11The message service processing thread corresponds to P when the second remainder is 112Message service processing thread, then the third client's message is sent to P12Message service processing thread, the message of the sixth client is sent to P12Message service processing threads, howeverAnd then calling corresponding application service from the application service cluster respectively.
When the user number or the order number is used, the user number is referred to, and details are not described herein.
Preferably, in the above technical solution, S3 further includes the following steps: and finishing the data processing of the message through the application service, and sending and storing the message to an application database. The processing data of the message, such as the called application service and the specific information of the message, can be checked at any time through the application database, and the specific information of the message comprises the message production time, the message consumption time and the like.
In another embodiment the message database and the application database may be made in the same database.
Preferably, in the above technical solution, S2 further includes the following steps: and after each message producer writes any message in the message list to be generated into the corresponding message theme, after the message middleware feeds back 'write success', writing the next message. And after the successful writing of the previous message is determined, the writing of the next message is performed, so that on one hand, the phenomena of loss and missing of the message are further prevented, and on the other hand, the generation of the time sequence of the message is ensured. And performing specific analysis according to the content, if the first message producer writes the third message into the first message theme of the message middleware, if the message middleware feeds back "write success", continuing to write the sixth message, and if the message middleware does not feed back "write success", suspending writing the sixth message and giving a corresponding prompt.
As shown in fig. 3, a distributed system message timing guarantee system 200 according to an embodiment of the present invention includes an obtaining module 210, a writing module 220, and a calling module 230, where the obtaining module 210 is configured to enable preset M message producers to respectively obtain a to-be-generated message list in chronological order from a message database, where M is a positive integer and is greater than or equal to 2;
the writing module 220 is configured to enable each message producer to sequentially write the messages in the message list to be generated into corresponding message topics, where the number of the message topics is M, and the M message topics are all preset in a message middleware;
the invoking module 230 is configured to enable each message topic to send the received message to each corresponding message consumer, and enable each message consumer to invoke a corresponding application service according to the received message.
The obtaining module 210 enables preset M message producers to obtain a to-be-generated message list from a message database respectively, that is, the message database performs fragmentation processing to generate corresponding M to-be-generated message lists, expands the processing capability of messages, and the writing module 220 enables the M message producers to write each message into corresponding M message themes in sequence, further expands the processing capability of messages, and effectively avoids the occurrence of single-point faults, because each message in the to-be-generated message list is ordered in sequence according to time and sequentially written into corresponding message themes, the calling module 230 enables each message consumer to consume the corresponding message, thereby ensuring the ordered production and consumption of message time sequence, and the M message producers preset in the message middleware are of the same type, that is, only one type of message middleware is needed, and do not need to rely on multiple types of message middleware, the applicability is strong. Therefore, the system 200 for guaranteeing the message time sequence of the distributed system can enhance the message processing capacity, ensure the ordered production and consumption of the message time sequence, has no single-point fault and has strong applicability.
Preferably, in the above technical solution, the message consumer further includes a thread delivery module, where the message consumer includes a message consumption main thread and N message service processing threads, where N is a positive integer and N is greater than or equal to 2, and the thread delivery module is configured to enable each message topic to respectively send the received message to the corresponding message consumption main thread, and then enable the message consumption main thread to send the received message to the corresponding message service processing thread; each message service processing thread calls the corresponding application service through the calling module 230 according to the received message. The processing capacity of the message is further expanded through the N message service processing threads.
Preferably, in the above technical solution, the system further includes a judging module, where the judging module is configured to enable the message consumption main thread to judge whether a service processing thread corresponding to the received message is in an idle state, if so, the thread passing module enables the message consumption main thread to resend the received message to the corresponding message service processing thread, and if not, the judging module repeatedly judges;
and the judging module firstly judges whether the service processing thread is in an idle state or not, and repeatedly judges whether the message is normally written or not when the service processing thread is not in the idle state so as to prevent the message from being lost and missed.
Preferably, in the above technical solution, when the determining module makes the message consumption main thread determine that the service processing thread corresponding to the received message is not in the idle state, it continues to determine whether the message buffer area of the service processing thread corresponding to the message received by the message consumption main thread is in the idle state, and if so, the writing module 220 buffers the message into the message buffer area.
When the judging module makes the message consumption main thread judge that the service processing thread corresponding to the received message is not in the idle state, the judging module continuously judges whether the message buffer area of the service processing thread is in the idle state, if the message buffer area is in the idle state, the writing module 220 writes the message into the message buffer area to increase the message processing speed, and if the message buffer area is not in the idle state, the judging module continuously judges whether the service processing thread and the message buffer area are in the idle state to repeatedly judge so as to determine whether the message is normally written in, thereby further preventing the message from being lost and missed.
Preferably, in the above technical solution, the apparatus includes a first remainder module, and the first remainder module
Taking M as a divisor to carry out remainder on the unique identifier of the business entity object to obtain a first remainder, dividing each message in the message database into M message lists to be generated according to the first remainder, and enabling each message producer to respectively obtain the message lists to be generated from the message database;
the business entity object is a client, a user or an order, and the corresponding unique identifier is a client number, a user number or an order number.
And obtaining a first remainder after taking the M as a divisor for the unique identifier of the business entity object, such as the customer number, the user number and the order number, and carrying out fragmentation processing on the message database according to the first remainder to generate M corresponding to-be-generated message lists, thereby ensuring that the same business entity object corresponds to the same message producer.
Preferably, in the above technical solution, the message consumption main thread sends the received message to the corresponding message service processing thread again according to the second remainder, where the second remainder module obtains the second remainder after taking N as a divisor to carry out another remainder on the unique identifier of the service entity object.
The second remainder module performs remainder again on unique identifiers of the business entity objects, such as a client number, a user number and an order number, by taking N as a divisor to obtain a second remainder, and resends the received messages to corresponding message business processing threads according to the second remainder, so as to ensure that the same business entity object corresponds to the same message business processing threads, that is, ensure that the same business entity object corresponds to the same message consumers. Preferably, in the above technical solution, the system further includes a storage module, the data processing of the message is completed through the application service, and the storage module is configured to send and store the data to the application database.
The processing data of the message, such as the called application service, the specific information of the message and the like can be checked at any time through the application database.
Preferably, in the above technical solution, the message processing device further includes a feedback module, where the feedback module is configured to, after each message producer writes any one of the messages in the to-be-generated message list into the corresponding message topic, and after the message middleware feeds back "write success", perform writing of the next message.
The above-mentioned steps for implementing the corresponding functions for each parameter and each unit module of the system for guaranteeing the message timing of the distributed system according to the present invention may refer to each parameter and step in the above-mentioned embodiment of the method for guaranteeing the message timing of the distributed system, which are not described herein again.
An electronic device according to an embodiment of the present invention includes a memory, a processor, and a program stored in the memory and running on the processor, where the processor implements the steps of the method for guaranteeing a distributed system message timing sequence according to any one of the above descriptions when executing the program.
The processor enables M preset message producers to respectively obtain the message list to be generated from the message database, namely the message database can be subjected to fragmentation processing to generate M corresponding message lists to be generated, the processing capacity of the messages is expanded, the M message producers write the messages into M corresponding message themes in sequence, the processing capacity of the messages is further expanded, and single-point faults are effectively avoided, because the messages in the message list to be generated are sequenced according to time and are written into the corresponding message themes in sequence, and the message consumers consume the corresponding messages, the ordered production and consumption of the message time sequence are ensured, and the M preset message producers in the message middleware are of the same type, namely only one type of message middleware is needed without depending on multiple types of message middleware, the applicability is strong. Therefore, the electronic equipment for guaranteeing the message time sequence of the distributed system can enhance the message processing capacity, ensure the ordered production and consumption of the message time sequence, has no single-point fault and has strong applicability.
The electronic device can be a computer, a mobile phone, or the like, and correspondingly, the program is computer software or a mobile phone APP, or the like.
In addition, the above parameters and steps in the electronic device according to the present invention may refer to the parameters and steps in the embodiment of the method for guaranteeing the message timing of the distributed system, which are not described herein again.
In the present invention, the terms "first", "second" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and that variations, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present invention.
Claims (10)
1. A method for guaranteeing message time sequence of a distributed system is characterized by comprising the following steps:
s1, presetting M message producers, and enabling each message producer to respectively obtain a to-be-generated message list in time sequence from a message database, wherein M is a positive integer and is more than or equal to 2;
s2, presetting M message themes in the message middleware, and enabling each message producer to write the messages in the message list to be generated into the corresponding message themes in sequence;
and S3, enabling each message theme to respectively send the received message to each corresponding message consumer, and enabling each message consumer to call the corresponding application service according to the received message.
2. The method of claim 1, wherein the message consumer comprises a message consumption main thread and N message service processing threads, wherein N is a positive integer and N is greater than or equal to 2;
s3 specifically includes the following steps:
s30, after each message theme sends the received message to the corresponding message consumption main thread, the message consumption main thread sends the received message to the corresponding message service processing thread;
and S31, each message service processing thread calls the corresponding application service according to the received message.
3. The method of claim 2, wherein the step of S30 further comprises:
and S301, enabling the message consumption main thread to judge whether the service processing thread corresponding to the received message is in an idle state, if so, continuing to execute S30, and if not, continuing to execute S301.
4. The method of claim 3, wherein S301 further comprises the following steps:
s3011, when the message consumption main thread judges that the service processing thread corresponding to the received message is not in an idle state, the message consumption main thread continuously judges whether a message buffer area of the service processing thread corresponding to the received message is in an idle state, if so, the message is buffered in the message buffer area, and if not, S301 is continuously executed.
5. The method for guaranteeing message timing of a distributed system according to any one of claims 2 to 4, wherein S1 specifically is:
taking M as a divisor to carry out remainder on the unique identifier of the business entity object to obtain a first remainder, dividing each message in the message database into M message lists to be generated according to the first remainder, and enabling each message producer to respectively obtain the message lists to be generated from the message database;
the business entity object is a client, a user or an order, and the corresponding unique identifier is a client number, a user number or an order number.
6. The method for guaranteeing message timing of a distributed system according to claim 5, wherein S30 specifically is:
and obtaining a second remainder after the unique identifier of the business entity object is remainded again by taking N as a divisor, and enabling the message consumption main thread to resend the received message to the corresponding message business processing thread according to the second remainder.
7. The method for guaranteeing message timing of a distributed system according to any one of claims 1 to 4 or 6, wherein S3 further comprises the following steps: and finishing the data processing of the message through the application service, and sending and storing the message to an application database.
8. The method for guaranteeing message timing of a distributed system according to any one of claims 1 to 4 or 6, wherein the step of S2 further comprises:
and after each message producer writes any message in the message list to be generated into the corresponding message theme, after the message middleware feeds back 'write success', writing the next message.
9. A distributed system message time sequence guarantee system is characterized by comprising an acquisition module, a writing module and a calling module,
the acquisition module is used for enabling preset M message producers to respectively acquire a to-be-generated message list in time sequence from a message database, wherein M is a positive integer and is more than or equal to 2;
the writing module is used for enabling each message producer to sequentially write the messages in the message list to be generated into corresponding message themes, wherein the number of the message themes is M, and the M message themes are all preset in a message middleware;
the calling module is used for enabling each message theme to respectively send the received message to each corresponding message consumer, and enabling each message consumer to call the corresponding application service according to the received message.
10. An electronic device comprising a memory, a processor and a program stored on the memory and running on the processor, wherein the processor implements the steps of a method of distributed system message timing assurance as claimed in any one of claims 1 to 8 when executing the program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910954949.8A CN110688228A (en) | 2019-10-09 | 2019-10-09 | Method, system and electronic equipment for guaranteeing distributed system message time sequence |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910954949.8A CN110688228A (en) | 2019-10-09 | 2019-10-09 | Method, system and electronic equipment for guaranteeing distributed system message time sequence |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110688228A true CN110688228A (en) | 2020-01-14 |
Family
ID=69111772
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910954949.8A Pending CN110688228A (en) | 2019-10-09 | 2019-10-09 | Method, system and electronic equipment for guaranteeing distributed system message time sequence |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110688228A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111522672A (en) * | 2020-04-24 | 2020-08-11 | 北京思特奇信息技术股份有限公司 | Method and system for repeated consumption of fusing data |
| CN111541824A (en) * | 2020-04-21 | 2020-08-14 | 北京思特奇信息技术股份有限公司 | Method and system for communication information transmission |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104092767A (en) * | 2014-07-21 | 2014-10-08 | 北京邮电大学 | Posting/subscribing system for adding message queue models and working method thereof |
| CN108021358A (en) * | 2017-12-15 | 2018-05-11 | 无线生活（杭州）信息科技有限公司 | A kind of data processing method and device |
| CN108874562A (en) * | 2018-06-21 | 2018-11-23 | 北京顺丰同城科技有限公司 | Distributed high concurrent message queue supplying system |
| CN109067844A (en) * | 2018-07-09 | 2018-12-21 | 上海瀚银信息技术有限公司 | A kind of message communication system |
| CN109684099A (en) * | 2018-11-23 | 2019-04-26 | 新华三大数据技术有限公司 | Message treatment method and device |
-
2019
- 2019-10-09 CN CN201910954949.8A patent/CN110688228A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104092767A (en) * | 2014-07-21 | 2014-10-08 | 北京邮电大学 | Posting/subscribing system for adding message queue models and working method thereof |
| CN108021358A (en) * | 2017-12-15 | 2018-05-11 | 无线生活（杭州）信息科技有限公司 | A kind of data processing method and device |
| CN108874562A (en) * | 2018-06-21 | 2018-11-23 | 北京顺丰同城科技有限公司 | Distributed high concurrent message queue supplying system |
| CN109067844A (en) * | 2018-07-09 | 2018-12-21 | 上海瀚银信息技术有限公司 | A kind of message communication system |
| CN109684099A (en) * | 2018-11-23 | 2019-04-26 | 新华三大数据技术有限公司 | Message treatment method and device |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111541824A (en) * | 2020-04-21 | 2020-08-14 | 北京思特奇信息技术股份有限公司 | Method and system for communication information transmission |
| CN111541824B (en) * | 2020-04-21 | 2021-11-26 | 北京思特奇信息技术股份有限公司 | Method and system for communication information transmission |
| CN111522672A (en) * | 2020-04-24 | 2020-08-11 | 北京思特奇信息技术股份有限公司 | Method and system for repeated consumption of fusing data |
| CN111522672B (en) * | 2020-04-24 | 2023-02-28 | 北京思特奇信息技术股份有限公司 | Method and system for repeated consumption of fusing data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107465767B (en) | Data synchronization method and system | |
| CN110808922B (en) | Message processing method and device, storage medium and electronic equipment | |
| CN107450971B (en) | Task processing method and device | |
| CN110096336B (en) | Data monitoring method, device, equipment and medium | |
| CN106528574A (en) | Data synchronization method and device | |
| CN109766172B (en) | Asynchronous task scheduling method and device | |
| CN110688228A (en) | Method, system and electronic equipment for guaranteeing distributed system message time sequence | |
| CN111541762B (en) | Data processing method, management server, device and storage medium | |
| CN111198662B (en) | Data storage method, device and computer readable storage medium | |
| CN113703954A (en) | Message backup method and device, electronic equipment and computer storage medium | |
| CN112650575B (en) | Resource scheduling method, device and cloud service system | |
| CN111459639A (en) | Distributed task management platform and method supporting global multi-machine-room deployment | |
| CN110071968B (en) | Block chain based message storage method and device | |
| CN115328625A (en) | Thread pool parameter processing method, system, computer device, medium, and program product | |
| CN107025131B (en) | Task scheduling method and device | |
| CN111475315A (en) | Server and subscription notification push control and execution method | |
| CN115023929A (en) | Data synchronization method, device, system, electronic equipment and storage medium | |
| CN111147585A (en) | Equipment upgrading method, device, storage medium and system | |
| JP2009515474A (en) | Independent message store and message transport agent | |
| CN114338584B (en) | Message withdraw method and message transmission system | |
| CN107423131B (en) | Sharing method and server | |
| CN112835862B (en) | Data synchronization method, device, system and storage medium | |
| CN111563069A (en) | Probe configuration method and system of industrial control equipment | |
| CN108574622B (en) | Instant message processing method and device based on XMPP | |
| CN112463514A (en) | Monitoring method and device for distributed cache cluster |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200114 |