CN113176967A - Data processing method and device, electronic equipment and computer readable storage medium - Google Patents
Data processing method and device, electronic equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN113176967A CN113176967A CN202110475234.1A CN202110475234A CN113176967A CN 113176967 A CN113176967 A CN 113176967A CN 202110475234 A CN202110475234 A CN 202110475234A CN 113176967 A CN113176967 A CN 113176967A
- Authority
- CN
- China
- Prior art keywords
- server
- data
- message
- processing
- thread
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1415—Saving, restoring, recovering or retrying at system level
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Retry When Errors Occur (AREA)
Abstract
The embodiment of the application provides a data processing method and device, electronic equipment and a computer readable storage medium; the method comprises the following steps: under the condition that the execution of a first processing operation corresponding to the first message fails, the first server writes first data corresponding to the first message into a retry queue; the first processing operation is any one of a plurality of processing operations included in the processing process of the first message by the first server; the first server starts a first thread and re-executes the first processing operation on the first data in the retry queue through the first thread; the plurality of processing operations are performed based on a second thread, the first thread being different from the second thread. By the method and the device, the operation which fails to be executed can be re-executed by another thread, so that the operation which fails to be executed is ensured to continue to be executed downwards, and the stability of the execution of the message is ensured.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a data processing method and apparatus, an electronic device, and a computer-readable storage medium.
Background
With the popularization of the internet, the requirements of high concurrency and quick response of part of industries on a server side are higher and higher; such as part of the system where extremely high flow rates may occur for short periods of time. With conventional systems, this very high traffic in a short period of time is catastrophic, may directly overwhelm the server,
in this regard, a Message Queue (MQ) is introduced in the prior art, and is a communication method between applications, and is generally used for data transfer between two services in different applications, and has functions of traffic peak clipping, decoupling, and asynchronization by establishing a persistent Message storage between two or more services. For example, for A, B two services with data transfer requirements (service A needs to transfer data to service B), since A and B have MQ services, MQ servers are added to the system where A and B are located; wherein, A is used as a message provider and is responsible for producing messages; the MQ server is used for storing messages and processing some message sending or receiving logics; and B is used as a message consumer and is responsible for consuming the message.
As can be seen, the introduction of MQ alleviates the problems of high concurrency and strong coupling of the server. However, in the process of processing the message after the message consumer receives the message, if one of the plurality of processing links fails, for example, the write operation fails, since the previous step is executed, the data in the current state cannot be rolled back, that is, the data can only be executed downwards from the step after the step that failed to be executed, so that the stability of the message processing at the consumer end is reduced.
Disclosure of Invention
An object of the embodiments of the present application is to provide a data processing method and apparatus, an electronic device, and a computer-readable storage medium, so as to ensure stability of message execution. The specific technical scheme is as follows:
in a first aspect of this embodiment, a method for processing data is provided, including: under the condition that execution of a first processing operation corresponding to a first message fails, a first server writes first data corresponding to the first message into a retry queue, wherein the first message is sent to a first server consuming messages by a second server producing messages; the first processing operation is any one of a plurality of processing operations included in the processing process of the first message by the first server; the first server starts a first thread and re-executes the first processing operation on the first data in the retry queue through the first thread, wherein the plurality of processing operations are executed based on a second thread, and the first thread is different from the second thread.
In a second aspect of the present application, there is also provided a data processing apparatus, applied to a first server, including: the device comprises a first writing module, a second writing module and a retry queue, wherein the first writing module is used for writing first data corresponding to a first message into the retry queue under the condition that the execution of a first processing operation corresponding to the first message fails, and the first message is sent to a first server consuming messages by a second server producing the messages; the first processing operation is any one of a plurality of processing operations included in the processing process of the first message by the first server; a first processing module, configured to start a first thread and re-execute the first processing operation on the first data in the retry queue through the first thread, where the plurality of processing operations are executed based on a second thread, and the first thread is different from the second thread.
In yet another aspect of this embodiment, there is also provided a computer-readable storage medium having stored therein instructions, which when executed on a computer, cause the computer to execute any of the above-described data processing methods.
In yet another aspect of this application implementation, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform the method for processing data as described in any one of the above.
In the data processing method provided in the embodiment of the present application, when the first processing operation corresponding to the first message fails to be executed, the first data corresponding to the first message is written into the retry queue, and then the first thread executes the write operation again on the first data; the first message is sent to the first server consuming the message by the second server producing the message, and the first processing operation is any one of a plurality of processing operations included in the processing process of the first message by the first server, that is, the first server receiving the message can restart the thread to execute the processing operation again on the data with the execution failure in the message execution process under the condition that the execution of any one of the plurality of processing operations fails, that is, the execution of the operation with the execution failure can be executed again by the other thread, so as to ensure that the downward execution is continued from the operation with the execution failure, thereby avoiding the problem that the whole process corresponding to the message cannot be completely executed because the execution is required to be executed after the step with the execution failure after the certain processing operation fails. Therefore, even if the processing operation of the execution failure occurs, the processing operation can be executed again on the data of the execution failure in the retry queue by another thread, so that the message execution process can be completely and smoothly executed downwards, and the stability of the message execution process is ensured.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.
FIG. 1 is a flow chart of a data processing method according to an embodiment of the present application;
FIG. 2 is a diagram illustrating a data retry process according to an embodiment of the present application;
fig. 3 is a second flowchart of a data processing method in the embodiment of the present application.
FIG. 4 is a third flowchart of a data processing method in the embodiment of the present application;
FIG. 5 is a fourth flowchart of a data processing method according to an embodiment of the present application;
FIG. 6 is a second schematic diagram illustrating data retry processing according to an embodiment of the present application;
FIG. 7 is a fifth flowchart of a data processing method according to an embodiment of the present application;
FIG. 8 is a sixth flowchart of a data processing method according to an embodiment of the present application;
FIG. 9 is a third diagram illustrating data retry processing according to an embodiment of the present application;
FIG. 10 is a diagram illustrating abnormal data statistics and reporting in an embodiment of the present application;
FIG. 11 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application;
fig. 12 is a schematic structural diagram of an electronic device in an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.
An embodiment of the present application provides a data processing method, as shown in fig. 1, the method includes:
102, under the condition that the execution of a first processing operation corresponding to a first message fails, a first server writes first data corresponding to the first message into a retry queue, wherein the first message is sent to a first server consuming messages by a second server producing messages; the first processing operation is any one of a plurality of processing operations included in the processing process of the first message by the first server;
and step 104, the first server starts a first thread and re-executes the first processing operation on the first data in the retry queue through the first thread, wherein a plurality of processing operations are executed based on a second thread, and the first thread is different from the second thread.
Through the above steps 102 and 104, in case that the execution of the first processing operation corresponding to the first message fails, writing the first data corresponding to the first message into the retry queue, and then starting another first thread to re-execute the write operation on the first data; the first message is sent to the first server consuming the message by the second server producing the message, and the first processing operation is any one of a plurality of processing operations included in the processing process of the first message by the first server, that is, the first server receiving the message can restart the thread to execute the processing operation again on the data with the execution failure in the message execution process under the condition that the execution of any one of the plurality of processing operations fails, that is, the execution of the operation with the execution failure can be executed again by the other thread, so as to ensure that the downward execution is continued from the operation with the execution failure, thereby avoiding the problem that the whole process corresponding to the message cannot be completely executed because the execution is required to be executed after the step with the execution failure after the certain processing operation fails. Therefore, even if the processing operation of the execution failure occurs, the processing operation can be executed again on the data of the execution failure in the retry queue by another thread, so that the message execution process can be completely and smoothly executed downwards, and the stability of the message execution process is ensured.
The following explains the above steps 102 to 104 with reference to a specific application scenario.
As shown in fig. 2, after the producer service a in the second server goes online, the message is sent to the MQ server, and then the MQ server obtains the message by pushing or actively pulling by the consumer service B in the first server, and the message enters the first server and then goes through a plurality of processing operations, for example: status update, warehousing, logging (corresponding to a plurality of processing operations for processing the first message in the embodiment of the present application). When one of the processing operations fails to be executed, for example, a warehousing operation (corresponding to the first processing operation), since the previous step (status update) is executed and cannot be rolled back, only the data of the executed status update operation can be saved, that is, only the processing operations (write logs) after the current execution failure can be executed downwards, and the processing operations cannot be completely executed, so that the stability of the message processing at the consumer end is low. For such a situation, the data corresponding to the to-be-binned operation may be written into a retry queue of the first server, where the retry queue is used to store the data that has failed in the binning operation, and another thread (corresponding to the first thread in the embodiment of the present application, and the thread that normally executes the multiple processing operations is the second thread) is started in the first server to perform the binning operation again on the data that corresponds to the binning operation that has failed to be performed, so that the first server may continue to re-perform the binning operation based on the data taken out from the retry queue, and then perform the step after binning.
In an optional implementation manner of the embodiment of the present application, for the first server involved in step 102 to write the first data corresponding to the first message into the retry queue, the method further includes: the first server writes the first data and a time stamp corresponding to the first data into a retry queue, wherein the time stamp is an execution time of the first processing operation.
As can be seen, in the embodiment of the present application, in addition to the first data corresponding to the first processing operation whose execution failed needs to be written into the retry queue, the execution time (timestamp) of the first processing operation corresponding to the first data needs to be written into the retry queue. After entering the first server with the first message, a plurality of processing operations are performed, for example: status update, warehousing, log writing are examples. The timestamps of the three processing operations are respectively a first time, a second time and a third time, and the first time is earlier than the second time and the second time is earlier than the third time. When a certain processing operation fails to be executed, for example, after the warehousing operation fails to be executed, the first server needs to write the data corresponding to the warehousing operation and the second time into the retry queue.
In another alternative implementation of the embodiment of the present application, as shown in fig. 3, the method of the embodiment of the present application may further include, in addition to the method steps in fig. 1, as described above:
in step 306, the first server prohibits starting the first thread when the second processing operation is successful and the first processing operation has not been re-executed on the first data by the first thread.
As can be seen from steps 302 to 306, in a case where there may exist a plurality of messages corresponding to the same data (the first data corresponds to both the first message and the second message), for example, the first data is 1, and both the first message and the second message may update the first data. If the processing operation of the first message is to update 1 to 2, the processing operation of the second message is to update data to 3 if the first data is 1, or to update data to 3 if the first data is 2, and the execution time of the first message is earlier than that of the second message. That is, the execution result of the processing operation corresponding to the second message will override the execution result of the processing operation corresponding to the first message, that is, when the first message and the second message are normally executed, 1 will be updated to 3 finally. However, in the case where the execution of the first message fails, the corresponding data is written into the retry queue, and in this case, it is necessary to further determine whether the execution of the second message is completed, and if the execution of the second message is completed, the data corresponding to the first message written into the retry queue due to the execution failure does not need to be re-executed. That is to say, the processing procedure of the normal service is that "1" is updated to "2", then "2" is updated to "3", if the execution of the updated operation in the current first message fails, after the data corresponding to the updated operation is written into the retry queue, it needs to be further determined whether the second message is successfully executed, that is, whether the number "1" is updated to "3", if the update is "3", it is not necessary to start a new thread to execute the write operation on the data in the retry queue, and if the update is not "3", it is necessary to re-execute the write operation on the data corresponding to the first message in the retry queue.
In another alternative implementation of the embodiments of the present application, as shown in fig. 4, the method of the embodiments of the present application may further include, in addition to the method steps in fig. 1:
It should be noted that before the processing process of the message is stopped, it means before all the processing operations corresponding to the message are completed, because the corresponding data is easily lost after the processing operations are completed. Moreover, in an optional implementation manner of the embodiment of the present application, in order to reduce a load of data storage, in the embodiment of the present application, data in the retry queue is preferably memory data, that is, corresponding data may also be lost when a plurality of processing operations are completed. In addition, if too much data is stored in the retry queue, memory overflow, i.e., data loss, may result. If the data is lost, the write operation cannot be re-performed on the data in the retry queue. Therefore, in the embodiment of the present application, before the processing of the message is stopped, or in a case that the data amount of the data in the retry queue is greater than or equal to the preset threshold, the data in the retry queue needs to be moved to the memory medium, that is, the data needs to be persisted. In a specific application scenario of the embodiment of the application, only one of the modes may be needed, or the two modes may be used in combination, and the specific mode may be set correspondingly in combination with a specific actual situation.
It should be noted that in the embodiment of the present application, the memory medium may be a disk of a server, and may also be a proprietary storage unit in other application scenarios, such as MySQL in a relational database management system; if the performance requirement exists, the data can be stored in a distributed cache, such as a REmote DIctionary Server (Redis).
When the persistent operation is triggered, it is indicated that the message processing problem is serious, at this time, reporting exception can be notified, and the specific notification level can be set correspondingly according to actual requirements. As shown in fig. 5, the method steps of the embodiment of the present application include, in addition to the method steps in fig. 4, after performing persistence processing on data corresponding to a message in a retry queue, the method of the embodiment of the present application may further include:
It can be seen that if more and more data are in the current retry queue, indicating that the current processing through the retry queue is more and more heavily loaded, a notification message needs to be sent to remind the relevant person. As shown in fig. 6, when the data in the retry queue is processed by the target thread, exception reporting (reporting of the first notification message) is performed while performing persistence.
In yet another alternative implementation of the embodiments of the present application, as shown in fig. 7, the method steps of the embodiments of the present application may include, in addition to the method steps in fig. 1:
in step 702, in the case that the first processing operation is successfully re-executed on the first data by the first thread, the first server deletes the first data from the retry queue.
As can be seen from the above step 702, for the data in the retry queue that has successfully executed the write operation, the write operation does not need to be executed again, and the data can be deleted from the retry queue, so as to make room for storing other data, and avoid redundancy of the data in the retry queue.
In another optional implementation manner of the embodiment of the present application, as shown in fig. 8, in a case that execution of a first write operation corresponding to a first message fails, before a first server related to step 102 in the embodiment of the present application writes first data corresponding to the first message into a retry queue, the method step of the embodiment of the present application may further include:
As shown in the above step 802, when the first processing operation fails to be executed, the write operation may be directly re-executed on the first processing operation, and if the processing operation can be directly re-executed successfully, the first data corresponding to the first processing operation does not need to be written into the retry queue. It should be noted that, in the embodiment of the present application, the processing operation may be re-executed for multiple times, and the number of times of re-executing the processing operation may be a preset value, for example, 3 times, 5 times, and the like. For the specific application scenario of this step 802, based on fig. 6, it may be that as shown in fig. 9, after the producer service a in the second server goes online, the message is sent to the MQ server, and then the MQ server may obtain the message by pushing or actively pulling by the consumer service B in the first server, and the message may go through a plurality of processing operations after entering the first server, for example: status update, warehousing, logging (corresponding to a plurality of processing operations for processing the first message in the embodiment of the present application). When one of the processing operations fails to be executed, for example, a warehousing operation (corresponding to a first processing operation), the warehousing operation is retried after the warehousing operation fails, the upper limit of the retry is three times, if the retry is successful within three times, data corresponding to the warehousing does not need to be written into a retry queue, and if the execution is still not successful after the retry is three times, the data corresponding to the warehousing is written into the retry queue.
In addition, in a specific application scenario, the manner involved in the above steps 804 and 806 may be that, as shown in fig. 10, the corresponding modules (e.g., the delivery collecting module) are started simultaneously when the service is started. When the storage fails, firstly writing the data into the log, storing the log into a disk, and defining different types (such as storage failure, queue data fullness and the like) in the log for later data statistics. After the abnormal log is written into the disk, the collecting and delivering module analyzes the stored log and delivers the analysis result to the statistical center; the statistical center aggregates log data such as statistics of abnormal attributes, e.g., abnormal type, abnormal number, storage of statistical results, etc. After the statistics is completed, the alarm center is called to send alarm short messages or mails (notification messages), meanwhile, the statistical results can also be sent to the display module, and abnormal conditions are displayed in different dimensions.
As shown in fig. 11, an embodiment of the present application further provides an apparatus for processing data, where the apparatus includes:
a first writing module 112, configured to write first data corresponding to a first message into a retry queue when execution of a first processing operation corresponding to the first message fails, where the first message is sent from a second server that produces the message to a first server that consumes the message; the first processing operation is any one of a plurality of processing operations included in the processing process of the first message by the first server;
the first processing module 114 is configured to start a first thread and re-execute a first processing operation on first data in the retry queue through the first thread, wherein the first thread is different from a second thread based on a plurality of processing operations being executed by the second thread.
By the device in the embodiment of the present application, under the condition that the execution of the first processing operation corresponding to the first message fails, the first data corresponding to the first message is written into the retry queue, and then the first thread is started to perform the writing operation again on the first data; the first message is sent to the first server consuming the message by the second server producing the message, and the first processing operation is any one of a plurality of processing operations included in the processing process of the first message by the first server, that is, the first server receiving the message can restart the thread to execute the processing operation again on the data with the execution failure in the message execution process under the condition that the execution of any one of the plurality of processing operations fails, that is, the execution of the operation with the execution failure can be executed again by the other thread, so as to ensure that the downward execution is continued from the operation with the execution failure, thereby avoiding the problem that the whole process corresponding to the message cannot be completely executed because the execution is required to be executed after the step with the execution failure after the certain processing operation fails. Therefore, even if the processing operation of the execution failure occurs, the processing operation can be executed again on the data of the execution failure in the retry queue by another thread, so that the message execution process can be completely and smoothly executed downwards, and the stability of the message execution process is ensured.
Optionally, the first writing module 112 in this embodiment of the present application further includes: and a writing unit, configured to write the first data and a timestamp corresponding to the first data into a retry queue, where the timestamp is an execution time of the first processing operation.
Optionally, the apparatus in this embodiment of the present application may further include: a determining module, configured to determine, before re-executing the first processing operation on the first data by the first thread and under a condition that the first data further corresponds to the second message, an execution state of a second processing operation corresponding to the second message; wherein the result of the execution of the second processing operation overrides the result of the execution of the first processing operation; the execution time of the second processing operation is later than that of the first processing operation; the second writing module is used for writing the first data corresponding to the second message into a retry queue under the condition that the execution of the second processing operation fails; and the forbidding module is used for forbidding to start the first thread under the condition that the second processing operation is successful and the first processing operation is not executed again on the first data through the first thread.
Optionally, the apparatus in this embodiment of the present application may further include: the persistence module is used for performing persistence processing on the data in the retry queue under the condition that at least one of the following conditions is met: before the processing process of the message corresponding to the data in the retry queue is stopped, the data quantity of the data in the retry queue is larger than or equal to a preset threshold value; the persistence processing refers to transferring data to a memory medium.
Optionally, the apparatus in this embodiment of the present application may further include: and the reporting module is used for reporting a first notification message to a third server after the data in the retry queue is subjected to persistence processing, wherein the first notification message is used for indicating that the writing operation is abnormal, and the third server is a server in the alarm platform.
Optionally, the apparatus in this embodiment of the present application may further include: and the deleting module is used for deleting the first data from the retry queue under the condition that the first processing operation executed again on the first data by the first thread is successful.
Optionally, the apparatus in this embodiment of the present application may further include: the execution module is used for re-executing the first processing operation before the first server writes the first data corresponding to the first message into the retry queue; the second processing module is used for writing the first data into the log under the condition that the first processing operation fails to be executed again, and storing the log to a magnetic disk; and the third processing module is used for counting the abnormal attributes of the logs in the disk and reporting a second notification message to the third server according to the counting result, wherein the second notification message is used for notifying that the writing operation is abnormal.
An embodiment of the present application further provides an electronic device, as shown in fig. 12, including a processor 1201, a communication interface 1202, a memory 1203, and a communication bus 1204, where the processor 1201, the communication interface 1202, and the memory 1203 complete communication with each other through the communication bus 1204,
a memory 1203 for storing a computer program;
the processor 1201 is configured to implement the following steps when executing the program stored in the memory 1203:
step 11, in case of a failure of executing a first processing operation corresponding to a first message, the first server writes first data corresponding to the first message into a retry queue, where the first message is sent from a second server that produces the message to the first server that consumes the message; the first processing operation is any one of a plurality of processing operations included in the processing process of the first message by the first server;
and step 12, the first server starts a first thread and re-executes the first processing operation on the first data in the retry queue through the first thread, wherein a plurality of processing operations are executed based on a second thread, and the first thread is different from the second thread.
The communication bus mentioned in the above terminal may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the terminal and other equipment.
The Memory may include a Random Access Memory (RAM) or a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the Integrated Circuit may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component.
In another embodiment provided by the present application, a computer-readable storage medium is further provided, which stores instructions that, when executed on a computer, cause the computer to execute the method for processing data described in any of the above embodiments.
In yet another embodiment provided by the present application, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform the method for processing data as described in any of the above embodiments.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the application to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
The above description is only for the preferred embodiment of the present application, and is not intended to limit the scope of the present application. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application are included in the protection scope of the present application.
Claims (10)
1. A method for processing data, comprising:
under the condition that execution of a first processing operation corresponding to a first message fails, a first server writes first data corresponding to the first message into a retry queue, wherein the first message is sent to a first server consuming messages by a second server producing messages; the first processing operation is any one of a plurality of processing operations included in the processing process of the first message by the first server;
the first server starts a first thread and re-executes the first processing operation on the first data in the retry queue through the first thread, wherein the plurality of processing operations are executed based on a second thread, and the first thread is different from the second thread.
2. The method of claim 1, wherein the writing, by the first server, the first data corresponding to the first message to a retry queue comprises:
the first server writes the first data and a timestamp corresponding to the first data into the retry queue, wherein the timestamp is an execution time of the first processing operation.
3. The method of claim 2, wherein prior to the first server re-executing the first processing operation on the first data by the first thread, the method further comprises:
in the case that the first data also corresponds to a second message, the first server determines an execution state of a second processing operation corresponding to the second message; wherein the result of the execution of the second processing operation is to override the result of the execution of the first processing operation; the execution time of the second processing operation is later than the execution time of the first processing operation;
under the condition that the execution of the second processing operation fails, the first server writes first data corresponding to the second message into the retry queue;
the first server prohibits starting the first thread if the second processing operation succeeds and the first processing operation has not been re-executed on the first data by the first thread.
4. The method of claim 1, further comprising:
the first server persists data in the retry queue on a condition that at least one of: before the processing process of the message corresponding to the data in the retry queue is stopped, the data amount of the data in the retry queue is greater than or equal to a preset threshold value;
wherein, the persistence processing refers to transferring the data to a memory medium.
5. The method of claim 4, wherein after the first server persists data in the retry queue, the method further comprises:
the first server reports a first notification message to a third server, wherein the first notification message is used for indicating that the writing operation is abnormal, and the third server is a server in an alarm platform.
6. The method of claim 1, further comprising:
in the event that re-execution of the first processing operation on the first data by the first thread is successful, the first server deletes the first data from the retry queue.
7. The method of claim 1, wherein before the first server writes the first data corresponding to the first message to a retry queue, the method further comprises:
the first server re-executes the first processing operation;
the first server writes the first data into a log under the condition that the first processing operation is failed to be executed again, and stores the log to a disk;
and the first server counts the abnormal attribute of the log in the disk, and reports a second notification message to a third server according to the statistical result, wherein the second notification message is used for notifying that the writing operation is abnormal.
8. A data processing device applied to a first server is characterized by comprising:
the device comprises a first writing module, a second writing module and a retry queue, wherein the first writing module is used for writing first data corresponding to a first message into the retry queue under the condition that the execution of a first processing operation corresponding to the first message fails, and the first message is sent to a first server consuming messages by a second server producing the messages; the first processing operation is any one of a plurality of processing operations included in the processing process of the first message by the first server;
a first processing module, configured to start a first thread and re-execute the first processing operation on the first data in the retry queue through the first thread, where the plurality of processing operations are executed based on a second thread, and the first thread is different from the second thread.
9. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any one of claims 1 to 7 when executing a program stored in the memory.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110475234.1A CN113176967B (en) | 2021-04-29 | 2021-04-29 | Data processing method and device, electronic equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110475234.1A CN113176967B (en) | 2021-04-29 | 2021-04-29 | Data processing method and device, electronic equipment and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113176967A true CN113176967A (en) | 2021-07-27 |
| CN113176967B CN113176967B (en) | 2023-08-15 |
Family
ID=76925315
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110475234.1A Active CN113176967B (en) | 2021-04-29 | 2021-04-29 | Data processing method and device, electronic equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113176967B (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110231702A1 (en) * | 2010-03-18 | 2011-09-22 | Microsoft Corporation | Coordinating communication medium state for subtasks |
| US8959530B1 (en) * | 2013-05-07 | 2015-02-17 | Sprint Communications Company L.P. | Messaging middleware processing thread count based events |
| US20150378796A1 (en) * | 2014-06-27 | 2015-12-31 | Amazon Technologies, Inc. | Client control in a distributed strict queue |
| CN107688503A (en) * | 2017-09-07 | 2018-02-13 | 北京奇艺世纪科技有限公司 | A kind of message treatment method based on ActiveMQ data/address bus, device and electronic equipment |
| US20180375783A1 (en) * | 2017-06-27 | 2018-12-27 | Atlassian Pty Ltd | Retry handling in messaging queues |
| CN109308214A (en) * | 2017-07-27 | 2019-02-05 | 北京京东尚科信息技术有限公司 | Data task processing method and system |
| CN112099935A (en) * | 2020-09-25 | 2020-12-18 | 北京奇艺世纪科技有限公司 | Task processing method and device |
| CN112148505A (en) * | 2020-09-18 | 2020-12-29 | 京东数字科技控股股份有限公司 | Data batching system, method, electronic device and storage medium |
| CN112328418A (en) * | 2020-11-27 | 2021-02-05 | 行吟信息科技（上海）有限公司 | Method and system for improving MQ synchronization reliability |
| CN112395097A (en) * | 2019-08-14 | 2021-02-23 | 北京京东尚科信息技术有限公司 | Message processing method, device, equipment and storage medium |
| CN112685145A (en) * | 2019-10-18 | 2021-04-20 | 上海哔哩哔哩科技有限公司 | Task management method and device and computer equipment |
-
2021
- 2021-04-29 CN CN202110475234.1A patent/CN113176967B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110231702A1 (en) * | 2010-03-18 | 2011-09-22 | Microsoft Corporation | Coordinating communication medium state for subtasks |
| US8959530B1 (en) * | 2013-05-07 | 2015-02-17 | Sprint Communications Company L.P. | Messaging middleware processing thread count based events |
| US20150378796A1 (en) * | 2014-06-27 | 2015-12-31 | Amazon Technologies, Inc. | Client control in a distributed strict queue |
| US20180375783A1 (en) * | 2017-06-27 | 2018-12-27 | Atlassian Pty Ltd | Retry handling in messaging queues |
| CN109308214A (en) * | 2017-07-27 | 2019-02-05 | 北京京东尚科信息技术有限公司 | Data task processing method and system |
| CN107688503A (en) * | 2017-09-07 | 2018-02-13 | 北京奇艺世纪科技有限公司 | A kind of message treatment method based on ActiveMQ data/address bus, device and electronic equipment |
| CN112395097A (en) * | 2019-08-14 | 2021-02-23 | 北京京东尚科信息技术有限公司 | Message processing method, device, equipment and storage medium |
| CN112685145A (en) * | 2019-10-18 | 2021-04-20 | 上海哔哩哔哩科技有限公司 | Task management method and device and computer equipment |
| CN112148505A (en) * | 2020-09-18 | 2020-12-29 | 京东数字科技控股股份有限公司 | Data batching system, method, electronic device and storage medium |
| CN112099935A (en) * | 2020-09-25 | 2020-12-18 | 北京奇艺世纪科技有限公司 | Task processing method and device |
| CN112328418A (en) * | 2020-11-27 | 2021-02-05 | 行吟信息科技（上海）有限公司 | Method and system for improving MQ synchronization reliability |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113176967B (en) | 2023-08-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107908494B (en) | Abnormal event processing method and device, electronic equipment and storage medium | |
| CN111585867B (en) | Message processing method and device, electronic equipment and readable storage medium | |
| CN108737132B (en) | Alarm information processing method and device | |
| CN108255620B (en) | Service logic processing method, device, service server and system | |
| CN112527472A (en) | Distributed transaction processing system and method based on microservice | |
| CN111327696A (en) | Data processing method and system | |
| US20210049087A1 (en) | Data monitoring methods, apparatuses, electronic devices, and computer readable storage media | |
| CN111010318A (en) | Method and system for discovering loss of connection of terminal equipment of Internet of things and equipment shadow server | |
| CN115328733A (en) | Alarm method and device applied to business system, electronic equipment and storage medium | |
| CN111782440A (en) | Data processing method and device and electronic equipment | |
| CN114138838A (en) | Data processing method and device, equipment and medium | |
| CN110955581A (en) | Online software abnormity warning method and device, electronic equipment and storage medium | |
| CN113326146A (en) | Message processing method and device, electronic equipment and storage medium | |
| CN110674153B (en) | Data consistency detection method and device and electronic equipment | |
| CN113176967A (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| CN110717130B (en) | Dotting method, dotting device, dotting terminal and storage medium | |
| CN112419018A (en) | General data reconciliation method, server and storage medium in distributed environment | |
| CN114896082A (en) | Message processing method and device, electronic equipment and storage medium | |
| CN115510036A (en) | Data migration method, device, equipment and storage medium | |
| CN112416731B (en) | Stability monitoring method and device applied to block chain system | |
| CN110633165B (en) | Fault processing method, device, system server and computer readable storage medium | |
| US11243979B1 (en) | Asynchronous propagation of database events | |
| CN110113187B (en) | Configuration updating method and device, configuration server and configuration system | |
| CN110362464B (en) | Software analysis method and equipment | |
| CN115344437A (en) | Disaster tolerance switching method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |