CN110941622A - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN110941622A CN110941622A CN201911055003.4A CN201911055003A CN110941622A CN 110941622 A CN110941622 A CN 110941622A CN 201911055003 A CN201911055003 A CN 201911055003A CN 110941622 A CN110941622 A CN 110941622A
- Authority
- CN
- China
- Prior art keywords
- message
- sent
- sending
- data
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/004—Error avoidance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/80—Database-specific techniques
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Computer Security & Cryptography (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a data processing method and a device, wherein the method comprises the following steps: acquiring a message to be sent and service data; the message to be sent is generated according to the service data; generating task information; the task information comprises a message to be sent; establishing an execution policy that associates a first operation with a second operation; and respectively executing the first operation and the second operation. The method can separate the message sending system from the service systems such as the sender system and the like, and can avoid the problems that the database transaction is not submitted for a long time due to overtime message sending, and the corresponding database connection is occupied and can not be released; when the access amount of the service system is large, the occupied database can not be released, so that the service request can not be connected with the database, the system availability is reduced at the system level, the operation failure occurs at the user level and the like; and then the database of the service system can be released in time, and the availability of the system is guaranteed.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data processing method and apparatus.
Background
In practical application, a message sender usually executes operations of writing a database and sending a message, and a message receiver processes the message after receiving the message; when the above operations are performed, it is necessary to ensure the consistency of the messages between the sender and the receiver, that is, both sides correctly send and receive the messages, and correctly process the service logic and the like according to the messages.
In order to ensure the consistency of the message, the operations of writing the data base and sending the message by the message sender are both successful, the correct operation of the service is ensured after the message receiver receives the message, the message is ensured not to be lost, the message receiver can receive the message certainly, and the receiver can only consume the same message once. A problem with any of the above processes can lead to inconsistencies.
In the process of implementing the invention, the inventor finds that when the service and the message belong to the same transaction, the following problems can exist:
if the message is overtime, the database transaction is not submitted for a long time, and the corresponding database connection is occupied and cannot be released; when the access amount of the service system is large, the service request of the user is not available for database connection, the system availability is reduced at the system level, and the user level has experience problems such as operation failure, retry request and the like.
In view of the technical problems in the related art, no effective solution is provided at present.
Disclosure of Invention
In order to solve the technical problem or at least partially solve the technical problem, the present application provides a data processing method and apparatus.
In a first aspect, an embodiment of the present application provides a data processing method, including:
acquiring a message to be sent and service data sent by a sender system; the message to be sent is generated according to the service data;
generating task information according to the message to be sent; wherein the task information comprises the message to be sent;
establishing an execution policy that associates a first operation with a second operation; the first operation comprises writing the business data into a database, the second operation comprises inserting the task information into a task table, and the execution strategy is used for enabling the first operation and the second operation to succeed or fail at the same time;
and respectively executing the first operation and the second operation.
Further, as in the foregoing data processing method, the executing the first operation and the second operation includes:
acquiring service processing result data in the service data;
judging whether the service data meet the writing requirements written into the database or not according to the service processing result data;
and when the service data meet the writing requirement, executing the first operation.
Further, as the foregoing data processing method, when the service data does not meet the write requirement, the method further includes:
rolling back an execution procedure when the second operation is executed.
In a second aspect, an embodiment of the present application provides a data processing method, including:
acquiring a message to be sent from a task table;
sending the message to be sent to a message consumer;
obtaining a sending result of sending the message to be sent to the message consumer;
and executing corresponding processing on the message to be sent according to the sending result.
Further, as in the aforementioned data processing method,
the acquiring the message to be sent from the task table includes:
scanning the task list according to a preset scanning frequency;
the sending the message to be sent to a message consumer includes:
and sending the message to be sent to the message consumer according to a preset sending frequency.
Further, as in the foregoing data processing method, the executing a corresponding processing policy on the message to be sent according to the sending result includes:
when the sending result represents that the message to be sent is sent successfully, stopping sending the message to be sent again;
when the sending result represents that the sending of the message to be sent fails and the retransmission times are within a preset task retry interval, retransmitting the message to be sent;
and when the sending result represents that the sending of the message to be sent fails and the retransmission times are not within a preset task retry interval, generating alarm information.
Further, as in the foregoing data processing method, the obtaining a sending result of sending the message to be sent to the message consumer includes:
acquiring message callback information sent by the message consumer; the message callback information is generated by the message consumer according to a callback field in the message to be sent, and is used for representing whether the message to be sent is successfully consumed by the message consumer;
determining a database routing field in the message callback information, and determining a first task table corresponding to the message callback information in a database according to the database routing field;
and changing the task state of the task information in the first task table according to the message callback information.
In a third aspect, an embodiment of the present application provides a data processing apparatus, including:
the acquisition module is used for acquiring a message to be sent and service data sent by a sender system; the message to be sent is generated according to the service data;
the generating module is used for generating task information according to the message to be sent; wherein the task information comprises the message to be sent;
a transaction module to establish an execution policy that associates a first operation with a second operation; the first operation comprises writing the business data into a database, the second operation comprises inserting the task information into a task table, and the execution strategy is used for enabling the first operation and the second operation to succeed or fail at the same time;
and the execution module is used for respectively executing the first operation and the second operation.
In a fourth aspect, an embodiment of the present application provides a data processing apparatus, including:
the first acquisition module is used for acquiring a message to be sent from the task table;
the sending module is used for sending the message to be sent to a message consumer;
the second acquisition module is used for acquiring a sending result of the message to be sent to the message consumer;
and the execution module is used for executing corresponding processing on the message to be sent according to the sending result.
In a fifth aspect, an embodiment of the present application provides a data processing system, including the data processing apparatus as described in the third aspect and the data processing apparatus as described in the fourth aspect.
In a sixth aspect, an embodiment of the present application provides an electronic device, including: the system comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are communicated with each other through the communication bus;
the memory is used for storing a computer program;
the processor is configured to implement the processing method according to any one of the first aspect and the second aspect when executing the computer program.
In a seventh aspect, this application embodiment provides a non-transitory computer-readable storage medium storing computer instructions that cause the computer to perform the processing method of any one of the first and second aspects.
The embodiment of the application provides a data processing method and a data processing device, wherein the data processing method comprises the following steps: acquiring a message to be sent and service data; the message to be sent is generated according to the service data; generating task information; wherein the task information comprises the message to be sent; establishing an execution policy that associates a first operation with a second operation; the first operation comprises writing the business data into a database, the second operation comprises inserting the task information into a task table, and the execution strategy is used for enabling the first operation and the second operation to succeed or fail at the same time; and respectively executing the first operation and the second operation. The method can separate the message sending system from the service systems such as the sender system and the like, and can avoid the problem that the database transaction is not submitted for a long time due to overtime message sending, so that the corresponding database connection can be occupied and can not be released; when the access amount of the service system is large, the occupied database can not be released, and finally the service request of the user can not have the database connection, so that the system availability is reduced on the system level, and the user level has experience problems of operation failure, retry request and the like; and then the database of the service system can be released in time, and the availability of the system is guaranteed.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
Fig. 1 is a schematic flowchart of a data processing method according to an embodiment of the present application;
fig. 2 is a schematic flowchart of another data processing method according to an embodiment of the present application;
fig. 3 is a schematic flowchart of another data processing method according to an embodiment of the present application;
fig. 4 is a schematic flowchart of another data processing method provided in the embodiment of the present application;
fig. 5 is a schematic flowchart of another data processing method according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of functional modules of a data processing apparatus according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of functional modules of another data processing apparatus according to an embodiment of the present application;
fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Fig. 1 is a data processing method according to an embodiment of the present application, including the following steps S11 to S14:
s11, obtaining a message to be sent and service data; the message to be sent is generated according to the service data;
specifically, the device for executing the method may be a database in a payment scenario, and a sender system of the message to be sent and the service data is generally a service system, for example: after the user performs online payment operation, the sender system needs to generate corresponding service data (including but not limited to payment time, payment amount, and payment merchant) and a message to be sent for notifying the user whether payment is successful (for example, a certain order is paid successfully) according to the payment operation; after the message to be sent and the service data are obtained, the service system needs to write the message to be sent and the service data into the database so as to realize the storage of the service data and the subsequent message sending;
s12, generating task information according to the message to be sent; the task information comprises a message to be sent;
specifically, the task information may be data for writing in a task table (schedule), and generally, the task table is established in a database, so that the task information is inserted into the task table, that is, the purpose of writing a message to be sent into the database is also achieved, and the task information is used for recording whether the message to be sent is successfully sent, and in addition, the task information in the task table can be actively changed according to a specific sending result according to a platform for sending the message to be sent; the task table is as follows: a tool for managing tasks, which can add tasks to a task list when in use; tasks needing to be processed immediately can be added into the task table, and the task state in the task table can be modified according to the actual execution result, for example, the tasks can be executed when being processed, and completed when the tasks are completed; optionally, the data stored in the task table includes: service type, task state, service data, database routing information, UUID duplication prevention, retry times and whether to call back fields.
The message to be sent can be a message to be sent to a user or a message required by other systems in the background of the service party (for example, the message to be sent needs to be acquired for storage or data processing, etc.), and the message to be sent is generally sent through a message sending platform; the task information comprises information such as a sending result of the corresponding message to be sent, so the task information comprises the message to be sent;
s13, establishing an execution strategy for associating the first operation with the second operation; the first operation comprises writing business data into a database, the second operation comprises inserting task information into a task table, and an execution strategy is used for enabling the first operation and the second operation to be successful or failed at the same time; optionally, the first operation and the second operation are set in the same transaction to implement the function corresponding to the execution policy in this step; furthermore, the first operation is also decoupled from the third operation; the third operation is to send a message to be sent; because the task table is established in the database, the task information is inserted into the task table, namely, the purpose of writing the message to be sent into the database is also realized;
specifically, in the prior art, the following problems may occur when the service data is written into the database and the message is sent in the same transaction:
1. if the message is sent overtime, the database transaction is not submitted for a long time, and the corresponding database connection is occupied and cannot be released; when the access amount of a service system is large, the service request of a user is not available in database connection, the system availability is reduced on a system level, and the user level has experience problems such as operation failure, retry request and the like;
2. when the message sending fails, if the message is captured abnormally, the service data cannot be synchronized to a downstream system, and the upstream and downstream data are inconsistent, if data compensation is performed through other modes, for example, abnormal information is recorded for data compensation, time and labor are consumed, and the final consistency of the upstream and downstream data cannot be completely guaranteed; if the exception is not captured, the service data is rolled back, and the return operation at the system level fails, so that although the consistency of upstream and downstream data can be ensured, the system availability and the user experience are influenced;
therefore, it is necessary to decouple the writing of the service data into the database (i.e. the first operation) and the sending of the message (i.e. the third operation) in order to solve the above technical problem;
however, it is also necessary to ensure that the service data and the message to be sent are simultaneously stored in the database to ensure the consistency of the data, and in order to achieve this purpose, the first operation and the second operation may be set in the same "Transaction". And the transaction: is a series of rigorous logical operations in a program, and all operations must be completed successfully, otherwise all changes made in each operation are undone; and the transaction has four characteristics as follows: 1. atomicity (Atomicity): when all instructions related to the same transaction are operated, all the instructions are executed successfully or not; if only one instruction fails to be executed, even if partial execution succeeds, rolling back the data which is successfully executed, judging that all instructions fail to be executed, and returning to the data state before the instructions are executed; 2. consistency (Consistency): the execution of a transaction causes the data to transition from one state to another, but the integrity of the entire data remains stable; 3. isolation (Isolation): the isolation is that when a plurality of users access the database concurrently, for example, when the same table is operated, the database is a transaction opened by each user and cannot be interfered by the operation of other transactions, and a plurality of concurrent transactions are isolated from each other; 4. durability (durabilty): when a transaction completes correctly, it is permanent to changes to the data. In the embodiment, the atomicity of the transaction is applied, so that the service data and the message to be sent are ensured to be written into the database at the same time.
S14, respectively executing a first operation and a second operation;
specifically, taking the example that the first operation and the second operation are set in the same "transaction", after the first operation and the second operation are executed respectively, according to the atomicity of the transaction, when one of the operations fails to be executed successfully, both the operations fail to be executed, and the data in the database is rolled back to the data state executed by each operation; wherein, whether the first operation is successfully executed is caused by a service execution result in the service data;
therefore, by the method in the embodiment, it is possible to realize: when the message is sent overtime, because the database affair is not submitted for a long time, the corresponding database connection is occupied and can not be released; when the access amount of the service system is large, the occupied database can not be released, and finally the service request of the user can not have the database connection, so that the system availability is reduced on the system level, and the user level has experience problems of operation failure, retry request and the like; and then the database of the service system can be released in time, and the availability of the system is guaranteed.
As shown in fig. 2, in some embodiments, as in the aforementioned data processing method, the first operation performed in step S14 includes the following steps S141 to S143:
s141, acquiring service processing result data in the service data;
specifically, the service processing result data is as follows: whether the service is successfully handled or executed, for example, when the service operation performed by the user is payment, the service processing result data is: whether the user successfully pays the deduction; when a user starts a payment process and fails to pay successfully due to network reasons, unwilling payment and other conditions in the final confirmation payment link, the corresponding execution service processing result is payment failure;
s142, judging whether the service data meet the writing requirements written into the database or not according to the service processing result data;
specifically, not all the service data need to be written into the database, whether the service data need to be written into the database can be matched through a preset writing requirement (rule), the writing requirement can be set according to the preset requirement, and the writing requirements corresponding to different service data types are generally different; the service data is only in accordance with the writing requirement (namely when the rule is matched) under the condition that the service processing result meets the requirement; for example, in the payment system, the database is only used for storing the payment success service data, and one optional determination method is as follows: judging through field information corresponding to the payment result, and writing in only when the field information corresponding to the payment result is success; therefore, in the generation of the service data a according to a payment behavior, if the field information corresponding to the payment result is success, the service data a meets the write-in requirement; in the generation of the service data B according to the other payment behavior, if the field information corresponding to the payment result is fail, the service data B does not meet the writing requirement;
s143, when the service data meet the writing requirement, executing a first operation;
that is, when the service data meets the predetermined writing requirement, the first operation is executed, so that writing of useless service data into the database can be avoided.
In some embodiments, as the foregoing data processing method, when the service data does not meet the writing requirement, the first operation is not executed, and the method further includes the following steps:
rolling back the execution process when the second operation is executed;
that is, when the service data does not meet the writing requirement, the judgment result is obtained, and the first operation is judged not to be executed according to the judgment result;
because the first operation and the second operation do not have a sequence when executed, there may be a case that the service data is determined not to meet the writing requirement after the second operation is written; in order to ensure the atomicity of the transaction, the process that the second operation has been executed needs to be rolled back, so that the data in the database is returned to the state that the second operation is not executed.
As shown in fig. 3, according to another embodiment of the present application, there is also provided a data processing method including the following steps S21 to S24:
s21, obtaining a message to be sent from a task table;
specifically, on the basis of the foregoing embodiment, the message to be sent may be actively obtained from the task table by the message sending platform that implements the method of this embodiment, or may be actively sent to the message sending platform by the task table;
s22, sending the message to be sent to a message consumer;
specifically, the message consumer is a receiver for receiving a message to be sent or a requester for actively requesting the message to be sent; for example: the message consumer can be a user side which receives the payment result after online payment, and can also be other systems which are positioned in the background of the same service side as the platform for sending the message to be sent; after receiving the message to be sent, the message consumer may perform a downstream RPC call (a way of processing data to a downstream system, such as short message content data) or update its own data according to the message to be sent, where the update of its own data may be, for example: when receiving a data notification from an external system, the service system can be temporarily stored in a task table of the service system, the task system scans an external data notification record in the task table and sends a message, the service system consumes the message and performs corresponding service processing and data updating, and thus the method has the advantages of facilitating service expansion of the service system, greatly promoting pressure reduction of the service system and the like;
s23, obtaining a sending result of a message to be sent by a targeted message consumer;
specifically, the sending result, that is, whether the message to be sent is successfully consumed by the message consumer, may be used to determine whether the message needs to be sent to the message consumer again, and the sending result may include: successful or failed transmission; if the transmission is successful, repeated transmission is not needed, and if the transmission is failed, the transmission is needed again until the message to be transmitted is successfully received by the message consumer; furthermore, different messages to be sent are provided with different duplication prevention fields, so that a message consumer can perform idempotent consumption on the messages to be sent according to the duplication prevention fields, namely, the situation of repeated consumption is prevented, and the problem of repeated writing of data or repeated sending of information is avoided, thereby affecting the system performance and the user experience;
s24, executing corresponding processing on the message to be sent according to the sending result;
specifically, the processing corresponding to different sending results is generally different, and the processing strategy can be set according to the preset requirement, as long as the message to be sent can be guaranteed to be successfully consumed.
In some embodiments, a data processing method, as previously described,
step S21 is to obtain a message to be sent from the task table, including:
scanning the task list according to a preset scanning frequency;
specifically, the message to be sent is obtained by scanning the task table actively; and the preset scanning frequency can be a frequency set by people, such as: once per second, 10 scans per second, etc.; if the message to be sent is obtained through scanning, the fact that the scanning party actively obtains the message to be sent means; thus, the tasks of messaging may all be performed by a server, such as a messaging platform;
sending a message to be sent to a message consumer, comprising:
sending the message to be sent to a message consumer according to a preset sending frequency;
that is to say, the frequency when sending the message is also controllable, and the message is sent at a fixed sending frequency, so that the problem of interface congestion can be effectively alleviated under the condition of high concurrent message sending requirements, and the processing efficiency of the system can be improved.
As shown in fig. 4, in some embodiments, as in the foregoing data processing method, the step S24 executes a corresponding processing policy on the message to be sent according to the sending result, including the following steps S241 to S243:
s241, when the sending result represents that the message to be sent is sent successfully, stopping sending the message to be sent again;
s242, when the sending result represents that the sending of the message to be sent fails and the retransmission times are within a preset task retry interval, retransmitting the message to be sent;
s243, when the sending result represents that the sending of the message to be sent fails and the retransmission times are not within a preset task retry interval, generating alarm information;
specifically, in the application, the message can be retransmitted through the message middleware, wherein the message middleware is a message queue. The message retry mechanism is a message compensation mechanism implemented by the message queue, and when a problem occurs in the message consumption process of the consumer (for example, calling a downstream interface is overtime or fails), the message queue can push the message to the message consumer again by using the compensation mechanism of the message queue, that is, the retry mechanism, and the consumer can receive the message again to perform corresponding service processing until the consumer successfully consumes the message, so that the message can be guaranteed to be correctly consumed;
the embodiment of the application further includes step S243, when the message queue is used for sending for multiple times (exceeding the number of task retries), the sending is not performed any more, and an alarm is performed instead, so that the sending of other messages in the message queue can be prevented from being influenced, and the problem that the message cannot be sent is judged and checked by changing the alarm to manual operation.
As shown in fig. 5, in some embodiments, as in the foregoing data processing method, the obtaining of the sending result of the message to be sent to the message consumer in step S23 includes the following steps S231 to S233:
s231, message callback information sent by a message consumer is acquired; the message callback information is generated by a message consumer according to a callback field in a message to be sent, and is used for representing whether the message to be sent is successfully consumed by the message consumer;
specifically, one method for acquiring the message callback information may be: the message body of the message to be sent comprises a field of 'whether to call back information', namely isCallback, a message consumer judges the isCallback after successful consumption, if the isCallback is 1, the isCallback needs to be called back, the message consumer updates the state of the corresponding message record in the task table of the service library to be successful consumption through a task system such as the message sending platform, and the like, and does not update when the consumption is failed. The consumer can also realize the failure retry mechanism of self consumption, and can retry by relying on the task system strategy (for example, after the message record in the task table is successfully sent, if the message record is not changed into the consumption success in a long time, the message is tried to be sent again for retrying)
S232, determining a database routing field in the message callback information, and determining a first task table corresponding to the message callback information in a database according to the database routing field;
generally, many service systems need to be accessed to the task system, each service system has its own task table, a database routing field (i.e. a database routing key) is the task table for distinguishing different service systems, and the corresponding message to be sent can be found and stored in which task table of which service system library is stored through the database routing field; in addition, the database routing field can exist in the message to be sent, so that when the message callback information is generated, the database routing field can be taken out and written into the message callback information, and then the first task table corresponding to the message callback information can be obtained according to the field positioning of the database route;
s233, changing the task state of the task information in the first task table according to the message callback information;
specifically, the updating of the task table data in the database is to update the corresponding message record state in the task table according to the processing result corresponding to the message callback information after the task system finishes processing the data, for example: after a certain message is successfully sent to a corresponding message consumer, the task system updates the state of the corresponding message in the task table to be sent, and when the message is not successfully sent, the task system updates the state of the corresponding message in the task table to be failed in sending, and the other states are changed in the same way.
As shown in fig. 6, according to another embodiment of the present application, there is also provided a data processing apparatus including:
an obtaining module 1, configured to obtain a message to be sent and service data; the message to be sent is generated according to the service data;
the generating module 2 is used for generating task information according to the message to be sent; the task information comprises a message to be sent;
a transaction module 3 for establishing an execution policy associating a first operation with a second operation; the first operation comprises writing business data into a database, the second operation comprises inserting task information into a task table, and an execution strategy is used for enabling the first operation and the second operation to be successful or failed at the same time;
and the execution module 4 is used for executing the first operation and the second operation.
Specifically, the specific process of each module in the apparatus according to the embodiment of the present invention to implement its function may refer to the related description in the corresponding method embodiment, and is not described herein again.
As shown in fig. 7, according to another embodiment of the present application, there is also provided a data processing apparatus including:
a first obtaining module 5, configured to obtain a message to be sent from a task table;
the sending module 6 is used for sending the message to be sent to the message consumer;
a second obtaining module 7, configured to obtain a sending result of sending a message to be sent to a message consumer;
and the execution module 8 is configured to execute corresponding processing on the message to be sent according to the sending result.
Specifically, the specific process of each module in the apparatus according to the embodiment of the present invention to implement its function may refer to the related description in the corresponding method embodiment, and is not described herein again.
According to another embodiment of the present application, there is also provided a data processing system, which includes the data processing apparatus in the foregoing embodiment shown in fig. 6 and the data processing apparatus in the foregoing embodiment shown in fig. 7.
The system can solve the problem that the service data and the information to be sent are simultaneously put in storage, ensure the success of other operations such as message sending and the like, realize the real decoupling of the service operation and the remote RPC and message sending, shorten the response time of the service operation and ensure the final consistency of the data.
Specifically, the specific process of implementing the functions of each module in the apparatus according to the embodiment of the present invention may refer to the related description in the method embodiment, and is not described herein again.
According to another embodiment of the present application, there is also provided an electronic apparatus including: as shown in fig. 8, the electronic device may include: the system comprises a processor 1501, a communication interface 1502, a memory 1503 and a communication bus 1504, wherein the processor 1501, the communication interface 1502 and the memory 1503 complete communication with each other through the communication bus 1504.
A memory 1503 for storing a computer program;
the processor 1501 is configured to implement the steps of the above-described method embodiments when executing the program stored in the memory 1503.
The bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the electronic equipment and other equipment.
The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.
Embodiments of the present application also provide a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the steps of the above-described method embodiments.
It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The foregoing are merely exemplary embodiments of the present invention, which enable those skilled in the art to understand or practice the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (12)
1. A data processing method, comprising:
acquiring a message to be sent and service data; the message to be sent is generated according to the service data;
generating task information; wherein the task information comprises the message to be sent;
establishing an execution policy that associates a first operation with a second operation; the first operation comprises writing the business data into a database, the second operation comprises inserting the task information into a task table, and the execution strategy is used for enabling the first operation and the second operation to succeed or fail at the same time;
and executing the first operation and the second operation.
2. The data processing method of claim 1, wherein the performing the first operation comprises:
acquiring service processing result data in the service data;
judging whether the service data meet the writing requirements written into the database or not according to the service processing result data;
and when the service data meet the writing requirement, executing the first operation.
3. The data processing method according to claim 2, further comprising, when the service data does not meet the write requirement:
rolling back an execution procedure when the second operation is executed.
4. A data processing method, comprising:
acquiring a message to be sent from a task table;
sending the message to be sent to a message consumer;
obtaining a sending result of sending the message to be sent to the message consumer;
and executing corresponding processing on the message to be sent according to the sending result.
5. The data processing method of claim 4,
the acquiring the message to be sent from the task table includes:
scanning the task table according to a preset scanning frequency;
the sending the message to be sent to a message consumer includes:
and sending the message to be sent to the message consumer according to a preset sending frequency.
6. The data processing method according to claim 4, wherein the performing a corresponding processing policy on the message to be sent according to the sending result comprises:
when the sending result represents that the message to be sent is sent successfully, stopping sending the message to be sent again;
when the sending result represents that the sending of the message to be sent fails and the retransmission times are within a preset task retry interval, retransmitting the message to be sent;
and when the sending result represents that the sending of the message to be sent fails and the retransmission times are not within a preset task retry interval, generating alarm information.
7. The data processing method of claim 4, wherein the obtaining of the transmission result of the message to be transmitted to the message consumer comprises:
acquiring message callback information sent by the message consumer; the message callback information is generated by the message consumer according to a callback field in the message to be sent, and is used for representing whether the message to be sent is successfully consumed by the message consumer;
determining a database routing field in the message callback information, and determining a first task table corresponding to the message callback information in a database according to the database routing field;
and changing the task state of the task information in the first task table according to the message callback information.
8. A data processing apparatus, comprising:
the acquisition module is used for acquiring a message to be sent and service data sent by a sender system; the service data comprises the message to be sent;
the generating module is used for generating task information according to the message to be sent; the message to be sent is generated according to the service data;
a transaction module to establish an execution policy that associates a first operation with a second operation; the first operation comprises writing the business data into a database, the second operation comprises inserting the task information into a task table, and the execution strategy is used for enabling the first operation and the second operation to succeed or fail at the same time;
and the execution module is used for respectively executing the first operation and the second operation.
9. A data processing apparatus, comprising:
the first acquisition module is used for acquiring a message to be sent from the task table;
the sending module is used for sending the message to be sent to a message consumer;
the second acquisition module is used for acquiring a sending result of the message to be sent to the message consumer;
and the execution module is used for executing corresponding processing on the message to be sent according to the sending result.
10. A data processing system comprising a data processing apparatus as claimed in claim 8 and a data processing apparatus as claimed in claim 9.
11. An electronic device, comprising: the system comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are communicated with each other through the communication bus;
the memory is used for storing a computer program;
the processor, when executing the computer program, implementing the processing method of any one of claims 1-7.
12. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the processing method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911055003.4A CN110941622A (en) | 2019-10-31 | 2019-10-31 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911055003.4A CN110941622A (en) | 2019-10-31 | 2019-10-31 | Data processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110941622A true CN110941622A (en) | 2020-03-31 |
Family
ID=69906397
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911055003.4A Pending CN110941622A (en) | 2019-10-31 | 2019-10-31 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110941622A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113190559A (en) * | 2021-05-21 | 2021-07-30 | 湖南快乐阳光互动娱乐传媒有限公司 | Service data retry method and related equipment |
| CN115174515A (en) * | 2022-07-07 | 2022-10-11 | 北京科创汇捷科技发展有限公司 | Message distribution method based on file persistence |
| CN115412208A (en) * | 2022-08-09 | 2022-11-29 | 唯阜德(武汉)科技有限公司 | Message transmission method and system for message queue management system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106155775A (en) * | 2015-04-28 | 2016-11-23 | 阿里巴巴集团控股有限公司 | Message treatment method, equipment and system |
| CN108965457A (en) * | 2018-08-02 | 2018-12-07 | 郑州云海信息技术有限公司 | A kind of message delivery method of distributed cluster system, device, equipment and medium |
| CN110049113A (en) * | 2019-04-02 | 2019-07-23 | 中国联合网络通信集团有限公司 | Service message processing method and device |
| CN110147287A (en) * | 2019-04-24 | 2019-08-20 | 珠海市珠澳跨境工业区好易通科技有限公司 | A kind of message queue receive-transmit system and method |
| CN111835467A (en) * | 2020-07-28 | 2020-10-27 | 中国平安财产保险股份有限公司 | Message sending method, device, computer equipment and storage medium |
-
2019
- 2019-10-31 CN CN201911055003.4A patent/CN110941622A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106155775A (en) * | 2015-04-28 | 2016-11-23 | 阿里巴巴集团控股有限公司 | Message treatment method, equipment and system |
| CN108965457A (en) * | 2018-08-02 | 2018-12-07 | 郑州云海信息技术有限公司 | A kind of message delivery method of distributed cluster system, device, equipment and medium |
| CN110049113A (en) * | 2019-04-02 | 2019-07-23 | 中国联合网络通信集团有限公司 | Service message processing method and device |
| CN110147287A (en) * | 2019-04-24 | 2019-08-20 | 珠海市珠澳跨境工业区好易通科技有限公司 | A kind of message queue receive-transmit system and method |
| CN111835467A (en) * | 2020-07-28 | 2020-10-27 | 中国平安财产保险股份有限公司 | Message sending method, device, computer equipment and storage medium |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113190559A (en) * | 2021-05-21 | 2021-07-30 | 湖南快乐阳光互动娱乐传媒有限公司 | Service data retry method and related equipment |
| CN115174515A (en) * | 2022-07-07 | 2022-10-11 | 北京科创汇捷科技发展有限公司 | Message distribution method based on file persistence |
| CN115412208A (en) * | 2022-08-09 | 2022-11-29 | 唯阜德(武汉)科技有限公司 | Message transmission method and system for message queue management system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111666162B (en) | Distributed message transmission method, device, computer equipment and storage medium | |
| CN110941622A (en) | Data processing method and device | |
| CN105814542B (en) | Method and apparatus for distributed transactions in a data communications network | |
| US20120011100A1 (en) | Snapshot acquisition processing technique | |
| CN111768288B (en) | Service processing method, device, electronic equipment and storage medium | |
| CN111813791A (en) | Method and equipment for distributed transaction compensation | |
| CN111897825A (en) | Distributed transaction processing method and device | |
| CN110018884B (en) | Distributed transaction processing method, coordination device, database and electronic equipment | |
| CN111667334B (en) | Audit failure order processing method and device, computer equipment and storage medium | |
| CN110888718A (en) | Method and device for realizing distributed transaction | |
| CN112446786A (en) | Abnormal transaction processing method and device, electronic equipment and readable storage medium | |
| CN108647105B (en) | Idempotent control method, device and system in system switching process | |
| CN113268471A (en) | Method, proxy connection pool, system, device, and medium for processing distributed transactions | |
| CN112561506B (en) | Live broadcast data processing method, system, equipment and medium based on virtual currency | |
| CN117762652A (en) | Distributed transaction processing method and device based on message middleware | |
| CN111367694B (en) | Event processing method, server and computer storage medium | |
| CN111159298B (en) | Service request processing method and device, electronic equipment and storage medium | |
| CN110674153B (en) | Data consistency detection method and device and electronic equipment | |
| CN109245941B (en) | Service compensation method and device | |
| CN111274047A (en) | Information processing method, terminal, system, computer device and storage medium | |
| CN115544034A (en) | Data consistency method and service system | |
| JPH11161530A (en) | Transaction processing system | |
| CN112181686A (en) | Data processing method, device and system, electronic equipment and storage medium | |
| CN117011058A (en) | Cross-system transaction method, device, equipment and storage medium | |
| CN114327792A (en) | Transaction processing method, system and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: Room 221, 2 / F, block C, 18 Kechuang 11th Street, Daxing District, Beijing, 100176 Applicant after: Jingdong Technology Holding Co.,Ltd. Address before: Room 221, 2 / F, block C, 18 Kechuang 11th Street, Daxing District, Beijing, 100176 Applicant before: Jingdong Digital Technology Holding Co.,Ltd. Address after: Room 221, 2 / F, block C, 18 Kechuang 11th Street, Daxing District, Beijing, 100176 Applicant after: Jingdong Digital Technology Holding Co.,Ltd. Address before: Room 221, 2 / F, block C, 18 Kechuang 11th Street, Daxing District, Beijing, 100176 Applicant before: JINGDONG DIGITAL TECHNOLOGY HOLDINGS Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200331 |
|
| RJ01 | Rejection of invention patent application after publication |