CN108459919B

CN108459919B - Distributed transaction processing method and device

Info

Publication number: CN108459919B
Application number: CN201810273667.7A
Authority: CN
Inventors: 周北春; 朱清沂; 周竣涛; 孟德君
Original assignee: CITIC Aibank Corp Ltd
Current assignee: CITIC Aibank Corp Ltd
Priority date: 2018-03-29
Filing date: 2018-03-29
Publication date: 2022-04-15
Anticipated expiration: 2038-03-29
Also published as: CN108459919A

Abstract

The invention belongs to the technical field of distributed computing, and discloses a distributed transaction processing method and device. The method comprises the steps of application development, transaction management attribute definition, event definition, streaming calculation and transaction management, wherein the steps of event information acquisition and identification are carried out in real time through the streaming calculation, the transaction management attribute is acquired in real time, the transaction state is verified periodically according to the event information and the transaction management attribute, a checking request and/or a correction request are/is initiated according to the transaction state, and a transaction state checking instruction and/or a correction instruction is sent through a transaction manager. According to the scheme, the transaction control and the application development are completely decoupled, the application development does not need to develop transaction control logic, the development difficulty is greatly simplified, the transaction can be quickly rolled back in an abnormal scene, the fault of any node is tolerated, and the problem of split brain does not exist.

Description

Distributed transaction processing method and device

Technical Field

The invention belongs to the technical field of distributed computing, relates to a distributed transaction processing method and device, and particularly relates to a distributed transaction processing method and device based on stream computing.

Background

Interpretation of related terms:

transaction (Transaction): generally to what is to be done or what is to be done. In computer terminology, refers to a program execution unit (unit) that accesses and possibly updates various data items in a database. A Transaction (Transaction) is a unit of program execution (unit) that accesses and possibly updates various data items in a database. Transactions are typically caused by the execution of user programs written in a high level database manipulation language or programming language (e.g., SQL, C + +, or Java) and are bounded in the form of begin transactions and end transactions statements (or function calls). A transaction consists of an ensemble of operations performed between a transaction start (begin transaction) and a transaction end (end transaction).

Transaction Processing (Transaction Processing): in many large, critical applications, computers are performing a large number of tasks per second. More often than not the tasks themselves, but the tasks are combined to complete a business requirement, called a transaction. What will happen if a task can be successfully executed and an error occurs in the second or third related task? This error is likely to leave the system in an inconsistent state. The transaction becomes very important at this point, which enables the system to escape from this inconsistent state. Products such as CICS, Tuxedo, and TopEnd are examples of transaction processing systems that provide transaction services for applications.

Streaming Computing (streaming Computing): in a conventional data processing flow, data is always collected first and then put into a Database (DB). When people need to do query (query) on data through the DB, the answer is obtained or relevant processing is carried out, and the method is called batch calculation. Thus, although it seems reasonable, the results are very compact and, especially in some real-time search application environments, some specific problems are not solved well by offline processing in a manner similar to MapReduce. This leads to a new data computation architecture, a stream computation approach, that can analyze large-scale streaming data well in real-time during the changing motion, capture potentially useful information, and send the results to the next compute node.

Unlike batch calculation, which accumulates data slowly, streaming calculation spreads a large amount of data at each time point, continuously transmits small batches, and the data continuously flows and is discarded after calculation. Batch computation is the maintenance of a data table that implements various computational logics. In contrast, streaming computing must first define the computing logic to submit to the streaming computing system, which is not modifiable throughout its run. On the calculation result, the result is transmitted after the batch calculation is carried out on all the data, and the streaming calculation is that the result can be immediately delivered to an online system after each small batch calculation, so that real-time display is realized.

Event drive (Event drive): event-driven is a strategy for making decisions in the process of continuous transaction management, namely following events occurring at the current time point, transferring available resources and executing related tasks, so that the problems which occur continuously are solved, and transaction accumulation is prevented. The method has application in the fields of computer programming, public relations, economic activities and the like.

ACID model: strong consistent transaction models, including Atomicity (Atomicity), Consistency (Consistency), Isolation (Isolation), and persistence (Durability).

Atomicity refers to all operations in the entire transaction, either all completed or not completed, and not likely to stall in the middle of a link. The transaction is subject to an error during execution and is rolled back (Rollback) to a pre-transaction state as if the transaction had never been executed.

Coherency means that a transaction can encapsulate state changes (unless it is read-only). Transactions must always keep the system in a consistent state regardless of how many concurrent transactions are at any given time.

Isolation refers to the isolation state to execute transactions so that they appear to be the only operation performed by the system in a given time, i.e. when multiple transactions access concurrently, the transactions are isolated from each other, and one transaction should not affect the operation of the other transactions.

Persistence refers to changes made to the database by a transaction that are persisted in the database and not rolled back after the transaction is completed.

BASE model: the final consistent transaction model is a shorthand for three phrases, basic Available, Soft state, and Eventually consistency. The core idea of BASE is: even if Strong consistency cannot be achieved, each application can adopt a proper mode to enable the system to achieve final consistency according to the service characteristics of the application. Basic availability means that the distributed system is allowed to lose part of its availability, i.e. to guarantee that the core is available, when it fails. When the e-commerce is big, in order to deal with the surge of the access amount, part of users may be guided to the degraded page, and the service layer may only provide the degraded service, which is the embodiment of losing part of the availability. Soft state refers to allowing an intermediate state to exist for the system without the intermediate state affecting the overall availability of the system. Generally, one copy of data in distributed storage has at least three copies, and the delay of copy synchronization between different nodes is the embodiment of a soft state. The final consistency means that all data copies in the system can finally reach a consistent state after a certain time.

CAP theory: the method refers to that Consistency, Availability and Partition tolerance cannot be compatible in a distributed system. Where three characteristics in the distributed system are defined as follows:

identity (C): all data backups in the distributed system are judged whether to have the same value at the same time;

availability (a): after a part of nodes in the cluster fail, whether the whole cluster can respond to the read-write request of the client side is judged;

partition fault tolerance (P): in practical terms, a partition corresponds to a time limit requirement for communication. If the system fails to achieve data consistency within the time limit, meaning that a partition has occurred, a tradeoff must be made between C and a for the current operation.

Two Phase Commit (Two Phase Commit, 2 PC): an Algorithm (Algorithm) designed based on the consistency of all nodes under the distributed system architecture in transaction commit. In a distributed system, each node can know success or failure of its own operation, but cannot know success or failure of other node operations. When a transaction spans a plurality of nodes, in order to maintain the ACID characteristic of the transaction, a component serving as a coordinator needs to be introduced to uniformly master the operation results of all nodes (called participants) and finally indicate whether the nodes need to actually commit the operation results (such as writing updated data to a disk and the like). Therefore, the algorithm idea submitted in two phases can be summarized as follows: the participants inform the coordinator of the success or failure of the operation, and the coordinator determines whether each participant needs to submit the operation or suspend the operation according to the feedback information of all the participants.

Two-Phase Lock (Two Phase Lock, 2 PL): the two-phase lock protocol means that the execution of each transaction can be divided into two phases: a growth phase (lock phase) and a decay phase (unlock phase).

The locking phase at this stage a locking operation may be performed. An S-lock is applied and acquired prior to a read operation on any data and an X-lock is applied and acquired prior to a write operation. If the locking is not successful, the transaction enters a waiting state, and the execution is not continued until the locking is successful.

The unlocking phase after the transaction releases a lockout, the transaction enters the unlocking phase, in which only unlocking operations can be performed and locking operations can not be performed.

The two-stage locking method can be realized by the following steps: the transaction begins in a locking phase until roll-back (ROLLBACK) and COMMIT (COMMIT) are performed. ROLLBACK (ROLLBACK) and COMMIT (COMMIT) cause the transaction to enter an unlock phase, i.e., the database management system (DBMS) releases all blockages in the ROLLBACK (ROLLBACK) and COMMIT (COMMIT) modules.

The core of the Transaction Processing (TP) is to ensure data consistency in the transaction processing process through transaction isolation, atomic commit or rollback, persistence processing and the like. Under a traditional centralized IT architecture, transaction control generally encapsulates transactions through transaction middleware, and the underlying technology mainly adopts two-phase commit (2PC) and two-phase lock (2PL) to ensure the ACID characteristics of the transactions, so that data is strong and consistent, that is, after an update operation is completed, any access of a plurality of subsequent processes or threads returns the latest updated value. This is the most user friendly, i.e. what the user has written last and what it can be guaranteed to read next. But this implementation has a large impact on performance.

However, with the rapid development of distributed technologies, cloud computing technologies based on cluster management of X86 are beginning to emerge and are widely used in industries such as the internet. Cloud computing has many characteristics such as low cost, open technology, and strong expansion capability, and because of these characteristics, some financial institutions have also started exploring in the field of cloud computing, for example, deploying non-accounting applications or query applications on a cloud platform. But distributed also brings some challenges, such as how to perform distributed transaction control in a distributed cluster. The main technical scheme at the present stage comprises the following 3 types:

the first scheme is as follows: using middleware to follow a 2 PC-based transaction control mechanism

Many IT vendors provide a lighter-weight middleware system for distributed X86 clusters. The scheme is used more in traditional financial enterprises.

Scheme II: abandoning middleware, guaranteeing transaction ACID by application

Some internet enterprises of financial nature have proposed application-level solutions that replace middleware, with applications ensuring the ACID of transactions, and in particular consistency. The most prevalent application development mode is the TCC mode. The TCC mode refers to a development mode submitted by an application implementation in two phases, and comprises three implementation logics:

the TRYING stage mainly detects a service system and reserves resources;

the configuration phase is a confirmation commit to the service system, and when the TRYING phase is successfully executed and the configuration phase is started to be executed, the default configuration phase is error-free. Namely: as long as TRYING succeeds, CONFIRMING succeeds;

the CANCELING stage is mainly to cancel the service and release the reserved resource when the service is executed in error and needs to be rolled back.

The configuration and CANCELING phases are in parallel, and after the TRYING phase, either the configuration phase or CANCELING phase is entered. TCC requires application support idempotency. Idempotency means that the execution of a business method call once is the same as the execution of multiple calls returns the result.

The third scheme is as follows: compensation patterns achieve Final consistency (BASE)

BASE is a new consistency control model proposed by Ebay in distributed practice, based on CAP theory to properly sacrifice consistency (C) in exchange for availability (a). BASE is widely used in the distributed field and is a popular technology of choice for cloud computing. The 2PC solution under the BASE model is completely abandoned in order to achieve final consistency in the transaction. The most typical BASE solution is the compensation mode.

The compensation mode requires the application to provide forward transaction and forward-rushing transaction, the transaction flow is defined through the service flow mode, the state of the transaction flow needs to be put in a warehouse in real time, and once the service flow is abnormal, the main control program initiates reverse-rushing. That is, if a wrong transaction is recovered, such as drawing 1000 but the money is not drawn and the balance is low, the 1000 transaction blocks are automatically flushed back to recover the balance.

However, all of the above three solutions have their own technical drawbacks, as follows

The first defect of the scheme is as follows: the main drawback of the 2PC scheme is that once a transaction Coordinator (Coordinator) has a system abnormality, the whole transaction control process has a split brain problem. A transaction in a split brain must wait until the transaction Coordinator (Coordinator) recovers and issues a Commit (Commit) or Rollback (Rollback) instruction to complete the transaction. However, because the transaction processing process adopts a two-stage lock, the lock is in a locked state for a long time in the process of no submission or rollback, and further other parallel transactions are blocked. If a split brain transaction happens to hold a lock for a hot table, the entire IT system may face total unavailability. This scheme is only suitable for scenarios where strong data consistency is required, but where the system is not available. To balance system unavailability and consistency, the 2PC may use empirical exception handling (heartbeat) to allow the transaction Participant (Participant) to take a pre-defined transaction operation (typically a default rollback) after waiting a period of time (Timeout), but this will result in data inconsistency requiring manual intervention. Distributed technologies typically abandon the 2PC solution in real market competition.

The second scheme has the defects that: the main problem of TCC is that the application system is too severely damaged, application developers are required to pay attention to not only service implementation but also design consistency control and exception handling, and resource reservation in the TRYING phase causes relatively large loss to the system under the abnormal condition, which limits the overall throughput. The TCC development mode has certain application in Internet enterprises with higher technical level, but the application is invasive and cannot be widely popularized in general.

The three defects of the scheme are as follows: the compensation mode is the mainstream choice of the distributed architecture at present. The main problems of this solution are:

1. the application still designs and realizes the correction type logic to realize the correction of the abnormal scene

2. When the conflict occurs, the orthogonality is already submitted for a period of time, and the conflict is likely to fail due to a long time difference, so that data inconsistency is caused. And the orthogonal conflict based on program call is easy to be abnormal, which causes the positive conflict failure

Disclosure of Invention

In order to overcome the problems in the prior art, the invention provides a distributed transaction processing method and a distributed transaction processing device.

The invention specifically discloses a distributed transaction processing method, which comprises the following steps:

setting an application: arranging a plurality of applications required for realizing transaction processing in a distributed system, wherein each application is provided with a local transaction data source in the distributed system so as to realize local data storage and update;

defining the transaction management attribute: defining transaction management attributes according to the requirements of transaction processing;

event definition: defining a plurality of specific state changes in the transaction processing process as events;

the streaming calculation step: acquiring and identifying event information in real time through stream computing, acquiring transaction management attributes in real time, periodically verifying the transaction state according to the event information and the transaction management attributes, and initiating a checking request and/or a correcting request according to the transaction state;

in an embodiment of the method of the present invention, which is used in banking, the forward request may be a rollback request of a transaction state change record.

And (3) transaction management: and the transaction manager sends a transaction state verification instruction to each application according to the verification request and/or sends a transaction forward instruction to a transaction data source of each application according to the forward request.

As a preferred mode, in the distributed transaction processing method of the present invention, in the step of setting the application, the application includes a distributed transaction coordinator and a plurality of distributed transaction participants, where the distributed transaction coordinator is an initiator and a submitter of the distributed transaction, and opens the distributed transaction by means of a transaction API, an annotation, or a transaction definition file; the distributed transaction coordinator and the distributed transaction participants realize remote calling of the downstream transaction participants through the transaction ID. Among them, Transaction API, such as Begin Transaction; annotations, such as @ Transacational; a transaction definition file, such as XML.

As a preferred mode, in the distributed transaction processing method of the present invention, the transaction ID includes a Global transaction ID (UOW _ Global _ ID), a Local transaction ID (UOW _ Local _ ID), and a branch number (Span _ ID), the Global transaction IDs of the applications are the same, the branch number of the distributed transaction coordinator is default to 0, the branch number is gradually increased according to the upstream and downstream relationship between the applications, and the distributed transaction participant generates the Local transaction ID based on the Global transaction ID (UOW _ ID) and the branch number (Span _ ID), and transmits the Local transaction ID to the downstream distributed transaction participant. The hierarchical transaction ID is the unique identification of the global transaction, so that the atomicity control of the transaction is formed, meanwhile, the update sequence of the transaction is recorded, and a basis is provided for the reversal of the sequence.

Preferably, the step of defining the transaction management attributes is performed in the application and/or in the transaction manager, and the transaction management attributes comprise one or more of transaction name, transaction propagation attribute, transaction isolation attribute, timeout setting, rollback control, commit control, read-only configuration, unknown state policy. Wherein, the transaction propagation attribute is provided, such as whether to start a new transaction or inherit the original transaction context; transaction isolation properties such as uncommitted reads, committed reads, repeatable reads, and serialization; rollback control, such as under what kind of anomaly rollback occurs; commit control, if any exceptions can be ignored.

Preferably, in the step of event definition, the event includes one or more of a transaction start event, a transaction end event, an exception event, an application calling event, an application called event, an application persistence layer event, and a transaction data source persistence event; wherein:

the information of the transaction start event comprises one or more of transaction ID, application information and time information;

the information of the transaction end event comprises one or more of transaction ID, application information, time information and transaction result;

the abnormal event refers to a transaction running state which is not captured by the application to process the transaction result;

the information of the application calling event comprises one or more of transaction ID, caller application information, callee application information and time information;

the information of the application called event comprises one or more of transaction ID, caller application information, callee application information and time information;

the information of the application persistence layer event comprises one or more of transaction ID and session information;

the information of the transaction data source persistence event comprises one or more of session information and a persistence log.

Preferably, in the step of event definition, a transaction log corresponding to each event is generated by applying a buried point technology or a communication bypass technology, and information of the event is acquired and identified in real time by the streaming calculation step. Wherein, a point embedding technology is applied, for example, a slice-oriented programming technology is used, slices are designed, and tangent points and slice actions are applied to form service logic non-invasive data acquisition; communication bypass techniques, such as network card capture of feature transmission messages.

Preferably, the Streaming computation is implemented by one or more Streaming computation platforms of Apache Storm, Spark Streaming, Flink, flash, Kafka. Apache Storm, Spark Streaming and Flink can be used for real-time calculation, Flume can be used for data acquisition, and Kafka can be used for message circulation. In particular, Apache Storm, Spark Streaming and Flink are mainstream computing frameworks in the industry, flash is commonly used for log collection, and Kafka is widely used as a data pipeline. The stream type calculation realizes the second-level or even millisecond-level real-time calculation of the hashed transaction information under the distributed environment, and meets the requirement of OLTP on transaction real-time performance.

Preferably, the step of streaming computation is implemented by:

the method comprises the steps of collecting event information in real time, identifying a transaction opening event, obtaining transaction management attributes from an application or a transaction manager, caching all transaction persistent information in the transaction processing process, and periodically verifying the transaction state, and specifically comprises the following steps:

1) if the distributed transaction coordinator commits the transaction, confirming the completion conditions of the transaction data sources in all the applications through stream computing so as to complete the transaction submission; if the confirmation result is completion, clearing the cache data; otherwise, sending a checking request to the transaction processor;

2) if the distributed transaction coordinator rolls back the transaction, confirming that the transaction data sources in all the applications complete the roll-back operation through stream type calculation: if the data is confirmed, clearing the cache data; otherwise, sending a rollback request to the transaction manager, and initiating a rollback instruction to the corresponding transaction data source by the transaction manager;

3) if the distributed transaction coordinator does not initiate submission or rollback in the transaction timeout period, the transaction enters an unknown state, an unknown state strategy in the transaction management attribute is judged through stream type calculation, if the strategy is rollback, the transaction source state in each application is checked through stream type calculation, a rollback request is generated for a transaction data source which is not rolled back, the rollback request is sent to a transaction manager, and the transaction manager initiates the rollback to the transaction data source which is not rolled back; and if the policy is commit, checking the commit status of the transaction data source of each application through streaming calculation, and if an uncommitted transaction exists, sending a checking request to the transaction manager.

In a preferred embodiment of the distributed transaction processing method according to the present invention, the execution of the rollback instruction operates in reverse order according to the branch number (Span _ ID) in the local transaction ID. And the correctness and the success rate of forward flushing operation are ensured according to the reverse sequence forward flushing of the transaction updating sequence.

In a preferred embodiment of the distributed transaction processing method according to the present invention, the method further includes the step of alarming: and when the transaction state fed back by the streaming calculation is an abnormal event, the transaction manager sends a transaction state verification instruction to each application, and when the feedback result of the streaming calculation result is still the abnormal event, the transaction manager sends an operation and maintenance alarm instruction.

The invention also provides a distributed transaction processing device, which comprises:

a distributed application module: the system comprises a plurality of applications which are distributed in a distributed system and used for transaction processing, and local transaction data sources corresponding to the applications in the distributed system so as to realize the storage and the update of the local data of the applications;

an event definition module: the event log is used for generating a transaction log corresponding to each event, wherein the event is a specific state change in the transaction processing process;

the transaction management module: the system comprises a stream type calculation module, a transaction state verification request sending module, a transaction data source sending module and a transaction data processing module, wherein the stream type calculation module is used for sending a transaction state verification request to each application according to the verification request sent by the stream type calculation module, and/or sending a transaction forward-looking command to the transaction data source of each application according to the forward-looking request sent by the stream type calculation module;

defining transaction management attributes in the applications and/or transaction management modules;

a streaming computation module: the system is used for acquiring and identifying the information of the event in real time, acquiring the transaction management attribute in real time, periodically verifying the transaction state, feeding back the transaction state to the transaction manager according to the transaction state, and initiating a checking request and/or a correcting request.

As a preferred embodiment, the distributed transaction processing apparatus may further include an alarm module, configured to send operation and maintenance alarm information according to an alarm instruction sent by the transaction management module, when the transaction state fed back by the streaming calculation module is an abnormal event, the transaction manager sends a transaction state verification instruction to each application, and when the result fed back by the streaming calculation result is still an abnormal event, the transaction manager sends an operation and maintenance alarm instruction.

In a distributed application module, the application comprises a distributed transaction coordinator and a plurality of distributed transaction participants, wherein the distributed transaction coordinator is an initiator and a submitter of the distributed transaction and opens the distributed transaction by means of a transaction API, an annotation or a transaction definition file; the distributed transaction coordinator and the distributed transaction participants realize remote calling of the downstream transaction participants through the transaction ID.

Preferably, the transaction ID includes a Global transaction ID (UOW _ Global _ ID), a Local transaction ID (UOW _ Local _ ID), and a branch number (Span _ ID), the Global transaction IDs of the applications are the same, the branch number of the distributed transaction coordinator is 0 by default, the branch number is gradually increased according to the upstream and downstream relationship between the applications, and the distributed transaction participant generates the Local transaction ID based on the Global transaction ID (UOW _ ID) and the branch number (Span _ ID) and transmits the Local transaction ID to the downstream distributed transaction participant.

Preferably, in the event definition module, the event includes one or more of a transaction start event, a transaction end event, an exception event, an application calling event, an application called event, an application persistence layer event, and a transaction data source persistence event; wherein:

As a preferred embodiment, the event definition module is composed of a slicing program, generates a transaction log corresponding to each event through the slicing program, and acquires the transaction log in real time through stream computing.

Preferably, the transaction management attribute comprises one or more of a transaction name, a transaction propagation attribute, a transaction isolation attribute, a timeout setting, a rollback control, a commit control, a read-only configuration, and an unknown state policy.

Preferably, the Streaming computation module is implemented by one or more Streaming computation platforms of Apache Storm, Spark Streaming, Flink, flash, Kafka.

The technical scheme of the invention adopts a stream type computing technology and an event-driven concept, and discloses a distributed transaction processing technology which is non-invasive to application and has strong and consistent data accuracy; the data are quasi-strong and consistent, and once the atomic operation has program exception, the data are quickly rolled back for updating, so that the atomicity is ensured; the data operation of the distributed nodes does not need to hold locks in time, and high concurrency and high throughput are guaranteed. The technical scheme of the invention realizes the rapid development of the application and simultaneously solves the problem of transaction consistency puzzling the distributed environment.

Specifically, compared with three common transaction processing methods in the prior art, the technical solution of the present invention has at least the following advantages:

1. compared with the second and third technical schemes in the prior art, the transaction control and the application development are completely decoupled in the technical scheme, an application developer only needs to care about service logic and does not need to design any distributed transaction control logic, for example, orthogonal operation is easy to greatly simplify the development difficulty, the development efficiency can be effectively improved, and the labor cost is reduced.

2. Compared with the second and third technical schemes in the prior art, the technical scheme of the invention realizes real-time calculation of transaction control through stream type calculation, ensures the whole submission of the transaction, realizes quick rollback of the transaction in an abnormal scene, ensures the atomicity of the transaction and the consistency of data accuracy, and quickly updates rollback data once the atomic operation has program abnormality.

3. Compared with the first technical scheme, the technical scheme of the invention adopts a distributed design, tolerates the fault of any node, has no brain crack problem, does not need to hold a lock in time for data operation of the distributed nodes, and ensures high concurrency and high throughput.

4. The application level and the transaction manager define transaction management attributes in combination with the requirements of transaction processing, and transaction control in a flexible hot loading mode is achieved.

Drawings

FIG. 1 is an exemplary diagram of an atomic transaction scenario;

FIG. 2 is a sequence diagram of an atomic transaction persistence operation;

FIG. 3 is a flow diagram illustrating triggering atomic transaction rollback due to an application exception;

FIG. 4 is a diagram of the overall logic structure of the distributed transaction processing method of the present invention;

FIG. 5 is a logical block diagram of a distributed transaction processing party embodiment of the present invention;

FIG. 6 is an exemplary diagram of a distributed transaction coordinator annotation code in accordance with an embodiment of the present invention;

FIG. 7 is an exemplary diagram of transaction management attribute definition code according to an embodiment of the present invention;

FIG. 8 is a tree diagram of transaction ID invocation according to an embodiment of the present invention;

FIG. 9 is a diagram illustrating an association relationship between a data source log and a transaction ID according to an embodiment of the present invention.

Detailed Description

In order to make the technical scheme of the invention easier to understand, the technical scheme of the invention is clearly and completely described by adopting a mode of a specific embodiment in combination with the attached drawings. It should be noted that the embodiments described herein are only some embodiments of the present invention, and not all implementations of the present invention, and the embodiments are only examples, which only serve to provide examiners and the public with a more intuitive and clear understanding of the present invention, and do not limit the technical solutions of the present invention. All other embodiments, as well as other simple substitutions and various changes to the technical solutions of the present invention, which can be made by those skilled in the art without inventive work, are within the scope of the present invention without departing from the spirit of the present invention.

The following takes bank transaction processing as an example to specifically describe the technical scheme of the invention.

As shown in an exemplary diagram of an atomic transaction scenario in fig. 1, for example, an IT system of 1 enterprise is composed of 4 application modules, a request sent by a client enters a production system through a channel layer (e.g., an API gateway), an application a serves as a front-end application to process a user request, operate a relational database, and then call an application B; b, after the application runs the processing logic, persistently processing the data and putting the data into a database, and then calling the application C; c, returning the database operation by the application; the application B further calls the application D; d, returning the database operation by the application; b, returning the application to A by the application; the a application records the customer information in KV store and then successfully returns to the customer. So in this atomic request, 4 application components have modified 4 databases and 1 key value store, respectively, in the order shown in FIG. 2. These modifications are the persisted results of atomic transaction commit (the D property in the ACID).

As shown in fig. 3, the atomicity of the transaction requires that once an exception occurs during the operation of the transaction, triggering the rollback of the transaction, all the stateful changes in the transaction process need to be rolled back to the state before the transaction starts. For example, in the above example, if the exception occurs during the calling process of the D application by the B application to trigger rollback, the changes that have been made and belong to the current transaction all need to be rolled back, including database operation 1 of the a application, database operation 2 of the B application, and database operation 3 of the C application. For the customer, a rollback transaction does not happen as if it were anything, for example, the customer transfers 1000 to another account, and if the transaction fails, the amount of the customer's own account cannot be reduced, and the amount of the transfer account cannot be increased.

In a plurality of transaction processing modes in the prior art, a scheme I is mainly applied to a centralized framework, under the traditional expensive centralized framework, the stability of hardware is very high, four application systems run in a commercial-grade mature middleware, and transaction control is realized through the scheme I (two stages); however, for the distributed cloud platform, the capability and stability of hardware individuals are poor, the frequency of occurrence of hardware faults and software defects is high, and a split brain phenomenon easily occurs in the first scheme, for example, once an application A is used as a dispatcher in two stages, a base table related to transaction processing is locked for a long time, so that the system is not available; the second scheme has the problems that an application system needs to provide three T/C/C realization interfaces to replace a middleware to realize transaction control, the development difficulty is high, the period is long, and the quality cannot be ensured; the third scheme still has application control transaction, but compared with the second scheme, the third scheme properly reduces the difficulty of transaction control, abandons the transaction consistency and pursues final consistency.

The present invention specifically discloses a distributed transaction processing method, as shown in fig. 4 and fig. 5, including:

setting an application: developing a plurality of applications required for realizing transaction processing in a distributed system, wherein each application is provided with a local transaction data source in the distributed system so as to realize local data storage and update;

in this embodiment, the applications are the initiator and the submitter of the distributed transaction, and the application can further refine the distributed transaction coordinator (e.g., application a in fig. 4) and the distributed transaction participants (e.g., applications B, C, D in the example of fig. 4). The distributed transaction coordinator may open the distributed transaction by means of a transaction API, an annotation, or a transaction definition file, for example, an annotation written in Java code (the present invention is not limited to Java implementation), and the annotation of the distributed transaction coordinator is shown in fig. 6.

Application A is the entry and coordinator of the distributed transaction, and @ Transactional is a Java general transaction declaration mode used for developing the transaction. The distributed transaction coordinator can also define transaction management attributes such as timeout information, rollback information, transaction identification, unknown state policy, etc. by a program, and the specific implementation mode can be implemented according to the method shown in fig. 7.

The transaction management attribute can also be defined in a transaction manager, so that more flexible transaction management is realized.

As shown in fig. 5, after the Transaction is started, a Transaction Context (Transaction Context) is formed in the application Context (application Context) of the application a, and a Global Transaction ID (UOW _ Global _ ID) is included in the Transaction Context in order to ensure the Global property. The UOW _ id will serve as a global identification of the distributed transaction and be propagated to downstream transaction participants. Meanwhile, the transaction coordinator is also provided with a Local transaction ID (UOW _ Local _ ID), and the Local transaction ID is based on the global transaction ID (UOW _ ID) and the branch number (Span _ ID, default increment). The Span _ id of the transaction coordinator is 0 by default. Each time the remote call is made, the transaction participant generates a local transaction ID based on the global transaction ID (UOW _ ID) and the branch number (Span _ ID), and transmits the local transaction ID to the downstream transaction participant, and the specific call manner is as shown in fig. 8. With this mechanism, the transaction ID is rotated upstream and downstream of the application call link, and the call order can be identified by Span _ ID.

As shown in FIG. 9, persistent updates generated by atomic transactions may be implemented by application-local transaction data sources, such as CRUD operations on relational databases, updates to key-value stores, and the like. For the application local transaction data source, each update generates a log file, and records the data change of the update, such as Binlog data of the MySQL database. The method associates the transaction ID with the log of the data source layer through a data access layer (DAO), and identifies each application local data source update log of the distributed transaction. All local data operations in the scheme are submitted locally, and data resources are released.

Defining the transaction management attribute: defining transaction management attributes according to the requirements of transaction processing; the transaction management attribute definition may be performed in the application layer or in the transaction manager. The transaction management attribute comprises one or more of a transaction name, a transaction propagation attribute, a transaction isolation attribute, a timeout setting, a rollback control, a commit control, a read-only configuration, and an unknown state policy.

Event definition: defining a plurality of specific state changes in the transaction processing process as events; as shown in table 1:

table 1: (example of event type and Collection information in the embodiments of the invention)

The event comprises one or more of a transaction start event, a transaction end event, an abnormal event, an application calling event, an application called event, an application persistence layer event and a transaction data source persistence event; wherein:

The information of each event can be used for acquiring and identifying the information of the event in real time in a streaming calculation step in a mode of generating a transaction log corresponding to each event by applying a buried point technology or a communication bypass technology.

The event forms an application log by applying a slice point burying mode, the data source generates an update log, and then the zero invasion of transaction information acquisition to application development is realized by acquiring through a streaming technology.

The streaming calculation step: the method comprises the steps of acquiring and identifying event information in real time through stream computing, acquiring transaction management attributes in real time, periodically verifying transaction states according to the event information and the transaction management attributes, feeding back the transaction states to a transaction manager according to the transaction states, and initiating a checking request and/or a correcting request. In an embodiment of the method of the present invention, which is used in banking, the forward request may be a rollback request of a transaction state change record.

The stream computing is realized by one or more stream computing platforms in Apache Storm, Spark Streaming, Flink, flash and Kafka. Apache Storm, Spark Streaming and Flink can be used for real-time calculation, Flume can be used for data acquisition, and Kafka can be used for message circulation.

As shown in fig. 4 and 5, the step of streaming computation is implemented by:

stream computing is used for establishing real-time acquisition event information, identifying a transaction opening event, acquiring a transaction management attribute from an application or a transaction manager, caching all transaction persistent information in the transaction processing process, and periodically verifying the transaction state according to the event information and the transaction management attribute, wherein the method specifically comprises the following steps:

In a preferred embodiment of the distributed transaction processing method according to the present invention, the execution of the rollback instruction operates in reverse order according to the branch number (Span _ ID) in the local transaction ID.

Another function of the transaction manager is to initiate a direct rollback instruction to the transaction data source according to the rollback instruction and rollback content captured by streaming, for example, if the forward update identified by parsing Binlog of MySQL as an INSERT instruction, the rollback instruction is to INSERT a record to perform DELETE operation. That is, the transaction manager completes the reverse operation that was in the flushing application. Execution of the rollback instruction operates in reverse order with the Span _ ID in the local transaction ID.

In a preferred embodiment of the distributed transaction processing method according to the present invention, the method further includes the step of alarming: and when the transaction state fed back by the streaming calculation is an abnormal event, the transaction manager sends a transaction state verification instruction to each application, and when the feedback result of the streaming calculation result is still the abnormal event, the transaction manager sends an operation and maintenance alarm instruction. If the stream type submission check is abnormal, the transaction manager performs secondary check and gives an operation and maintenance alarm according to the condition.

the transaction management module: the system comprises a stream type calculation module, a transaction state verification request sending module and a transaction forward-flushing request sending module, wherein the stream type calculation module is used for sending a transaction state verification instruction to each application according to the verification request sent by the stream type calculation module and/or sending a transaction forward-flushing instruction to each application according to the forward-flushing request sent by the stream type calculation module;

Claims

1. A distributed transaction processing method, comprising:

setting an application: arranging a plurality of applications required for realizing transaction processing in a distributed system, wherein each application is provided with a local transaction data source in the distributed system;

event definition: defining a number of specific state changes in a transaction as events;

the streaming calculation step: the method comprises the steps of acquiring and identifying event information in real time through stream computing, acquiring transaction management attributes in real time, periodically verifying transaction states according to the event information and the transaction management attributes, and initiating a checking request and/or a correction request according to the transaction states;

2. The distributed transaction processing method of claim 1, wherein: in the step of setting the application, the application comprises a distributed transaction coordinator and a plurality of distributed transaction participants, wherein the distributed transaction coordinator is an initiator and a submitter of the distributed transaction and opens the distributed transaction by means of a transaction API, an annotation or a transaction definition file; the distributed transaction coordinator and each distributed transaction participant realize remote calling of downstream transaction participants through transaction IDs.

3. The distributed transaction processing method of claim 2, wherein: the transaction ID comprises a Global transaction ID (UOW _ Global _ ID), a Local transaction ID (UOW _ Local _ ID) and a branch number (Span _ ID), the Global transaction IDs of all applications are the same, the branch number of the distributed transaction coordinator is defaulted to 0, the branch number is gradually increased according to the upstream and downstream relation among the applications, and the distributed transaction participants generate the Local transaction ID based on the Global transaction ID (UOW _ ID) and the branch number (Span _ ID) and transmit the Local transaction ID to the downstream distributed transaction participants.

4. A distributed transaction processing method according to claim 3, characterized in that: the step of defining the transaction management attribute is performed in each application and/or in the transaction manager, and the transaction management attribute comprises one or more of a transaction name, a transaction propagation attribute, a transaction isolation attribute, a timeout setting, rollback control, commit control, read-only configuration, and an unknown state policy.

5. The distributed transaction processing method of claim 4, wherein: in the step of event definition, the event comprises one or more of a transaction start event, a transaction end event, an abnormal event, an application calling event, an application called event, an application persistence layer event and a transaction data source persistence event; wherein:

6. The distributed transaction processing method of claim 5, wherein: in the step of event definition, a transaction log corresponding to each event is generated by applying a buried point technology or a communication bypass technology, and information of the event is acquired and identified in real time in the flow calculation step.

7. The distributed transaction processing method of claim 1, wherein: the stream computing is realized by one or more stream computing platforms in Apachesterm, spark streaming, Flink, flash, Kafka.

8. The distributed transaction processing method of claim 5, wherein: the step of streaming computation is implemented by:

collecting event information in real time, identifying a transaction opening event, acquiring a transaction management attribute from an application or a transaction manager, caching all transaction persistent information in the transaction processing process, and periodically verifying the transaction state:

9. The distributed transaction processing method of claim 8, wherein: the execution of the rollback instruction operates in reverse order according to the branch number (Span _ ID) in the local transaction ID.

10. A distributed transaction method according to any of claims 1-9, further comprising the step of alerting: and when the transaction state fed back by the streaming calculation is an abnormal event, the transaction manager sends a transaction state verification instruction to each application, and when the feedback result of the streaming calculation result is still the abnormal event, the transaction manager sends an operation and maintenance alarm instruction.

11. A distributed transaction processing apparatus, comprising:

a distributed application module: the system comprises a plurality of applications which are distributed in a distributed system and used for transaction processing, and local transaction data sources corresponding to the applications in the distributed system;

12. The distributed transaction processing apparatus of claim 11, wherein:

the system also comprises an alarm module used for sending operation and maintenance alarm information according to the alarm instruction sent by the transaction management module, when the transaction state fed back by the stream type calculation module is an abnormal event, the transaction manager sends a transaction state verification instruction to each application, and when the result fed back by the stream type calculation module is still an abnormal event, the transaction manager sends an operation and maintenance alarm instruction.

13. The distributed transaction processing apparatus of claim 11, wherein: in a distributed application module, the application comprises a distributed transaction coordinator and a plurality of distributed transaction participants, wherein the distributed transaction coordinator is an initiator and a submitter of the distributed transaction and opens the distributed transaction in a transaction API, annotation or transaction definition file mode; the distributed transaction coordinator and the distributed transaction participants realize remote calling of the downstream transaction participants through the transaction ID.

14. The distributed transaction processing apparatus of claim 13, wherein: the transaction ID comprises a Global transaction ID (UOW _ Global _ ID), a Local transaction ID (UOW _ Local _ ID) and a branch number (Span _ ID), the Global transaction IDs of all applications are the same, the branch number of the distributed transaction coordinator is defaulted to 0, the branch number is gradually increased according to the upstream and downstream relation among the applications, and the distributed transaction participants generate the Local transaction ID based on the Global transaction ID (UOW _ ID) and the branch number (Span _ ID) and transmit the Local transaction ID to the downstream distributed transaction participants.

15. The distributed transaction processing apparatus of claim 14, wherein: in the event definition module, the event comprises one or more of a transaction start event, a transaction end event, an abnormal event, an application calling event, an application called event, an application persistence layer event and a transaction data source persistence event; wherein:

16. The distributed transaction processing apparatus of claim 15, wherein: the event definition module is composed of a slicing program, generates a transaction log corresponding to each event through the slicing program, and obtains the transaction log in real time through stream computing.

17. The distributed transaction processing apparatus of claim 11, wherein: the transaction management attribute comprises one or more of a transaction name, a transaction propagation attribute, a transaction isolation attribute, a timeout setting, a rollback control, a commit control, a read-only configuration, and an unknown state policy.

18. The distributed transaction processing apparatus of claim 11, wherein: the stream computing is realized by one or more stream computing platforms in Apachesterm, spark streaming, Flink, flash, Kafka.