WO2023077971A1 - 事务处理方法、装置、计算设备及存储介质 - Google Patents
事务处理方法、装置、计算设备及存储介质 Download PDFInfo
- Publication number
- WO2023077971A1 WO2023077971A1 PCT/CN2022/118941 CN2022118941W WO2023077971A1 WO 2023077971 A1 WO2023077971 A1 WO 2023077971A1 CN 2022118941 W CN2022118941 W CN 2022118941W WO 2023077971 A1 WO2023077971 A1 WO 2023077971A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- data
- time
- analysis result
- transaction
- state data
- Prior art date
Links
- 238000003860 storage Methods 0.000 title claims abstract description 149
- 238000003672 processing method Methods 0.000 title claims abstract description 24
- 238000004458 analytical method Methods 0.000 claims abstract description 464
- 230000008859 change Effects 0.000 claims abstract description 104
- 238000012545 processing Methods 0.000 claims abstract description 93
- 238000000034 method Methods 0.000 claims description 81
- 238000012986 modification Methods 0.000 claims description 31
- 230000004048 modification Effects 0.000 claims description 31
- 238000004590 computer program Methods 0.000 claims description 22
- 230000015654 memory Effects 0.000 claims description 20
- 238000010223 real-time analysis Methods 0.000 abstract description 57
- 238000004364 calculation method Methods 0.000 description 102
- 239000010410 layer Substances 0.000 description 59
- 230000008569 process Effects 0.000 description 47
- 230000006870 function Effects 0.000 description 44
- 238000005192 partition Methods 0.000 description 39
- 238000004422 calculation algorithm Methods 0.000 description 34
- 238000007726 management method Methods 0.000 description 31
- 238000010586 diagram Methods 0.000 description 25
- 230000007704 transition Effects 0.000 description 20
- 238000005516 engineering process Methods 0.000 description 19
- 230000007246 mechanism Effects 0.000 description 12
- 238000013500 data storage Methods 0.000 description 9
- 230000002123 temporal effect Effects 0.000 description 9
- 238000007405 data analysis Methods 0.000 description 7
- 230000000694 effects Effects 0.000 description 7
- 238000012360 testing method Methods 0.000 description 7
- 230000002093 peripheral effect Effects 0.000 description 6
- 239000002356 single layer Substances 0.000 description 5
- 238000013519 translation Methods 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 230000000977 initiatory effect Effects 0.000 description 4
- 238000003780 insertion Methods 0.000 description 4
- 230000037431 insertion Effects 0.000 description 4
- 238000013508 migration Methods 0.000 description 4
- 230000005012 migration Effects 0.000 description 4
- 238000013473 artificial intelligence Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000007689 inspection Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 235000019633 pungent taste Nutrition 0.000 description 3
- 238000012935 Averaging Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 230000000903 blocking effect Effects 0.000 description 2
- 230000003111 delayed effect Effects 0.000 description 2
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000002688 persistence Effects 0.000 description 2
- 230000002085 persistent effect Effects 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 230000032683 aging Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000007429 general method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
- 238000012384 transportation and delivery Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2379—Updates performed during online database operations; commit processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2465—Query processing support for facilitating data mining operations in structured databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/219—Managing data history or versioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2474—Sequence data queries, e.g. querying versioned data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2477—Temporal data queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Definitions
- the present application relates to the technical field of databases, in particular to a transaction processing method, device, computing device and storage medium.
- HTAP Hybrid Transaction/Analytical Processing
- the HTAP system includes OLTP (On-Line Transaction Processing, online transaction processing) clusters and OLAP (On-Line Analytical Processing, online analysis and processing) clusters.
- OLTP clusters are used to process short-term user transaction transactions, and have a short transaction execution time. , high concurrency, and high throughput requirements per unit time;
- OLAP clusters are used to process complex analytical transactions, and have the characteristics of long transaction execution time, large resource occupation, and large amount of processed data.
- a data migration module is configured between the OLAP cluster and the OLTP cluster to migrate the latest relevant data in the OLTP cluster to the OLAP cluster to ensure the consistency of data analysis and processing between the OLAP cluster and the OLTP cluster
- the HTAP system provided by related technologies only supports online transaction and analysis functions for the latest data, and has poor analysis and processing capabilities.
- the embodiment of the present application provides a transaction processing method, device, computing device, and storage medium, which can not be limited to only provide analysis functions for the latest data, but provide Online real-time analysis and processing services improve the analysis and processing capabilities.
- the technical solution is as follows:
- a transaction processing method performed by a computing device, the method comprising:
- the first moment is the moment when the historical state data is dumped the last time, and the historical state data is data that meets the data query conditions of the transaction to be analyzed ;
- the full analysis result is a result obtained by processing the historical state data based on the analysis semantics of the transaction to be analyzed;
- the specific change is a change that occurs between the first moment and the end of the query time period
- the incremental analysis result is a result obtained by processing the data obtained by the specific change based on the analysis semantics
- the final analysis result of the transaction to be analyzed is obtained.
- a transaction processing apparatus is provided, the apparatus is deployed on a computing device, and the apparatus includes:
- a determining module configured to determine a first moment from the query time period of the transaction to be analyzed, the first moment is the moment when the historical state data is dumped the last time, and the historical state data is consistent with the transaction to be analyzed
- a first acquisition module configured to acquire a full analysis result based on the historical state data, where the full analysis result is a result obtained by processing the historical state data based on the analysis semantics of the transaction to be analyzed;
- the second acquisition module is configured to acquire incremental analysis results based on a specific change that occurs on the current state data corresponding to the historical state data, the specific change being the change from the first moment to the query time period
- the incremental analysis result is the result of processing the data obtained from the specific change based on the analysis semantics
- a third acquiring module configured to acquire the final analysis result of the transaction to be analyzed based on the full analysis result and the incremental analysis result.
- a computing device in one aspect, includes one or more processors and one or more memories, at least one computer program is stored in the one or more memories, and the at least one computer program is loaded and executed by the one or more processors to realize the above-mentioned A transaction method for any of the possible implementations.
- a storage medium is provided, and at least one computer program is stored in the storage medium, and the at least one computer program is loaded and executed by a processor to implement the transaction processing method in any possible implementation manner above.
- a computer program product or computer program comprising one or more pieces of program code stored in a computer-readable storage medium.
- One or more processors of the computing device can read the one or more pieces of program code from the computer-readable storage medium, and the one or more processors execute the one or more pieces of program code, so that the computing device can Execute the transaction processing method in any one of the foregoing possible implementation manners.
- a full analysis result is calculated for the historical state data before the first moment, and specific changes after the first moment
- the obtained data calculates an incremental analysis result, combines the full analysis result and the incremental analysis result, and finally obtains the final analysis result, so that for any query time period on the full-temporal time axis, it can provide online Real-time analysis and processing services are not limited to only providing analysis functions for the latest data, thus greatly improving the transaction processing performance of database systems such as HTAP systems or HTASP systems.
- Fig. 1 is the implementation environment architecture diagram of a kind of HTAP system that the embodiment of the application provides;
- FIG. 2 is a logical block diagram of a transaction processing flow provided by an embodiment of the present application
- FIG. 3 is a schematic diagram of an implementation environment of a transaction processing method provided in an embodiment of the present application.
- FIG. 4 is a flow chart of a transaction processing method provided by an embodiment of the present application.
- Fig. 5 is a schematic diagram of the relationship between historical state data and cold, hot and temperature data provided by the embodiment of the present application;
- FIG. 6 is a schematic diagram of adding a real-time analysis slave copy in a master-slave copy mechanism provided by an embodiment of the present application
- Fig. 7 is a schematic diagram of a cold and hot data storage model provided by the embodiment of the present application.
- FIG. 8 is a schematic diagram of a storage form of full-time data records provided by the embodiment of the present application.
- FIG. 9 is a flow chart of a transaction processing method provided by an embodiment of the present application.
- Fig. 10 is a schematic flow chart of a real-time analysis model provided by the embodiment of the present application.
- FIG. 11 is a schematic diagram of the storage form of data records of a streaming computing model provided by the embodiment of the present application.
- FIG. 12 is an implementation environment architecture diagram of a HTASP system provided by the embodiment of the present application.
- Fig. 13 is a statistical diagram of the access volume ratio and data volume distribution of hot and cold data provided by the embodiment of the present application.
- Fig. 14 is a schematic structural diagram of a transaction processing device provided by an embodiment of the present application.
- Fig. 15 is a schematic structural diagram of a computing device provided by an embodiment of the present application.
- FIG. 16 is a schematic structural diagram of a computing device provided by an embodiment of the present application.
- first and second are used to distinguish the same or similar items with basically the same function and function. It should be understood that “first”, “second” and “nth” There are no logical or timing dependencies, nor are there restrictions on quantity or order of execution.
- the term "at least one" means one or more, and the meaning of "multiple” means two or more, for example, a plurality of first positions means two or more first positions.
- Cloud Technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network in a wide area network or a local area network to realize data calculation, storage, processing, and sharing, that is, a business model based on cloud computing
- the general term for applied network technology, information technology, integration technology, management platform technology, application technology, etc. can form a resource pool, which can be used on demand and is flexible and convenient.
- Cloud computing technology will become an important support in the field of cloud technology.
- the background services of technical network systems require a lot of computing and storage resources, such as video websites, picture websites and more portal websites. With the rapid development and application of the Internet industry, each item may have its own identification mark in the future, which needs to be transmitted to the background system for logical processing. Data of different levels will be processed separately, and all kinds of industry data need to be powerful.
- the system backing support can be realized through cloud computing.
- Cloud Storage It is a new concept extended and developed from the concept of cloud computing.
- Distributed cloud storage system (hereinafter referred to as storage system) refers to the functions of cluster application, grid technology and distributed storage file system.
- a storage system that integrates a large number of different types of storage devices (storage devices are also called storage nodes) in the network to work together through application software or application interfaces, and jointly provide data storage and service access functions.
- Database In short, it can be regarded as an electronic filing cabinet—a place where electronic files are stored. Users can add, query, update, and delete data in files.
- database is a collection of data that is stored together in a certain way, can be shared with multiple users, has as little redundancy as possible, and is independent of the application program.
- Computing equipment Computing nodes or computing engine nodes, which mainly complete the computing functions of the SQL (Structured Query Language) layer, and can also be called SQL Engines (SQL Engine).
- SQL Structured Query Language
- a computing device refers to a database (usually a cluster database, for a stand-alone database, a stand-alone device itself is a computing device) used to process user-specific computing requests (referred to as user requests), and is mainly responsible for executing user requests Node devices.
- Storage device storage node or storage engine node (Storage Engine), which mainly completes transaction and storage functions.
- Storage Engine storage engine node
- the storage device refers to the node used to process and store data records in the database and complete transaction execution and submission.
- OTP On-Line Transaction Processing
- the online transaction processing system is mainly used to process transaction-type transactions initiated by users in a short period of time.
- the characteristics of transaction-type transactions are: short transaction execution time and high concurrency, so for The online transaction processing system requires high throughput per unit time.
- On-Line Analytical Processing (OLAP) system an online analytical processing system, mainly used to process complex analytical transactions.
- the characteristics of analytical transactions are: long transaction execution time, large resource occupation, and processing time. Big amount of data.
- Hybrid Transaction/Analytical Processing (HTAP) system This is a data processing system that mixes the two loads of online transaction and online analysis. It can be regarded as a fusion system. It is expected that both Transactional transactions are supported as well as analytical transactions.
- HTAP Hybrid Transaction/Analytical Processing
- Hybrid Transaction/Analytical/Stream Processing (HTASP) system refers to a data processing system based on the HTAP system and introducing streaming computing capabilities.
- Extract-Transform-Load Used to describe the process of extracting and transforming data from the source to the destination.
- the ETL method is more commonly used in data warehouses, but the operable objects of ETL are not limited to data warehouses.
- Paxos algorithm It is a consensus (Consensus) algorithm based on message passing and highly fault-tolerant.
- Raft algorithm It is a consensus algorithm used to replace Paxos. Compared with the Paxos algorithm, the goal of the Raft algorithm is to provide a clearer logical division of labor, so that the Raft algorithm itself can be better understood. At the same time, the Raft algorithm is more secure and can provide some additional features.
- the Raft algorithm can provide a general method for deploying finite state machines among computer clusters, and ensure that any node in the cluster is consistent in certain state transitions.
- the database involved in this embodiment of the present application stores multiple data tables, and each data table can be used to store data records.
- the database can be any type of database based on MVCC (Multi-Version Concurrency Control, multi-version concurrency control).
- MVCC Multi-Version Concurrency Control, multi-version concurrency control
- the type of the database is not specifically limited.
- the data in the above database is based on state attributes and can include three states: current state, transition state, and historical state. Each of the different status attributes in the data can be used to identify the status of the data in its life cycle track.
- Transitional State It is not the latest version of the data record nor the historical version. It is in the process of transitioning from the current state to the historical state. The data in the transitional state is called half-life data.
- Historical State A state in which data is recorded in history, and its value is the old value, not the current value.
- the state of the data in the historical stage is called the historical state.
- the above three states of data exist while under the non-MVCC mechanism, the data can only exist in the historical state and the current state.
- the new value of the data after the transaction commits is in the current state.
- the data generated by the transaction before the smallest transaction in the current active transaction list is in the historical state.
- the blocking concurrent access control mechanism after the transaction is committed, the value of the data before submission becomes the value of the historical state, that is, the old value of the data record is in the historical state.
- the balance of account A in the user table changes from 10 yuan to 20 yuan, and then consumes 15 yuan to 5 yuan.
- the financial institution B reads the data for inspection and has been in progress. After A recharges 20 yuan to 25 yuan, then 25 yuan is the current state data, and the 5 yuan that B is reading is the transition state, and the other two values 20 and 10 are states that have existed in history and are all historical state data.
- the visibility of data records (that is, the visibility of data records) is specific to transactions. A certain data record may be visible to some transactions and invisible to some transactions. In the embodiment of this application, it involves judging whether the data records in the historical state (that is, the historical state data) are visible within the query time period of the transaction to be analyzed, and the relevant visibility judgment algorithm will be described in subsequent embodiments, here I won't go into details.
- the HTAP system provided by the related technology includes an OLTP cluster and an OLAP cluster.
- the OLTP cluster is used to process short-term user transaction transactions, and the OLAP cluster is used to process complex analytical transactions.
- a data migration module is configured between the OLAP cluster and the OLTP cluster to migrate the latest relevant data in the OLTP cluster to the OLAP cluster to ensure the consistency of data analysis and processing between the OLAP cluster and the OLTP cluster. Since the OLTP cluster does not provide the storage function for historical state data after a specific change is made to the data record, that is, when a data record is changed from the old value to the new value, the old value will be directly overwritten by the new value.
- the HTAP system only supports online transaction and analysis functions for the latest data, and cannot provide online analysis functions for historical data at any point in time or within a time period specified on the time axis.
- the embodiment of the present application provides a transaction processing method for each data record stored in the HTAP system, providing a function of saving full-state data in the time dimension, that is, saving all current state data, For transition state data and historical state data, when a data record is modified, the latest version of the data record (current state data) and the data record transformed into the historical version (historical state data) will be stored in the storage device of the storage cluster.
- the stream computing processing engine is also used to provide stream analysis and processing functions to the outside world.
- the HTAP system integrated with the stream computing processing engine is also called the HTASP system.
- a HTASP system that integrates stream computing processing engines is provided.
- OLTP clusters and OLAP clusters can use the same full-time data to complete transactions or analysis, and greatly reduce The coupling between the OLTP cluster and the OLAP cluster caused by the data migration module is eliminated, so that the OLTP cluster and the OLAP cluster are isolated from each other and do not affect each other when processing transactions.
- the OLAP cluster Since the relevant information of the time dimension is added to each data record, the OLAP cluster not only supports the function of analyzing the latest version of the data record, but also allows the user to customize the real-time degree parameters of the online analysis by recording the details at the record level. Granularity is used to analyze historical data, and after users customize the time window of incremental analysis results and the calculation period of full analysis results, they can perform full analysis every other calculation period. An incremental analysis is performed in a time window, that is, the combination of parallel computing and stream computing at the smallest granularity is provided, and the function of building the HTASP system on the same full-time data and providing high-quality data services to the outside world is realized.
- the embodiments of the present application can be applied to various data processing systems that support mixed transactions/analysis, such as HTAP systems and HTASP systems.
- the embodiments of the present application can also be applied to a database system based on blockchain technology (hereinafter referred to as "blockchain system"), the above-mentioned blockchain system is essentially a decentralized distributed database system, which uses a consensus algorithm to keep the ledger data recorded by different computing devices on the blockchain consistent. , use cryptographic algorithms to ensure encrypted transmission of ledger data between different computing devices and cannot be tampered with, use scripting systems to expand ledger functions, and use network routing to connect different computing devices.
- the blockchain system can include one or more blockchains.
- the blockchain is a series of data blocks associated with cryptographic methods. Each data block contains a batch of network transaction information for Verify the validity of its information (anti-counterfeiting) and generate the next block.
- the log records of each completed transactional transaction are used as ledger data, and a blockchain system is constructed to store a batch of network transaction information (ie, log records) in each computing device. ), the traceability of historical transaction information (historical state data) at any time can be realized after the network transaction information is uploaded to the chain.
- computing devices can form a peer-to-peer (Peer To Peer, P2P) network.
- P2P protocol is an application layer protocol that runs on top of the Transmission Control Protocol (TCP).
- TCP Transmission Control Protocol
- any computing device can have the following functions: 1) routing, the basic functions of computing devices, used to support communication between computing devices; 2) applications, used to deploy in the blockchain, Realize specific business according to actual business needs, record the data related to the realization function to form ledger data, carry digital signature in the ledger data to indicate the data source, and send ledger data to other computing devices in the blockchain system for other computing devices When the source and integrity of the ledger data are successfully verified, the ledger data is added to the temporary block.
- the business implemented by the application can include wallets, shared ledgers, smart contracts, etc.; Blocks that are consecutive in time sequence, once a new block is added to the blockchain, it will not be removed.
- the block records the ledger data submitted by the computing devices in the blockchain system.
- each block may include the hash value of the transaction records stored in this block (the hash value of this block) and the hash value of the previous block, and each block is formed by connecting the hash values
- the block can also include information such as the time stamp when the block was generated.
- FIG. 1 is an implementation environment architecture diagram of an HTAP system provided by an embodiment of the present application.
- an HTAP system 100 includes an OLTP cluster 110 and an OLAP cluster 120 .
- the OLTP cluster 110 is used to provide online transaction functions.
- the OLTP cluster 110 includes multiple TP computing nodes.
- the TP computing nodes are used to process transactional transactions. When the transactional transactions modify the data records, the generated historical data will be Dump to the distributed file system 130 through the storage interface.
- Each TP computing node is configured with its own database engine.
- Each TP computing node can be a stand-alone device or a cluster device with one master and two backups. The embodiment of this application does not specifically limit the type of TP computing nodes.
- the OLAP cluster 120 is used to provide online analysis functions.
- the OLAP cluster 120 includes a plurality of AP computing nodes, and the AP computing nodes are used to process analytical transactions.
- the analytical transactions include but are not limited to: querying historical state data or querying obtained
- the analytical transaction T1 is for the user to query the total amount of bills in the past month, then this analytical transaction T1 needs to be retrieved from the distributed file system In 130, query the user’s transaction records in the past month, and sum the amount spent (SUM).
- the total billing flow obtained is the analysis result of the analytical transaction T1.
- the analytical transaction Transaction T2 is for the user to query the total amount of new parking spaces in each administrative area of a certain city in the past week, then this analytical transaction T2 needs to use each administrative area in the city as a unit, from the distributed file system 130 Query the insertion records of new parking spaces in each administrative area, and sum (SUM) the number of inserted new parking spaces to obtain a set composed of the total number of new parking spaces in each administrative area It is the analysis result of analytical transaction T2.
- Both the OLTP cluster 110 and the OLAP cluster 120 can access the distributed file system 130 through the storage interface, and both the OLTP cluster 110 and the OLAP cluster 120 can directly or indirectly communicate with the distributed file system 130 through wired or wireless communication. connection, which is not specifically limited in this embodiment of the present application.
- the distributed file system 130 can provide unlimited storage functions for the OLTP cluster 110 and the OLAP cluster 120.
- the distributed file system 130 includes multiple storage nodes, that is, storage devices. Data with the same level is divided into storage devices under the same partition for storage, or data with the same cold and hot level is divided into the same storage medium of storage devices under the same partition for storage.
- the distributed file system 130 includes, but is not limited to: HDFS (Hadoop Distributed File System, Hadoop Distributed File System), Ceph (a distributed file system under a Linux system), Alluxio (a distributed file system based on memory) file system), etc.
- the TP computing device since the TP computing device provides an online transaction function, when any transaction type transaction is submitted and completed, while generating new current state data, historical state data corresponding to the current state data will also be generated , and because the historical state data will take up a lot of storage space, and the historical state data has preservation value, the historical state data will be dumped to the corresponding storage device in the distributed file system for persistent storage. After the data is dumped, the TP computing node can delete the historical state data dumped this time to clean up more storage space.
- the metadata of the historical state data dumped this time can also be registered in the metadata (Meta Data, MD) manager , which is convenient for the storage device to count the metadata of the stored historical state data based on the metadata manager.
- the database engine parses the user load (that is, user affairs, including transactional affairs or analytical affairs), based on SQL routing (Structured Query Language Router, SQL Router, referred to as SR) layer provides query statements, query operation semantics and metadata, and routes the user load to the OLTP cluster 110 or OLAP cluster 120 for processing.
- SQL routing Structured Query Language Router, SQL Router, referred to as SR
- the OLTP cluster 110 mainly provides query and modification services to current state data
- the OLAP cluster 120 mainly provides query and analysis services to historical state data (does not support modification or deletion of historical state data).
- the semantics of the query operation is the operation intent obtained by analyzing the query statement, for example, the condition of the WHERE clause may represent the intent of the WHERE clause.
- the database instance set of one or more databases corresponding to each TP computing node or AP computing node is called a SET (set).
- the database instance of the stand-alone device is a SET.
- the computing node is a cluster device with one master and two backups
- the SET of the computing node is a collection of the master database instance and two backup database instances.
- cloud database Cloud Database
- each SET supports linear expansion to meet the business processing requirements in big data scenarios.
- the OLTP cluster 110 or the OLAP cluster 120 implements the management of the TP computing nodes or AP computing nodes through a distributed coordination system (such as ZooKeeper), for example, makes a certain TP computing node or AP computing node fail (also That is, the TP computing node or AP computing node is deleted from the OLTP cluster 110 or the OLAP cluster 120).
- a distributed coordination system such as ZooKeeper
- the above-mentioned HTAP system 100 can be regarded as a server that provides mixed transaction and analysis functions externally.
- the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a Provide cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN (Content Delivery Network, content distribution network), and big data and artificial intelligence platform Cloud servers for basic cloud computing services.
- CDN Content Delivery Network, content distribution network
- Cloud servers for basic cloud computing services.
- the user load is a user transaction initiated by a user terminal to the HTAP system 100, where the terminal on the user side is also referred to as user equipment, terminal equipment, user terminal, mobile terminal, smart terminal, communication device, etc.
- the device type of the terminal includes: a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle terminal, a smart home appliance, a smart voice interaction device, etc., but is not limited thereto.
- FIG. 2 is a logical block diagram of a transaction processing flow provided by the embodiment of the present application.
- the HTAP system receives a SQL statement initiated by the user, and the SQL statement Input it into the query parser (Query Parser), and successively perform lexical analysis (Lexical Analysis), syntactic analysis (Syntax Analysis), temporal translation (Temporal Translation) and semantic check (Semantic Check) on the SQL statement.
- the query statement (SELECT Query) needs to be rewritten according to the query tree first, and then the rewritten query statement is input into the optimizer (Optimizer) for query optimization, and then from the transaction state
- the manager (Transaction Status Manager) applies for a transaction status (Transaction Status) for the query transaction of the query statement, and then inputs the query statement output by the optimizer and the transaction status output by the transaction status manager to the executor (Executor) for query execution.
- the current state data and historical state data are stored in different storage devices. For example, storing data records with high coldness and hotness levels (for example, high access frequency and modification frequency) in the current state data storage device, as time goes by, if some data records in the current state data storage device are cold The heat level is reduced, and the part of the data records whose cold and heat level has been reduced can be extracted and converted by ETL, and finally loaded into the historical state data storage device for persistent storage.
- high coldness and hotness levels for example, high access frequency and modification frequency
- the historical state data storage device can compress the loaded historical state data, for example, for each historical version of a certain data record, only store the full data of a part of the historical version, and the remaining part of the historical version is the same as the previous one. The incremental data between the full amount of data is stored.
- each data record is from insertion to update, and the versions arranged according to the time dimension (including the current version and one or more historical versions) are completely preserved.
- the HTAP system or the HTASP system can be realized based on only one set of full-time data, and the OLTP cluster
- the OLAP cluster and the OLAP cluster can minimize the coupling degree between the OLTP cluster and the OLAP cluster, that is, minimize the mutual influence between the OLTP cluster and the OLAP cluster.
- a distributed database cluster by dumping historical state data in units of data blocks, the impact of dumping historical state data on the OLTP cluster can be reduced, thereby ensuring that the normal operation of the OLTP cluster is not affected .
- Fig. 3 is a schematic diagram of the implementation environment of a transaction processing method provided by the embodiment of the present application. As shown in Fig. 3, it shows a real-time analysis system or real-time analysis model based on HTAP architecture. Since the real-time analysis system introduces The traditional HTAP does not support analysis on historical state data, and every time the current state data needs to be analyzed in full (equivalent to a layer of computing mode), but the real-time data provided by the embodiment of this application The analysis system, because of the dump of historical state data, supports analysis on historical state data, and does not need to conduct full analysis of current state data every time, but can be converted into the full amount calculated at a certain point in time before The analysis results are used as the baseline, combined with the incremental analysis results calculated from this time point to the time point specified by the real-time analysis transaction (if not specified by the user, it can be calculated according to the real-time degree parameters of the system), and combined to obtain the final final analysis result , which provides a two-layer real-time analysis model.
- this real-time analysis system includes a computing layer 310 and a hierarchical distributed shared storage system 320.
- an OLTP cluster 311 and an OLAP cluster 312 are included in the computing layer 310.
- the OLTP cluster 311 is usually aimed at initiating on current state data and transition state data.
- the short-term transaction type transactions provide online transaction functions, and the OLAP cluster 312 usually provides online analysis functions for complex analysis type transactions initiated on historical state data.
- Both the OLTP cluster 311 and the OLAP cluster 312 can access the full-time data stored in the hierarchical distributed shared storage system 320.
- all the data in the system are divided according to the hot and cold levels of the full-time data There are the following partitions: hot data, warm data, cold data, etc. The manner of dividing the cold and hot grades will be introduced in detail in subsequent embodiments, and will not be repeated here.
- the OLAP cluster 312 if the analytical transaction arriving at the OLAP cluster 312 does not involve hot data, then there is no need to interact with the OLTP cluster 311 during the execution of the analytical transaction. If the analytical transaction arriving at the OLAP cluster 312 involves hot data, and this part of the newly modified hot data has not been dumped into the hierarchical distributed shared storage system 320, then the OLAP cluster 312 will, according to the business requirements of the analytical transaction, The hot data that has not been dumped is queried from the OLAP cluster 311 based on the temporal condition (that is, the query time period).
- FIG. 4 is a flow chart of a transaction processing method provided by an embodiment of the present application.
- this embodiment is applicable to the data processing system that supports real-time analysis such as HTAP system, HTASP system or block chain system, and this transaction processing method is carried out by any computing device in the OLAP cluster, and this embodiment comprises the following steps :
- the computing device determines the first moment from the query time period of the transaction to be analyzed.
- the first moment is the moment when the historical state data is dumped last.
- the historical state data is data that meets the data query condition of the transaction to be analyzed.
- the transaction to be analyzed is any possible analysis-type transaction, for example, the analysis-type transaction includes but is not limited to: query transaction, read-only transaction, or transaction that performs analysis or processing on query results, etc.
- the analysis-type transaction includes but is not limited to: query transaction, read-only transaction, or transaction that performs analysis or processing on query results, etc.
- an exemplary analytical transaction in a financial scenario is: query the total amount of bills of a user in the past month; another example, an exemplary analytical transaction in a smart traffic scenario is: query The total amount of new parking spaces in each administrative area.
- the query time period is a time interval formed by a start time and an end time.
- the start time or end time of the query time period can be defined by the user request. If the start time or end time is not defined in the user request, the database engine configures the query time period for the transaction to be analyzed.
- the first moment refers to the moment when all the historical state data that meet the data query condition of the transaction to be analyzed are dumped last time.
- the data query condition is obtained by parsing the SQL statement of the transaction to be analyzed.
- the above dump process Refers to: migrating the historical state data generated by the OLTP cluster to the storage device of the storage cluster for storage.
- the user logs in to the application client on the terminal, triggering the application client to generate a user request, for example, in a financial scenario, the user request is a request to query the total amount of the user's bills in the past month, and for example, smart transportation In the scenario, the user request is to query the total amount of new parking spaces in each administrative area of a certain city in the past week, and the embodiment of the present application does not specifically limit the type of user request.
- the application client After the application client generates the user request, it can call an API (Application Programming Interface, application programming interface) to send the user request to the computing device.
- the API can be MySQL API (an API provided by a relational database system) .
- the computing device receives any user request from the application client, parses the header field of the user request, and when the header field indicates that the user request is an analytical transaction, parses the data field of the user request to obtain the user
- the SQL statement carried in the request (such as SELECT Query statement, etc.), put the SQL statement into the query parser, and use the query parser to perform lexical analysis, syntax analysis, tense translation and semantic inspection on the SQL statement to obtain the SQL
- the data query conditions and analysis semantics of the statement (such as summation, averaging, sorting, etc.), and if the start time of the query time period is specified in the SQL statement, the start time is obtained.
- the computing device creates a process or thread for the transaction to be analyzed, or reuses a certain process or thread that has been created, and processes the transaction to be analyzed through the process or thread affairs.
- the start time specified by the transaction to be analyzed shall prevail; when the transaction to be analyzed does not specify the start time of the query time period, in a
- the effective time of the latest version of the data queried by the firm to be analyzed shall prevail, that is, the effective time of the latest version of the data that meets the data query conditions shall be determined as the starting time; when it comes to querying multiple For each data, it is enough to select the corresponding visible version for analysis based on the effective time of the latest version of each data.
- the time shall prevail, that is, the effective time of the insertion version of the data that meets the data query conditions shall be determined as the starting time; when it comes to querying multiple data, each data shall be selected according to the effective time of the respective insertion version It only needs to analyze the visible version of , and the method of determining the start time is not specifically limited in this embodiment of the present application.
- the real-time degree parameter in the database system is obtained, and the real-time degree parameter is used to characterize the maximum delay supported by the analytical transaction processing data in the database system, in other words, how long ago the database system is allowed to use
- setting the real-time level parameter to 1 hour means that the database system allows analysis of historical data before 1 hour.
- the real-time level parameter is customized by the management user.
- the default value for example, 1 second
- the default value is also Modifications may be made, which are not specifically limited in this embodiment of the present application.
- the end time is determined based on the real-time level parameter. In a possible implementation, the time before the current time and the time before the current time reaches the real-time level parameter is determined as the end time. The embodiment does not specifically limit the manner of determining the end time.
- the query time period formed from the start time to the end time can be obtained.
- the query time period is a closed interval, that is, the query time period includes both the start time of the left endpoint and the end time of the right endpoint, or the query time period is a left-closed right-open
- the time interval of that is, the query time period includes the start moment of the left endpoint, but does not include the end moment of the right endpoint.
- the specificity of the left-closed and right-open time interval is that: for the start time, the data that takes effect exactly at the start time is visible, and for the end time, the data that takes effect exactly at the end time is invisible.
- the user can specify whether the query time period includes the end time of the right endpoint. If the end time is specified, the query time period is a closed interval. If the end time is not specified, then The query time period is a time interval that is closed on the left and open on the right. If the user does not specify whether to include the end time, the end time can be included by default, that is, the query time period is a closed interval by default.
- the OLTP cluster will periodically migrate the historical state data generated by its own transactional transactions to the storage device of the storage cluster for persistence Storage
- the above process of migrating historical state data is the process of dumping historical state data, because relevant records are usually generated each time a dump is performed, such as recording the dumped data in the meta information manager of the storage cluster What are the historical state data, or maintain a special table to record the completion time of each dump, so it is easy to query all the historical state data that meet the data query conditions of the transaction to be analyzed and the last time the dump is completed ( i.e. the first moment).
- the computing device obtains a full analysis result based on the historical state data, and the full analysis result is a result obtained by processing the historical state data based on the analysis semantics of the transaction to be analyzed.
- the query time period can be divided into two sub-intervals, the first sub-interval formed from the start time to the first time, and the second sub-interval formed from the first time to the end time Sub-intervals, the full analysis results of the first sub-interval and the incremental analysis results of the second sub-interval are combined to obtain the final final analysis results.
- the database system may cache the intermediate analysis results at the second moment between the start moment and the first moment, according to the visibility judgment algorithm, only the cached intermediate analysis results need to be saved at this time Combined with the incremental analysis results of the second sub-interval, the final analysis results can be obtained, that is, there is no need to repeatedly calculate the full analysis results at the first moment.
- the way to obtain the full amount of analysis results based on the historical state data may be to query whether to cache the intermediate analysis results of the first historical state data visible to the second moment, which is the query time period At any time between the start time of the first time and the first time, then, if the intermediate analysis result has been cached, determine the intermediate analysis result as the full analysis result; if the intermediate analysis result is not cached , generating the full analysis result of the second historical state data visible at the first moment.
- the visibility judgment algorithm and the method of generating the full analysis results will be described in detail in the next embodiment, and will not be repeated here.
- the computing device obtains incremental analysis results based on a specific change that occurs on the current state data corresponding to the historical state data, and the specific change is a change that occurs between the first moment and the end of the query time period , the incremental analysis result is the result of processing the data obtained by the specific change based on the analysis semantics.
- the first moment is the latest time when the corresponding historical data is dumped, it means that the data between the first moment and the end time has not been dumped from the OLTP cluster to the storage cluster.
- Obtain the log corresponding to the specific change play back the log generated by the specific change, and obtain the data corresponding to the specific change.
- the analysis semantics of the SQL statement processes the comparison result to generate the incremental analysis result.
- the incremental analysis result only represents the change from the last full analysis result, not the full analysis result obtained from the new calculation.
- the computing device acquires a final analysis result of the transaction to be analyzed based on the full analysis result and the incremental analysis result.
- the incremental analysis result includes the analysis result before the specific change and the analysis result after the specific change. Therefore, when combining the full analysis result and the incremental analysis result, it is necessary to subtract the specific analysis result from the full analysis result.
- the analysis result before modification is used to obtain the first analysis result; the final analysis result is obtained by adding the first analysis result to the specific modified analysis result.
- the traditional single-layer calculation model can be converted into a two-layer real-time analysis model, which can be adapted to the HTAP system based on the full-time database or HTASP system, and can save computing resources in the whole system.
- a full analysis result is calculated for the historical state data before the first moment , calculate an incremental analysis result for the data obtained after the specific change after the first moment, combine the full analysis result and the incremental analysis result, and finally get the final analysis result, so that for any one on the full temporal time axis
- it can provide online real-time analysis and processing services, and is not limited to only providing analysis functions for the latest data, thus greatly improving the transaction processing performance of database systems such as HTAP systems or HTASP systems.
- the traditional single-layer calculation model can be converted into a two-layer real-time analysis model, which can be adapted to the HTAP system or HTASP system based on the full-time database, and can conservees computing resources throughout the system.
- the real-time degree parameter is used to characterize the maximum delay supported by the analytical transaction processing data in the database system. If the parameter is 1 hour, it means that the database system allows analysis with the historical state data before 1 hour.
- the above commands for setting real-time level parameters enable management users to determine the level of online analysis of the current database according to their own SLA (Service Level Agreement, Service Level Agreement) requirements (that is, set the real-time level parameters of the current database as needed).
- SLA Service Level Agreement, Service Level Agreement
- the unit of the real-time degree parameter includes: second, minute or hour.
- the real-time degree parameter when the real-time degree parameter is set to 0, it means that the real-time analysis result is used, and it is necessary to perceive the latest transaction system (ie OLTP cluster)
- the real-time level parameter can be configured as a default value for the current database, and the default value is any value greater than or equal to 0, for example, the default value is 1s (second).
- the real-time degree parameter is represented by REAL_TIME_ANALYZING_DEGREE
- the command to view the real-time degree parameter of the current database is as shown in the following code:
- the above code shows a view command for the real-time level parameter of the current database. It can be seen that the displayed real-time level parameter set by the current user is 1 second, which means that the real-time granularity and the latest data are delayed by 1 second, that is, AP Computing nodes (that is, computing devices) can perform online analysis from historical or transitional versions of data that differ by 1 second from the latest data.
- AP Computing nodes that is, computing devices
- the real-time degree parameter is represented by REAL_TIME_ANALYZING_DEGREE
- the command to set the real-time degree parameter of the current database is as shown in the following code:
- the above code shows a setting command for the real-time level parameters of the current database. It can be seen that the meaning of this setting command is: allow the data version delayed by 5 seconds from the latest data to be used as the data version of the AP computing node for analysis .
- the embodiment of the present application uses The real-time level parameter of the second-level unit as an example for illustration.
- management users can modify it to granularity such as minute unit and hour unit according to their own SLA requirements, and then let the OLAP cluster determine all the data versions required for this query.
- the time window size refers to the step size at which the streaming computing engine (or streaming computing model) running on the AP computing node or streaming computing node calculates the incremental analysis results.
- the command to modify the size of the time window is mainly used to adjust the size of the minimum divided time window, and the streaming computing engine will start the division and incremental analysis of each data version in each time window according to the size of the time window. calculate.
- the unit of the time window size includes: seconds, minutes, or hours. Users can customize a time window size, but for the database engine, 1 second is the smallest time window unit. Therefore, the time window size set by the user is divided into multiple second-level time window units. In other words, the time window size set by the user is an integer multiple of the smallest time window unit. If the management user does not set the time window size, it can be used for the current database Configure the time window size as a default value, which is any value greater than 0, for example, the default value is 1s (second).
- the system will divide the data according to the smallest time window unit, but only when the time window size specified by the management user is reached. Perform calculations for incremental analysis results.
- time window size is represented by TIME_WINDOW
- command to view the time window size of the current database is SHOW TIME_WINDOW, as shown in the following code:
- the time window size is represented by TIME_WINDOW
- the command to set the time window size of the current database is SET TIME_WINDOW, as shown in the following code:
- the calculation period of the full analysis results is also the calculation frequency of the full analysis results.
- the calculation period is used to define how often the full analysis results are calculated in the streaming computing engine.
- the management user can define a calculation period for the full analysis results according to the business characteristics. By periodically initiating the calculation of the full analysis results, it is possible to reduce the time when the final analysis results are generated each time. Old, resulting in too many incremental analysis results that need to be merged in the middle, resulting in a decrease in computing performance. In other words, it can be guaranteed that the incremental analysis results that need to be merged with the last full analysis results are adapted to the business characteristics when the final analysis results are generated each time.
- the unit of the calculation cycle of the full analysis result includes: second, minute or hour, and the user can customize the calculation cycle of a full analysis result, but for the database engine, it will be 1 second The smallest time window unit, so that the calculation period of the full analysis result set by the user is divided into multiple second-level time window units. In other words, the calculation period of the full analysis result set by the user is an integer multiple of the smallest time window unit. If the management user has not set the calculation cycle of the full analysis results, the calculation cycle of the full analysis results can be configured as a default value for the current database. The default value is any value greater than 0.
- the default value is 1 minute
- the The default value of the calculation period is usually larger than the default value of the time window size of the incremental analysis results, because the system does not want to calculate the full analysis results frequently.
- management users can adjust the calculation cycle of the set full analysis results according to their own business conditions (currently also supports adjusting the default value).
- the current full analysis can also be perceived according to the performance monitoring module Whether the calculation cycle of the results will lead to fluctuations in resource usage, so as to adaptively adjust the calculation cycle of the full analysis results, that is, when the current system is relatively busy, adjust the calculation cycle of the full analysis results, that is, reduce the calculation frequency of the full analysis results, currently When the system is relatively idle, reduce the calculation cycle of the full analysis results, that is, increase the calculation frequency of the full analysis results.
- the calculation cycle of the full analysis results set by the management user is greater than the minimum time window unit of the system, the system will divide the data according to the minimum time window unit, but only when the full analysis result specified by the management user is reached The calculation of the full analysis results is performed during the calculation period of the results.
- the calculation period of the full analysis result is represented by FULL_ANALAZY_RESULT_PERIOD, then the command to set the calculation period of the full analysis result of the current database is SET FULL_ANALAZY_RESULT_PERIOD, as shown in the following code:
- MySQL[test]>Set FULL_ANALAZY_RESULT_PERIOD 1m; set the calculation period of the full analysis result to 1 minute;
- the hot and cold level is used to represent the frequency of operations on historical state data in the historical time period, because the operations that occur include but are not limited to read operations, that is, query or access operations, and write operations, that is, modification or update operations. Therefore, based on the number of visits to the historical state data and the number of revisions to the current state data corresponding to the historical state data within the historical time period, the hot and cold level of each version of the historical state data can be comprehensively determined.
- the management user can define the classification standard of hot and cold grades according to their own business system conditions, that is, the management user defines what kind of data to classify into which cold and hot grade.
- the following types of hot and cold grades are included: hot data, warm data, cold data and archived data, and the division standards for each hot and cold grade are set separately, and Partition the full-time data according to the cold and hot grades, divide the data with the same cold and hot grades into the storage devices under the same partition for storage, or divide the data with the same cold and hot grades into the same partition of the storage devices under the same partition Store in a storage medium, so as to achieve the best database use effect and the cheapest database use cost.
- HOT_COLD_DATA_LEVEL is used to represent the division standard of hot and cold levels
- SHOW HOT_COLD_DATA_LEVEL which means that when the access times of data in a certain temporal state within a specified time period is less than the specified value , it will be dumped to the next level of storage device or storage medium, as shown in the following code:
- the management user can specify the division standard of hot and cold grades from two dimensions, that is, based on a certain period (such as several hours, 1 day, 1 week, 1 month, 1 year, etc., the minimum is 1 second) for The modification and/or access times of a certain data record is used to determine whether this data record needs to be dumped to the next level of storage device or storage medium.
- a certain period such as several hours, 1 day, 1 week, 1 month, 1 year, etc., the minimum is 1 second
- HOT_COLD_DATA_LEVEL is used to represent the division standard of hot and cold levels, and the command to set the division standard of hot and cold levels of the current database is SET HOT_COLD_DATA_LEVEL;, which is used for management users to customize the dump standard according to their own business needs.
- the division standard of hot and cold grades can be modified according to the above format, and the detailed commands are omitted.
- a default classification standard of cold and hot grades can be provided, for example , divided into 4 hot and cold grades as follows: hot data (data that is frequently modified within 1 day, for example, the number of times of modification in 1 day> 100 times), warm data (the number of times of modification in 1 month is average, but will be accessed, such as 10 Times ⁇ 1 month modification times ⁇ 50 times, and 1 month visit times > 50 times), cold data (data will not be modified within 1 year, but will be accessed, for example, visit times within 1 year ⁇ 10 times) and archived data (Data that has not been accessed for more than 1 year).
- the above-mentioned default standard for classifying hot and cold grades may also be modified by the management user, which is not specifically limited in this embodiment of the present application.
- FIG. 5 is a schematic diagram of the relationship between historical state data and cold, hot, and warm data provided by an embodiment of the present application. It is measured by the number of revisions, so that under the full-time database, the corresponding relationship between historical data and cold data and warm data is naturally realized. That is, since hot data refers to data records that are frequently modified in a short period of time (such as 1 day), this means that hot data corresponds to current state data and transition state data, and cold data and warm data are relatively almost unmodified , will only be accessed, so warm data corresponds to historical state data, and cold data corresponds to older historical state data.
- transition state data is also dumped as historical state data over time.
- the full-temporal database will periodically start the dump algorithm to dump the transition state data from the storage device corresponding to the hot data to the storage device corresponding to the warm data (used to store historical state data).
- the storage device corresponding to the hot data can directly discard the dumped transition state data to save more storage space.
- both the current state data and the transition state data correspond to hot data
- multiple cold and hot levels can be defined according to user characteristics to meet the specific needs of their own business, for example, For historical data, it is divided into three hot and cold levels: warm data, cold data and archived data. Once any data is dumped as historical state data, it will become read-only data, that is, the system does not allow modification of historical state data, that is, prevents tampering with historical state data to protect the security of historical state data.
- the current state data and transition state data with high modification frequency can also be corresponding to thermal data
- the current state data and transition state data with low modification frequency can be corresponding to temperature data
- the historical state data The data is divided into cold data and archived data.
- the above Figure 5 only shows a relationship between the full-time data and the data under the cold and hot levels, but it is not limited to the relationship shown in Figure 5 .
- both the current state data and the transition state data correspond to the currently active data records.
- the latest version of the data records will be continuously generated; (During the target time period) if it has not been modified or accessed continuously, it will be dumped as historical data (such as cold data). The continuous aging of historical data will be dumped into cooler data (such as archived data).
- Fig. 6 is a schematic diagram of adding a real-time analysis slave copy in a master-slave copy mechanism provided by an embodiment of the present application.
- a storage cluster ie, a storage engine
- a master-slave copy mechanism is usually introduced, that is, at least one slave copy is saved for each piece of data (that is, the master copy) to ensure that even if the master copy goes down, it is still Related data services can be provided on the replica.
- an additional real-time analysis slave copy will be introduced in each set of master-slave copies of the storage cluster.
- the master-slave copy will read and write data in the traditional way, and write related data to the hot data
- the real-time analysis slave copy is only responsible for synchronizing data, but will not participate in the election process of the master copy, which is due to the election process caused by some consistency protocols that may periodically re-elect the master copy, and
- the real-time analysis slave replica will not participate in any voting process under the consensus agreement (that is, it will not participate in the majority agreement under the consensus agreement).
- Real-time analysis of the node device where the copy is located is used to provide the dump function of historical data.
- the dump function of historical data can be migrated from the master copy to the real-time analysis slave copy, which can subtract the node device where the master copy is located. computing pressure, thereby completely decoupling the interaction between OLTP clusters and OLAP clusters.
- real-time analysis of the slave copy does not necessarily pull data from the master copy every time, but can select the most favorable copy (such as the most idle copy) to synchronize, so that the synchronization task of the real-time analysis copy minimizes the impact on the OLTP cluster.
- the real-time analysis can periodically clean up the dumped data (primary replica and other It can also be cleaned up synchronously from the copy), for example, for data that has been modified less than 50 times and accessed less than 100 times in one day, it will be dumped to the storage device corresponding to the historical state data, and then the dumped data will be directly deleted.
- Fig. 7 is a schematic diagram of a cold and hot data storage model provided by the embodiment of the present application. As shown in 700, as described above, for the data tables in the full-time database, the division standard HOT_COLD_DATA_LEVEL of hot and cold levels is added This parameter, below, will be divided into four types of hot data, hot data, cold data, and archived data as an example, and introduce how to store corresponding data based on the four cold and hot levels.
- management users can use the relevant commands provided in (1.4) above to set the respective division standards for hot data, warm data, cold data, and archived data according to their own business characteristics when the cluster is established.
- this embodiment of the application is not limited to being divided into 4 levels, and does not limit the division criteria of each level. Management users can be subdivided into any number of levels and set arbitrary division standards.
- the system on the basis of setting the classification standard of hot and cold grades, when creating a data table later, the system will automatically partition the data table according to the hot and cold grades of the data, and divide the data table into Store in the storage device under the corresponding partition.
- the background thread of the storage device will also regularly judge whether the hot or cold level of the data changes. When the hot or cold level changes, the data needs to be dumped to other storage devices in the corresponding partition.
- the background storage processing tasks for the data stored in different partitions for example, scan the data stored in each partition All data, to determine whether each version of the data record meets the conditions for dumping to the next storage level. If the dumping condition is met, the data records of each version that meet the requirements need to be moved to the corresponding storage partition.
- the full-time state The data will retain all versions of all data records, and historical versions cannot be modified.
- the data records of historical versions will be gradually migrated from hot data partitions to warm data partitions, cold data partitions, and finally to archive data partitions (only saved but not normally accessed).
- the management user can change at any time or the system automatically and dynamically adjusts the division standard of cold and hot grades without stopping.
- the background thread of the storage device will adjust its own partition according to the load situation. For example, when adding a partition, first add the corresponding new partition, and then adjust the data of adjacent partitions; for example, when reducing a partition, although an existing partition is reduced, the data in the reduced partition will not be deleted. Therefore, it is necessary to migrate the data in the deleted partition to the adjacent partition, and then delete the specified partition; for example, when modifying a partition, the background thread will transfer the data that does not meet the division standard of the current partition Data is migrated to adjacent partitions. For example, if the division standard of the warm data partition is changed from less than 100 access times to less than 90 access times, then in the warm data partition, the data with the original access times between 90 and 100 will need to Migrated to the hot data partition.
- the query calculation model of the computing equipment in the OLAP cluster also needs to be restructured to provide real-time analysis and calculation with high efficiency and low resource usage in the embodiment of the present application capabilities, and supports the basic query execution model of a streaming computing engine based on time window granularity.
- the data services usually provided by relational databases will be transformed into the processing of SQL statements submitted by users. Since historical data is not allowed to be modified or deleted, it is read-only data, so the calculation statements involved in the embodiment of this application are only query (Select ) statement, and other addition, deletion, and modification statements only need to be implemented according to the semantics of data extraction in the SQL:2011 tense standard. Therefore, since the Select statement must involve judging the visibility of data records, the algorithm for judging the visibility of data records will be described below.
- FIG. 8 is a schematic diagram of a full-time data record storage form provided by the embodiment of the present application. As shown in 800, the horizontal axis represents the time axis, and represents the passage of time from left to right.
- the records stored in the database 1, Record 2 and record 3 are taken as examples to illustrate the visible versions of each record at four moments from T1 to T4.
- Record 1 (Record 1, R1): R1 is inserted into the database on the leftmost side of the time axis, the inserted version is V1, and R1 is modified at some point before T1, so the first line segment represents R1.V1 (V1 version of R1) life cycle, which represents the valid time of the corresponding version, which can be determined based on the valid time defined by the user or based on the transaction time of the data. The next three line segments represent the respective life cycles of R1.V2, R1.V3, and R1.V4.
- the analysis shows that at time T1, the visible version of R1 is V2; at time T2, version V2 just ends, and due to time continuity, the left end value of the life cycle of version V3 is equal to the right end value of the life cycle of version V2, so at time T2,
- the visible version of R1 is V3; at time T3, it is still in the life cycle of V3, so the visible version of R1 is V3; at time T4, it is in the life cycle of V4, so the visible version of R1 is V4.
- Record 2 (Record 2, R2): R2 appears as a ray on the entire time axis, which means that R2 has not been modified after being inserted into the database at the far left of the time axis, and has always been in the R2.V1 version.
- Record 3 (Record 3, R3): R3 was inserted into the database after T1 time, so it has not been inserted at T1 time, T1 time is invisible, and then it is deleted at some time between T1 time and T2 time, so Has been deleted after time T2 and is not visible after time T2.
- the life cycle of each version of the data record is identified by a time interval of left-closed and right-open, for example, R1.V1[2020-08-09 12:30:30,2020-08-19 22 :15:30) indicates that record R1 was inserted at 2020-08-09 12:30:30 and expired at 2020-08-19 22:15:30.
- the query transaction at each time point can only see one version of a certain data record, but some query transactions are not initiated for the time point, but may specify Therefore, the data visibility judgment algorithm for a certain time period (that is, the query time period) is described below:
- a query time period is divided into four consecutive time periods as follows: [min_time,T1), [T1,T3), [T3,T2), [T2,current_time], if a query transaction specifies a query Time period (or called effective time interval, transaction time interval, etc.), then for the query time period, the algorithm for determining the visibility of data records is as follows.
- time1 refers to the left end value of the above query time period (ie min_time)
- time2 refers to the right end value of the above query time period (ie current_time).
- its visibility judgment algorithm For a single time period, its visibility judgment algorithm is expressed as: determine the right end value of a single time period, and use the data visibility judgment algorithm at a single moment on the right end value of the single time period to determine the data relative to the single time period The visible version of the data.
- a single time period [min_time, current_time] is split into [min_time, T1), [T1, T3), [T3, T2) and [T2, current_time] ]
- These 4 time periods namely [min_time, current_time] -> [min_time, T1), [T1, T3), [T3, T2), [T2, current_time].
- the visible data version is judged according to the visibility judgment algorithm of a single time period; for the rightmost time period among the multiple time periods, each The latest visible record version value of the data record, and find the leftmost record version value, and then execute a multi-level query result correction algorithm on the two versions of the data records to ensure the correctness of the query results; at the rightmost time
- the segment is a left-closed right-open interval
- this data record will be skipped, and this data record will be regarded as invalid. It can be seen that when the rightmost time period is a closed interval, at this time, the visible version under the right end value of the rightmost time period is used as the visible value of this record.
- FIG. 9 is a flowchart of a transaction processing method provided by an embodiment of the present application.
- this embodiment is applicable to data processing systems that support real-time analysis such as HTAP system, HTASP system or blockchain system, and the transaction processing method is executed by any computing device in the OLAP cluster, and this embodiment includes the following steps :
- the computing device parses the user request, and obtains the data query condition, analysis semantics, and query time period of the transaction to be analyzed corresponding to the user request.
- the computing device receives any user request from the application client, parses the header field of the user request, and when the header field indicates that the user request is an analytical transaction, parses the data field of the user request to obtain the user
- the SQL statement carried in the request (such as SELECT Query statement, etc.), put the SQL statement into the query parser, and use the query parser to perform lexical analysis, syntax analysis, tense translation and semantic inspection on the SQL statement to obtain the SQL
- the data query conditions and analysis semantics of the statement (such as summation, averaging, sorting, etc.), and if the start time of the query time period is specified in the SQL statement, the start time is obtained.
- the end time of the segment then get the end time.
- the start time specified by the transaction to be analyzed shall prevail;
- the effective time of the latest version of the data queried by the analysis firm shall prevail, that is, the effective time of the latest version of the data that meets the data query conditions shall be determined as the starting time; when it comes to querying multiple data, each data It is enough to select the corresponding visible version for analysis based on the effective time of their latest version.
- the effective time of the inserted version of the data queried by the transaction to be analyzed shall prevail, that is, the effective time of the inserted version of the data that meets the query conditions of the data will be effective.
- the time is determined as the start time. When it comes to querying multiple data, it is enough to select the corresponding visible version for each data based on the effective time of the respective inserted version for analysis.
- the method of determining the start time in the embodiment of the present application Not specifically limited.
- the transaction to be analyzed specifies the end time of the query time period
- the specified end time of the transaction to be analyzed shall prevail; when the transaction to be analyzed does not specify the end time of the query time period, the database
- the real-time degree parameter in the system which is used to characterize the maximum delay supported by the analytical transaction processing data in the database system, in other words, how long ago the database system can allow the use of historical state data for analysis, such as , setting the real-time level parameter to 1 hour means that the database system allows analysis with historical data from 1 hour ago.
- the real-time level parameter is customized by the management user.
- the default value for example, 1 second
- the default value is also Modifications may be made, which are not specifically limited in this embodiment of the present application.
- the end time is determined based on the real-time degree parameter.
- the time before the current time before the current time reaches the real-time degree parameter is determined as the end time.
- the embodiment of the present application does not set the end time
- the determination method is specifically limited.
- the query time period formed from the start time to the end time can be obtained.
- the query time period is a closed interval, that is, the query time period includes both the start time of the left endpoint Also includes the end time of the right endpoint, or, the query time period is a time interval of left-closed and right-open, that is, the query time period includes the start time of the left endpoint, but does not include the end time of the right endpoint, wherein, left-closed right
- the specificity of the open time interval is: for the start time, the data that takes effect exactly at the start time is visible, and for the end time, the data that takes effect exactly at the end time is invisible, which is used for judgment assistance Data visibility.
- the user specifies whether the query time period includes the end time of the right endpoint. If the end time is specified, the query time period is a closed interval. If the end time is not specified, the query time period is left-closed and right-open. Time interval, if the user does not specify whether to include the end time, the end time can be included by default, that is, the query time period is a closed interval by default.
- the computing device determines a first moment from the query time period.
- the first moment is the moment when the historical state data is dumped last.
- the historical state data is data that meets the data query condition of the transaction to be analyzed.
- the OLTP cluster will periodically migrate the historical state data generated by its own transactional transactions to the storage device of the storage cluster for persistence Storage
- the above process of migrating historical state data is the process of dumping historical state data, because relevant records are usually generated each time a dump is performed, such as recording the location of the last synchronized log (Checkpoint), or Record the historical state data of this dump in the meta information manager of the storage cluster, or maintain a table to record the completion time of each dump, so it is easy to query all the transactions that match the transaction to be analyzed
- the moment when the historical state data of the data query condition was dumped last time that is, the first moment).
- the computing device queries whether to cache the intermediate analysis results of the first historical state data visible at a second moment, where the second moment is any moment between the start moment of the query time period and the first moment.
- the query time period can be divided into two subintervals, the first subinterval [start moment, first moment) formed from the start moment to the first moment, and the first subinterval formed from the first moment
- the second sub-interval [first time, end time] is formed from time to end time, and the final analysis result can be obtained by merging the full analysis results of the first sub-interval and the incremental analysis results of the second sub-interval.
- the analysis work of a query time period is divided into the analysis work for two sub-intervals (ie, two time periods).
- the embodiment of the present application takes the second sub-interval as an example for illustration, but the user can specify whether the query time period includes the end time of the right-hand value, and if the end time is specified, the second sub-interval The interval is a closed interval. If the specified end time is not included, the second sub-interval is a left-closed right-open interval. If the user does not specify whether to include the end time, the end time can be included by default, that is, the default second sub-interval is closed interval.
- the database system may cache the intermediate analysis results at the second moment between the start moment and the first moment, according to the visibility judgment algorithm, if the query time period is divided into more than two time segment, then it is only necessary to examine the leftmost time segment and the rightmost time segment to determine the visible version of the data record, and then perform analysis and processing on the visible version.
- the query time period is divided into three sub-intervals: [start moment, second moment), [second moment, first moment), [first moment, end moment ], you only need to combine the cached intermediate analysis results of [start time, second time) and the incremental analysis results of the second sub-interval [first time, end time] to get the final analysis results, that is, no need Repeat the calculation to obtain the full analysis result at the first moment.
- the computing device when the computing device queries the intermediate analysis result, it determines whether the previous historical transaction has been analyzed for the same data query condition according to the data query condition of the transaction to be analyzed.
- the data query condition of the transaction enters the following step 904-B; if the data query condition of any previous historical transaction is hit, the transaction ID (Identification, identification) of the historical transaction hit is used as an index to query Whether there is a full analysis result corresponding to the index cached, if any cached full analysis result is hit, determine the hit full analysis result as the intermediate analysis result, and enter the following step 904-A, if all caches are not hit If it is determined that the intermediate analysis result has not been cached, enter the following step 904-B.
- step 904-A after querying whether to cache the intermediate analysis result of the first historical state data visible at the second moment, if the intermediate analysis result has been cached, the following step 904-A is performed; if the intermediate analysis result is not cached In the case of intermediate analysis results, the following step 904-B is performed.
- the computing device determines the intermediate analysis result as the full analysis result.
- the intermediate analysis result at any second moment from the start moment to the first moment is cached, the intermediate analysis result will be directly used as the full analysis result without redundant calculation of the full analysis result , which can greatly save the computing resources of the computing device.
- the computing device If the intermediate analysis result is not cached, the computing device generates a full analysis result of the second historical state data visible at the first moment.
- the computing device needs to use a single time period visibility judgment algorithm to determine that the first sub-interval [start time, first time) is visible from the historical state data
- the second historical state data of that is to say, only for the first moment of the right end value of the first subinterval [start time, first moment), execute the single-moment visibility judgment algorithm to determine the second historical state data.
- step 901 After determining the second historical state data from each version of the historical state data, based on the analysis semantics obtained by parsing in step 901 above, perform the processing corresponding to the analysis semantics on the second historical state data to obtain the full analysis result. For example, if the analysis semantics is to average the value of a certain field, then it is necessary to calculate the average of the Value values of the corresponding fields of each second historical state data. The Value values of the corresponding fields of the second historical state data are added to obtain a sum value obtained by the addition.
- the computing device obtains a full analysis result based on the historical state data.
- the full analysis result is the result of processing the historical state data based on the analysis semantics of the transaction to be analyzed, so that the The full analysis result of step 904-A or 904-B is combined with the incremental analysis result of step 905 below to obtain the final analysis result, that is, the single-layer calculation model is transformed into a full-scale + incremental two-layer calculation model.
- the storage cluster of the HTAP system or the HTASP system can be divided into corresponding partitions according to the classification standard of cold and hot grades, and store historical state data corresponding to the hot and cold grades in the storage devices in each partition.
- historical state data with different cold and hot levels are stored in different storage devices or different storage media of the same storage device. Therefore, when the computing device reads the second historical state data, it first determines the hot and cold grades of the second historical state data, and the hot and cold grades are used to represent the changes that occurred on the second historical state data within the historical time period. The frequency of operation; then read the second historical state data from the storage device corresponding to the hot and cold level.
- the computing device when the computing device determines the hot and cold levels of the second historical state data, it acquires the number of visits to the second historical state data within the historical time period, and obtains the access times of the second historical state data within the historical time period.
- the modification times of the current status data corresponding to the status data and determine the hot and cold levels based on the access times and the modification times.
- the historical time period refers to the time period (such as 1 day, 1 month, 1 year, etc.) that needs to be judged in the classification standard of hot and cold levels under the current database set by the management user.
- the hot and cold levels of the second historical state data can be determined.
- the computing device obtains incremental analysis results based on a specific change that occurs on the current state data corresponding to the historical state data, and the specific change is a change that occurs between the first moment and the end of the query time period , the incremental analysis result is the result of processing the data obtained by the specific change based on the analysis semantics.
- the first moment is the latest time when the corresponding historical data is dumped, it means that the data between the first moment and the end time has not been dumped from the OLTP cluster to the storage cluster.
- Obtain the log corresponding to the specific change play back the log generated by the specific change, and obtain the data corresponding to the specific change.
- the analysis semantics of the SQL statement processes the comparison result to generate the incremental analysis result.
- the incremental analysis result only represents the change from the last full analysis result, not the full analysis result obtained from the new calculation.
- the incremental analysis results include the analysis results before the specific change and the analysis results after the specific change.
- the computing device compares the historical state data with the same primary key identifier and the data obtained by the specific change, it needs to first check the primary key Identify the same historical state data and data obtained by specific changes, and determine the fields that have specific changes.
- the primary key identifies the same historical state data and data obtained by specific changes here refers to the data records with the primary key identification Different versions are generated when specific changes are made at different times, for example, for the V1 version of the historical state of recording R1 and the V4 version obtained by the latest specific changes, determine the field where the specific changes occur.
- the data query condition does not involve the field with a specific change, it means that the field queried by the transaction to be analyzed has not changed, so the data with the primary key identifier can be skipped, for example, if It is the field c1 that has changed between the V1 version and the V4 version of the record R1, while the other fields have not changed.
- the data query condition is querying the field c2, which means that there is no incremental change between the first moment and the end moment.
- the computing device acquires a final analysis result of the transaction to be analyzed based on the full analysis result and the incremental analysis result.
- the incremental analysis result includes the analysis result before the specific change and the analysis result after the specific change. Therefore, when combining the full analysis result and the incremental analysis result, it is necessary to subtract the specific analysis result from the full analysis result.
- the analysis result before modification is used to obtain the first analysis result; the final analysis result is obtained by adding the first analysis result to the specific modified analysis result.
- the traditional single-layer calculation model can be converted into a two-layer real-time analysis model, which can be adapted to the HTAP system based on the full-time database or HTASP system, and can save computing resources in the whole system.
- a full analysis result is calculated for the historical state data before the first moment , calculate an incremental analysis result for the data obtained after the specific change after the first moment, combine the full analysis result and the incremental analysis result, and finally get the final analysis result, so that for any one on the full temporal time axis
- it can provide online real-time analysis and processing services, and is not limited to only providing analysis functions for the latest data, thus greatly improving the transaction processing performance of database systems such as HTAP systems or HTASP systems.
- the traditional single-layer calculation model can be converted into a two-layer real-time analysis model, which can be adapted to the HTAP system or HTASP system based on the full-time database, and can conservees computing resources throughout the system.
- the transaction processing flow of the database system such as the HTAP system or the HTASP system is introduced in detail, and in the embodiment of the present application, a specific data query statement will be combined to describe in detail how the computing devices in the OLAP cluster The final analysis results are obtained by analyzing the model in real time.
- Fig. 10 is a schematic flowchart of a real-time analysis model provided by the embodiment of the present application. As shown in 1000, it will be introduced that when the real-time degree parameter REAL_TIME_ANALYZING DEGREE is set to 1 second, the system enters a data query statement at a certain moment (that is, the transaction to be analyzed is a query transaction), how the computing device will process the data query statement.
- the data query statement is:
- the user can add corresponding time syntax to the data query statement to limit the target data set of the query. Assuming that the user does not set any time state information like some common analysis queries, then the data query statement of this Online real-time analysis process (i.e. real-time calculation algorithm) includes:
- Step 1 Start to execute the analysis query.
- step 1 it is necessary to increase the transaction time range for the query transaction. For example, first, determine the value of the real-time degree parameter REAL_TIME_ANALYZING_DEGREE, that is, determine the real-time performance requirements of the user for the OLAP cluster (assuming that REAL_TIME_ANALYZING_DEGREE is set to 1 second). Modify the data query statement to:
- min_time refers to the start time of the query time period
- current_time refers to the current time
- current_time–REAL_TIME_ANALYZING_DEGREE refers to the end time of the query time period.
- Step 2 Determine whether there is an intermediate calculation (ie intermediate analysis result) in the warm layer. If there is an intermediate calculation in the warm layer, go to step 4; if there is no intermediate calculation in the warm layer, go to step 3.
- intermediate calculation ie intermediate analysis result
- the warm layer refers to the storage partition where the warm data is located, that is, under the partition where the warm data is located in the storage cluster. Query whether to cache the intermediate analysis result in the storage device.
- cold and archive Layer data is mainly used to separate and store old historical data (that is, analytical transactions usually do not access cold data and/or archived data, which are only separated to reduce storage pressure).
- Step 3 Divide the query execution into two sections and go to step 7.
- the single query time period [min_time, current_time–REAL_TIME_ANALYZING_DEGREE] in the above data query statement is converted into the following two time periods: [min_time, time1) and [time1, current_time–REAL_TIME_ANALYZING_DEGREE], where time1 is the In the real-time analysis involved in the data dump program, the slave copy dumps the real-time data of the master copy or other slave copies to the latest time period of the storage device corresponding to the partition of the storage cluster through the background thread, that is, time1 refers to the first Moment (the moment when the historical state data that meets the data query conditions of the transaction to be analyzed is dumped the last time). It should be noted that the data stored in the copy for real-time analysis may include some warm data in the warm layer (because some warm data may not have been dumped yet, but this does not affect the analysis and calculation of query transactions).
- Step 4. Divide the query execution into three sections and go to step 5.
- Step 5 Directly obtain the full analysis results (ie, intermediate analysis results) of the previously saved warm layer, and proceed to step 6.
- Step 6 Calculate the incremental analysis result of the incremental part, and proceed to step 8.
- Step 7. Calculate the full analysis results of the warm layer and the incremental analysis results of the hot layer in parallel, and proceed to step 8.
- step 3 the calculation process for the respective analysis results of the above two time periods can be started in parallel, and the parallel calculation has a higher calculation effect than the serial calculation.
- the full analysis result of the time period [min_time, time1) in the warm layer calculates the full analysis result of the time period [min_time, time1) in the warm layer, and after the full analysis result is calculated, if the user sets to save the full analysis result, the full analysis result will be saved in the warm layer
- the analysis results are used as reference for subsequent similar query transactions, that is, the full analysis results of this time are cached as the intermediate analysis results queried by subsequent similar query transactions in step 2.
- a full calculation can be re-initiated periodically in the warm layer to avoid that the last full calculation is too old due to the lack of full calculation for a long time, so there are too many data-specific changes that lead to the required calculation data set If it is too large, the resulting query performance will be degraded.
- Step 8 Combine the full analysis results of the warm layer and the incremental analysis results of the hot layer to obtain the final analysis results.
- the full analysis results of the worm layer and the result set of the data obtained from the specific change of the hot layer are combined to obtain the final analysis results.
- Step 9 return the query result of this time, that is, return the final analysis result obtained by combining.
- Step 10 the real-time analysis ends.
- a two-stage real-time computing model is provided by combining the full analysis results of the worm layer and the incremental analysis results of the hot layer, which can minimize the gap between the OLTP cluster and the OLAP cluster.
- the degree of coupling that is, when the OLAP cluster performs real-time analysis, it only needs to scan the data of specific changes in the hot layer of the OLTP cluster through real-time analysis from the copy, and if the user does not need high real-time performance, it may be warm
- the warm data in the warm layer can meet the calculation requirements of real-time analysis. At this time, there is no need to access the hot data in the hot layer. In this case, the OLTP cluster is completely decoupled.
- the intermediate calculation results cached in the warm layer are It can be reused, so it can save computing resources to the greatest extent.
- step 7 and step 8 of the above-mentioned embodiment it involves calculating the incremental analysis results for the data obtained by specific changes, and finally merging the full analysis results and the incremental analysis results.
- step 7 and step 8 of the above-mentioned embodiment it involves calculating the incremental analysis results for the data obtained by specific changes, and finally merging the full analysis results and the incremental analysis results.
- the real-time computing task is decomposed into two-layer computing mode of full amount + incremental, and a full amount of analysis result is generated in the warm layer, and at the same time, the full amount of analysis result is supported to be cached to achieve multiple reuse, so as to reduce the calculation Resources are wasted due to double counting.
- the real-time analysis capability of the entire system presents the best performance to users and improves the user experience.
- the time period for the full calculation of the worm layer refers to the leftmost time period among the multiple time periods
- the time period for the incremental calculation of the hot layer refers to multiple time periods The rightmost time segment in the segment.
- the analysis results before the specific change refer to the old values of each record, due to the time period of the incremental calculation
- the left end value of is the first moment, so the above old value refers to the relevant field value of the visible version at the first moment, and the analysis result after the specific change refers to the new value of each record, because the right end value of the incremental calculation time period is the end time (that is, the time obtained by subtracting the real-time degree parameter from the current time), so the above new value refers to the relevant field value of the visible version at the end time.
- the identification information row_id can be added to each version during calculation, so that each version obtained by making multiple specific changes to the same data record have the same row_id.
- efficient streaming real-time analysis computing capabilities can also be provided to convert the HTAP system into the HTASP system, so as to achieve the fastest and most efficient real-time analysis and calculation, and Minimize the use of internal computing resources in the system, thereby improving the resource utilization of the system.
- the streaming computing engine if the streaming computing mode is turned on, assuming that the final analysis result of the transaction to be analyzed is the first full analysis result in the streaming computing, then, from the Starting at the end of the query period of the analysis transaction, every first time period (that is, the time window size involved in the previous embodiment), obtain a specific incremental analysis result of a specific change within the first time period; the cache obtains Analysis results for each specific increment of . Every second time interval (that is, the calculation cycle of the full analysis results involved in the previous embodiment), based on the last full analysis result and each specific incremental analysis result generated within the second time length, the full analysis of this time is obtained result.
- every first time period that is, the time window size involved in the previous embodiment
- FIG 11 is a schematic diagram of the storage form of data records of a streaming computing model provided by the embodiment of the present application.
- the size of the time window, that is, the first duration can be determined by
- the extended commands involved in the previous embodiments are used to set and view, perform incremental calculations on the visible version of the data records in each time window, and obtain the results of each incremental analysis.
- the calculation period based on the full analysis results that is, the second duration , it is possible to divide a timeline every interval of the calculation cycle, and combine all the incremental analysis results in each time window in the two adjacent timelines, and finally obtain the full analysis result of this time.
- the calculation method of the full analysis result and the incremental analysis result is the same as that in the previous embodiment, and will not be repeated here.
- Fig. 12 is an implementation environment architecture diagram of a HTASP system provided by an embodiment of the present application.
- the real-time analysis system includes a computing layer 1210 and a hierarchical distributed shared storage system 1220.
- the computing layer 1210 Including OLTP cluster 1211, OLAP cluster 1212 and streaming computing cluster 1213, OLTP cluster 1211 usually provides online transaction functions for short-term transaction transactions initiated on current state data and transition state data, OLAP cluster 1212 usually provides online transaction functions for Complex analytical transactions initiated on data provide online analysis capabilities.
- the stream computing cluster 1213 can be integrated inside the OLAP cluster 1212, for example, load a stream computing engine to some or all of the AP computing nodes in the OLAP cluster 1212, so as to provide conventional analysis functions and stream analysis functions at the same time.
- the streaming computing cluster 1213 is used to periodically perform incremental computing and full computing based on the above streaming computing model, so as to continuously output full and real-time analysis results in continuous time periods, making the entire HTASP system highly capable of transaction analysis performance.
- the OLTP cluster 1211, the OLAP cluster 1212, and the stream computing cluster 1213 can all access the full-time data stored in the hierarchical distributed shared storage system 1220.
- the All data in the system is divided into the following partitions: hot data, warm data, cold data, archived data, etc.
- the division criteria of hot and cold levels have been introduced before, so I won’t go into details here.
- the streaming computing function is only provided for analytical transactions that are specified by the user to perform streaming computing. For example, the user adds a Stream field in the data query statement to specify the start of streaming computing.
- the system can macro-analyze the rules of the query transactions initiated by each user. If the user initiates a certain type of similar query transactions at intervals (such as querying the total amount of the month before the repayment date of each month) Billing flow), then according to the time dimension, according to the historical query transaction initiated by the user and the data characteristics queried by the historical query transaction, a stream computing model for the user's query business can be created.
- the full temporal data is divided into the first level through the calculation period of the full analysis results.
- the data obtained by each specific change is further refined according to the data characteristics of the query. Multiple time windows are used to calculate incremental analysis results, thereby improving the horizontal scalability of computing power. It should be noted that if there are not many specific changes in data in a calculation period, then this calculation period can be divided into fewer time windows or no time windows can be divided, but directly obtained from all specific changes in two calculation periods The data is incrementally calculated.
- the streaming computing model includes the following steps:
- Step 1 Start streaming computing.
- the first time window is the leftmost time period of real-time calculation, that is, the leftmost time period in the query time period specified by the initiating transaction of streaming computing.
- the initiating transaction will calculate the first revelation Analyze the results in full.
- Step 2 For each subsequent time window, determine the visible data-specific change set (that is, the set of data obtained from the specific change operation in this time window), and calculate the incremental analysis results for the data-specific change set, but do not Do the merge operation.
- Step 3 Whenever the calculation cycle of the full analysis result is reached, based on the last full analysis result and all incremental analysis results generated from the last full analysis result to the current moment, they are sequentially merged to obtain this time Analyze the results in full.
- Step 4 When the user initiates a new query transaction for the relevant data that is being streamed, first, find the latest full analysis result as the basis, and then merge all the previous full analysis results sequentially Incremental analysis results of the complete time window, and finally, the incremental analysis results of the data obtained by specific changes from the last time window to the time when the query transaction is issued (the time span of this part of data is less than one time window and therefore need to be calculated separately).
- Step 5 Return the combined result of the newly arrived query transaction.
- the streaming analysis algorithm can convert real-time analysis calculations into a small amount of incremental calculations in units of time windows, and as the time window moves backwards, each incremental analysis in any calculation cycle The result is merged with the last full analysis result to obtain a new full analysis result, so that the real-time analysis results of the stream computing that the user has built can be completed with a relatively stable and predictable constant execution time, and stream computing
- the calculation efficiency of the system will not change due to the increase of the data volume, and can be kept within a small and stable execution time.
- it can be automatically determined according to the busyness of system resources and the periodic changes of the query business initiated by the user. Adjust the size of the time window and calculation cycle to further optimize the resource usage efficiency of the HTASP system.
- the processing flow of the analytical transaction by the HTAP system and the HTASP system is respectively introduced, which mainly involves the full + incremental two-tier computing model and the streaming computing model.
- the level division standard after dividing the full-time data into data with different degrees of hotness and coldness, can also optimize the storage model of the storage cluster based on the degree of hotness and coldness of the data.
- Figure 13 is a statistical chart of the proportion of visits to hot and cold data and the distribution of data volume provided by the embodiment of the present application. As shown in 1300, it can be found that 8% of the active data volume in one month undertakes 91% of the visits volume, while the access volume of cold data beyond one year only occupies 1% of the total access volume. Therefore, in order to better build the distributed architecture of the HTAP system or HTASP system based on the full-time database, so that the system can continuously provide stable access quality to the outside world, according to the different levels of hot and cold data, different access frequency Data is stored in different levels of storage media, which can not only meet the user's SLA performance requirements, but also reduce the storage cost of the database, thereby providing a high-quality, low-cost storage solution.
- Hot data stored in enhanced SSD (Solid State Disk, solid state drive) disk, and based on the Raft consistency protocol to save a master copy and two slave copies, to ensure high availability and high reliability of data;
- enhanced SSD Solid State Disk, solid state drive
- Warm data stored in SSD disks, and based on the Raft consensus protocol, one master copy and two slave copies are saved to ensure high availability and reliability of data;
- Residual verification is used to reduce the capacity synthesis of historical state data, thus bringing a reliability of 99.9999999999%, which means that there is a probability of 0.0000000001% loss of data, and even if the disk has bad sectors, the remaining data can be calculated according to the above erasure code Part, so as to calculate the content of the wrong part.
- Archived data use CAS (Content-Addressable Storage, content addressable storage) method for cloud storage
- SATA disk is a cheap storage device that can save archived data at a very low cost, and uses erasure codes instead of multiple copies. Increase storage density and use hard drives with programmable power-off.
- the user can flexibly set the HOT_COLD_DATA_LEVEL parameter of the division standard HOT_COLD_DATA_LEVEL according to its specific business, so as to configure appropriate storage media for the data corresponding to the cold and hot levels.
- an HTAP system capable of supporting online analysis functions for any period of time is constructed based on a full-time database, and a streaming computing capability, namely the HTASP system, is provided on the basis of the HTAP system.
- the full use of the time dimension of data can effectively decouple the OLTP cluster and the OLAP cluster, enabling the OLTP cluster to provide online transaction and analysis functions for the latest data on the master copy, and the OLAP cluster to provide online real-time analysis on the slave copy and historical data.
- Analysis and processing functions enable two clusters to calculate and operate independently of each other for the same data.
- management users can determine the real-time degree parameters of online analysis according to their own business needs, so as to not only meet business needs, but also quantify the maximum delay allowed between OLTP clusters and OLAP clusters.
- the real-time analysis mode of full + incremental calculation is introduced, which can save the use of computing resources to the greatest extent, so as to achieve the best real-time analysis effect, and, because the time state is for each record level
- the division of the time window dimension can be added to each record in the time state of the record granularity, so as to provide streaming computing capabilities to the outside world in combination with the original data characteristics, so as to achieve the task division of the smallest granularity at the record level and improve computing efficiency.
- users can customize the division standard of hot and cold grades to improve the overall service capability of the database system, and reduce the overall usage of the database system through partitioned storage of different storage devices and different storage media at different levels. Cost, because every time the full-time data is modified, a corresponding historical record (that is, historical data) will be generated.
- Fig. 14 is a schematic structural diagram of a transaction processing device provided by an embodiment of the present application, please refer to Fig. 14, the device includes:
- the determining module 1401 is configured to determine a first moment from the query time period of the transaction to be analyzed, the first moment is the moment when the historical state data is dumped last, and the historical state data is consistent with the transaction to be analyzed The data of the data query condition;
- the first acquiring module 1402 is configured to acquire a full analysis result based on the historical state data, where the full analysis result is a result obtained by processing the historical state data based on the analysis semantics of the transaction to be analyzed;
- the second acquisition module 1403 is configured to acquire incremental analysis results based on a specific change that occurs on the current state data corresponding to the historical state data, the specific change being from the first moment to the query time period Changes that occur between the end times of , the incremental analysis result is the result of processing the data obtained from the specific change based on the analysis semantics;
- the third obtaining module 1404 is configured to obtain the final analysis result of the transaction to be analyzed based on the full analysis result and the incremental analysis result.
- the device provided in the embodiment of this application calculates a full analysis result for the historical state data before the first moment by determining the query time period for the transaction to be analyzed, and using the first moment as a time point boundary within the query time period , calculate an incremental analysis result for the data obtained after the specific change after the first moment, combine the full analysis result and the incremental analysis result, and finally get the final analysis result, so that for any one on the full temporal time axis In the query time period, it can provide online real-time analysis and processing services, and is not limited to only providing analysis functions for the latest data, thus greatly improving the transaction processing performance of database systems such as HTAP systems or HTASP systems.
- the first acquiring module 1402 includes:
- a query unit configured to query whether to cache the intermediate analysis results of the first historical state data visible at a second moment, where the second moment is any moment between the start moment of the query time period and the first moment;
- a determining unit configured to determine the intermediate analysis result as the full analysis result if the intermediate analysis result has been cached
- the first generating unit is configured to generate the full analysis result of the second historical state data visible at the first moment without caching the intermediate analysis result.
- the first generation unit includes:
- the determining subunit is used to determine the hot and cold level of the second historical state data, and the hot and cold level is used to represent the frequency of operations on the second historical state data within the historical time period;
- the reading subunit is used to read the second historical state data from the storage device corresponding to the hot and cold level;
- the processing subunit is configured to process the second historical state data based on the analysis semantics to obtain the full analysis result.
- the determining subunit is used for:
- the hot and cold levels are determined.
- historical state data with different cold and hot levels are stored in different storage devices or different storage media of the same storage device.
- the second obtaining module 1403 includes:
- An acquisition unit configured to acquire the data obtained by the specific change based on the log generated by the specific change
- the second generation unit is used to compare the historical state data with the same primary key identifier and the data obtained by a specific change, and generate the incremental analysis result.
- the incremental analysis result includes an analysis result before a specific change and an analysis result after a specific change
- the second generating unit is used for:
- the data query condition involves the specific changed field, generate the analysis result before the specific change for the second historical state data visible at the first moment; generate the target data obtained by the specific change visible at the end time Analysis results after specific changes.
- the third acquiring module 1404 is configured to:
- the first analysis result is added to the specific changed analysis result to obtain the final analysis result.
- the determination module 1401 is also used to:
- the transaction to be analyzed does not specify the end time of the query time period
- the time before the current time before the current time and reaching the real-time degree parameter is determined as the end time
- the real-time degree parameter is used to characterize the analytical transaction in the database system The maximum latency supported by the processed data.
- the second acquiring module 1403 is also configured to:
- the third acquiring module 1404 is also configured to:
- the transaction processing device when the transaction processing device provided by the above-mentioned embodiments processes transactions, it only uses the division of the above-mentioned functional modules as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional modules according to needs.
- the internal structure of the computing device is divided into different functional modules to complete all or part of the functions described above.
- the transaction processing device and the transaction processing method embodiments provided by the above embodiments belong to the same idea, and the specific implementation process thereof is detailed in the transaction processing method embodiments, which will not be repeated here.
- FIG. 15 is a schematic structural diagram of a computing device provided by an embodiment of the present application.
- the computing device is a terminal 1500 as an example for illustration.
- the device types of the terminal 1500 include: smart phones, tablet computers, MP3 players (Moving Picture Experts Group Audio Layer III, moving picture experts compression standard audio layer 3), MP4 (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV, motion picture experts compress standard audio layer 4) player, laptop or desktop computer.
- the terminal 1500 may also be called user equipment, portable terminal, laptop terminal, desktop terminal and other names.
- the terminal 1500 includes: a processor 1501 and a memory 1502 .
- the processor 1501 includes one or more processing cores, such as a 4-core processor, an 8-core processor, and the like.
- Processor 1501 can adopt at least one hardware form in DSP (Digital Signal Processing, digital signal processing), FPGA (Field-Programmable Gate Array, field programmable gate array), PLA (Programmable Logic Array, programmable logic array) accomplish.
- the processor 1501 includes a main processor and a coprocessor, and the main processor is a processor for processing data in a wake-up state, also called a CPU (Central Processing Unit, central processing unit);
- a coprocessor is a low-power processor for processing data in a standby state.
- the processor 1501 is integrated with a GPU (Graphics Processing Unit, image processor), and the GPU is used for rendering and drawing the content that needs to be displayed on the display screen.
- the processor 1501 further includes an AI (Artificial Intelligence, artificial intelligence) processor, where the AI processor is used to process computing operations related to machine learning.
- AI Artificial Intelligence, artificial intelligence
- memory 1502 includes one or more computer-readable storage media, which are optionally non-transitory.
- the memory 1502 also includes high-speed random access memory and non-volatile memory, such as one or more magnetic disk storage devices and flash memory storage devices.
- the non-transitory computer-readable storage medium in the memory 1502 is used to store at least one program code, and the at least one program code is used to be executed by the processor 1501 to implement the various embodiments provided in this application. Transaction method.
- the terminal 1500 may optionally further include: a peripheral device interface 1503 and at least one peripheral device.
- the processor 1501, the memory 1502, and the peripheral device interface 1503 can be connected through buses or signal lines.
- Each peripheral device can be connected to the peripheral device interface 1503 through a bus, a signal line or a circuit board.
- the peripheral device includes: at least one of a radio frequency circuit 1504 , a display screen 1505 , a camera component 1506 , an audio circuit 1507 , a positioning component 1508 and a power supply 1509 .
- the terminal 1500 further includes one or more sensors 1510 .
- the one or more sensors 1510 include, but are not limited to: an acceleration sensor 1511 , a gyro sensor 1512 , a pressure sensor 1513 , a fingerprint sensor 1514 , an optical sensor 1515 and a proximity sensor 1516 .
- FIG. 15 does not constitute a limitation to the terminal 1500, and may include more or less components than shown in the figure, or combine some components, or adopt a different component arrangement.
- FIG. 16 is a schematic structural diagram of a computing device provided by an embodiment of the present application.
- the computing device 1600 may have relatively large differences due to different configurations or performances.
- the computing device 1600 includes one or more processors (Central Processing Units, CPU) 1601 and one or more memories 1602, wherein at least one computer program is stored in the memory 1602, and the at least one computer program is loaded and executed by the one or more processors 1601 to realize the above-mentioned various embodiments provided Transaction method.
- the computing device 1600 also has components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output.
- the computing device 1600 also includes other components for implementing device functions, which will not be described in detail here.
- a computer-readable storage medium such as a memory including at least one computer program
- the at least one computer program can be executed by a processor in the terminal to complete the transaction processing method in each of the above-mentioned embodiments .
- the computer-readable storage medium includes ROM (Read-Only Memory, read-only memory), RAM (Random-Access Memory, random-access memory), CD-ROM (Compact Disc Read-Only Memory, read-only disc), Magnetic tapes, floppy disks, and optical data storage devices, etc.
- a computer program product or computer program comprising one or more pieces of program code stored in a computer readable storage medium.
- One or more processors of the computing device can read the one or more program codes from the computer-readable storage medium, and the one or more processors execute the one or more program codes, so that the computing device can execute to complete The transaction processing method in the above embodiment.
- the program is stored in a computer-readable storage medium.
- the above-mentioned The storage medium is a read-only memory, a magnetic disk or an optical disk, and the like.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本申请公开了一种事务处理方法、装置、计算设备及存储介质,属于数据库技术领域。本申请通过针对待分析事务确定查询时间段,并以第一时刻作为查询时间段内的一个时间点分界线,对第一时刻之前的历史态数据计算出一个全量分析结果,对第一时刻之后特定变更得到的数据计算出一个增量分析结果,将全量分析结果和增量分析结果两者结合,最终得到最终分析结果,使得针对全时态的时间轴上的任一个查询时间段,都能够提供在线实时分析和处理服务,如金融场景下支持查询任意时间段内的流水,又如智慧交通场景下支持查询任意时间段内空闲的停车位总数等,能够大大提升数据库***的事务处理性能。
Description
本申请要求于2021年11月5日提交中国专利局、申请号202111305207.6、申请名称为“事务处理方法、装置、计算设备及存储介质”的中国专利申请的优先权,其全部内容通过引用结合在本申请中。
本申请涉及数据库技术领域,特别涉及一种事务处理方法、装置、计算设备及存储介质。
随着数据库技术的发展,在数据仓库、大数据分析等场景中,通常会采用基于HTAP(Hybrid Transaction/Analytical Processing,混合交易/分析处理)架构的数据处理***。在HTAP***内包括OLTP(On-Line Transaction Processing,在线交易处理)集群和OLAP(On-Line Analytical Processing,在线分析处理)集群,OLTP集群用于处理短时用户交易型事务,具有事务执行时间短、并发度高等特点,对单位时间吞吐量要求较高;OLAP集群用于处理复杂分析型事务,具有事务执行时间长、资源占用多、处理的数据量大等特点。
在上述HTAP***中,OLAP集群和OLTP集群之间配置有数据迁移模块,用来将OLTP集群内最新的相关数据迁移到OLAP集群中,以保证OLAP集群和OLTP集群对数据分析和处理的一致性,换言之,相关技术提供的HTAP***仅支持对最新数据提供在线交易和分析功能,分析处理能力较差。
发明内容
本申请实施例提供了一种事务处理方法、装置、计算设备及存储介质,能够并不局限于仅对最新数据提供分析功能、而是针对全时态的时间轴上的任一个查询时间段提供在线实时分析和处理服务,提高了分析处理能力。该技术方案如下:
一方面,提供了一种事务处理方法,该方法由计算设备执行,该方法包括:
从待分析事务的查询时间段中确定第一时刻,所述第一时刻为对历史态数据最近一次完成转储的时刻,所述历史态数据为符合所述待分析事务的数据查询条件的数据;
基于所述历史态数据,获取全量分析结果,所述全量分析结果为基于所述待分析事务的分析语义对所述历史态数据进行处理所得的结果;
基于在所述历史态数据对应的当前态数据上发生的特定变更,获取增量分析结果,所述特定变更为在所述第一时刻起至所述查询时间段的结束时刻之间发生的变更,所述增量分析结果为基于所述分析语义对所述特定变更得到的数据进行处理所得的结果;
基于所述全量分析结果和所述增量分析结果,获取所述待分析事务的最终分析结果。
一方面,提供了一种事务处理装置,该装置部署在计算设备上,该装置包括:
确定模块,用于从待分析事务的查询时间段中确定第一时刻,所述第一时刻为对历史态数据最近一次完成转储的时刻,所述历史态数据为符合所述待分析事务的数据查询条件的数据;
第一获取模块,用于基于所述历史态数据,获取全量分析结果,所述全量分析结果为基于所述待分析事务的分析语义对所述历史态数据进行处理所得的结果;
第二获取模块,用于基于在所述历史态数据对应的当前态数据上发生的特定变更,获取增量分析结果,所述特定变更为在所述第一时刻起至所述查询时间段的结束时刻之间发生的变更,所述增量分析结果为基于所述分析语义对所述特定变更得到的数据进行处理所得的结果;
第三获取模块,用于基于所述全量分析结果和所述增量分析结果,获取所述待分析事务的最终分析结果。
特定变更特定变更特定变更特定变更特定变更特定变更特定变更特定变更特定变更特定变更特定变更特定变更特定变更特定变更最终分析结果待分析事务待分析事务特定变更一方面,提供了一种计算设备,该计算设备包括一个或多个处理器和一个或多个存储器,该一个或多个存储器中存储有至少一条计算机程序,该至少一条计算机程序由该一个或多个处理器加载并执行以实现如上述任一种可能实现方式的事务处理方法。
一方面,提供了一种存储介质,该存储介质中存储有至少一条计算机程序,该至少一条计算机程序由处理器加载并执行以实现如上述任一种可能实现方式的事务处理方法。
一方面,提供一种计算机程序产品或计算机程序,所述计算机程序产品或所述计算机程序包括一条或多条程序代码,所述一条或多条程序代码存储在计算机可读存储介质中。计算设备的一个或多个处理器能够从计算机可读存储介质中读取所述一条或多条程序代码,所述一个或多个处理器执行所述一条或多条程序代码,使得计算设备能够执行上述任一种可能实施方式的事务处理方法。
本申请实施例提供的技术方案带来的有益效果至少包括:
通过针对待分析事务确定查询时间段,并以第一时刻作为查询时间段内的一个时间点分界线,对第一时刻之前的历史态数据计算出一个全量分析结果,对第一时刻之后特定变更得到的数据计算出一个增量分析结果,将全量分析结果和增量分析结果两者结合,最终得到最终分析结果,使得针对全时态的时间轴上的任一个查询时间段,都能够提供在线实时分析和处理服务,而并不局限于仅对最新数据提供分析功能,从而大大提升了HTAP***或HTASP***等数据库***的事务处理性能。
为了更清楚地说明本申请实施例中的技术方案,下面将对实施例描述中所需要使用的附图作简单地介绍,显而易见地,下面描述中的附图仅仅是本申请的一些实施例,对于本领域普通技术人员来讲,在不付出创造性劳动的前提下,还能够根据这些附图获得其他的附图。
图1是本申请实施例提供的一种HTAP***的实施环境架构图;
图2是本申请实施例提供的一种事务处理流程的逻辑框图;
图3是本申请实施例提供的一种事务处理方法的实施环境示意图;
图4是本申请实施例提供的一种事务处理方法的流程图;
图5是本申请实施例提供的一种历史态数据与冷热温数据的关系示意图;
图6是本申请实施例提供的一种主从副本机制内增加实时分析从副本的原理性示意图;
图7是本申请实施例提供的一种冷热数据的存储模型的原理性示意图;
图8是本申请实施例提供的一种全时态的数据记录的存储形态原理图;
图9是本申请实施例提供的一种事务处理方法的流程图;
图10是本申请实施例提供的一种实时分析模型的原理性流程图;
图11是本申请实施例提供的一种流式计算模型的数据记录的存储形态的原理性示意图;
图12是本申请实施例提供的一种HTASP***的实施环境架构图;
图13是本申请实施例提供的一种冷热数据的访问量比例和数据量分布的统计图;
图14是本申请实施例提供的一种事务处理装置的结构示意图;
图15是本申请实施例提供的一种计算设备的结构示意图;
图16是本申请实施例提供的一种计算设备的结构示意图。
为使本申请的目的、技术方案和优点更加清楚,下面将结合附图对本申请实施方式作进一步地详细描述。
本申请中术语“第一”“第二”等字样用于对作用和功能基本相同的相同项或相似项进行区分,应理解,“第一”、“第二”、“第n”之间不具有逻辑或时序上的依赖关系,也不对数量和执行顺序进行限定。
本申请中术语“至少一个”是指一个或多个,“多个”的含义是指两个或两个以上,例如,多个第一位置是指两个或两个以上的第一位置。
在介绍本申请实施例之前,需要引入一些云技术领域内的基本概念。
云技术(Cloud Technology):是指在广域网或局域网内将硬件、软件、网络等系列资源统一起来,实现数据的计算、储存、处理和共享的一种托管技术,也即是基于云计算商业模式应用的网络技术、信息技术、整合技术、管理平台技术、应用技术等的总称,可以组成资源池,按需所用,灵活便利。云计算技术将变成云技术领域的重要支撑。技术网络***的后台服务需要大量的计算、存储资源,如视频网站、图片类网站和更多的门户网站。伴随着互联网行业的高度发展和应用,将来每个物品都有可能存在自己的识别标志,都需要传输到后台***进行逻辑处理,不同程度级别的数据将会分开处理,各类行业数据皆需要强大的***后盾支撑,均能通过云计算来实现。
云存储(Cloud Storage):是在云计算概念上延伸和发展出来的一个新的概念,分布式云存储***(以下简称存储***)是指通过集群应用、网格技术以及分布存储文件***等功能,将网络中大量各种不同类型的存储设备(存储设备也称之为存储节点)通过应用软件或应用接口集合起来协同工作,共同对外提供数据存储和业务访问功能的一个存储***。
数据库(Database):简而言之可视为一种电子化的文件柜——存储电子文件的处所,用户可以对文件中的数据进行新增、查询、更新、删除等操作。所谓“数据库”是以一定方式储存在一起、能与多个用户共享、具有尽可能小的冗余度、与应用程序彼此独立的数据集合。
以下,对本申请实施例所涉及的术语进行解释说明。
计算设备:即计算节点或计算引擎节点,主要完成SQL(Structured Query Language,结构化查询语言)层的计算功能,亦可称为SQL引擎(SQL Engine)。换言之,计算设备是指数据库(通常指集群数据库,对于单机数据库来说单机设备本身就是一个计算设备)中用来处理用户特定计算请求(简称用户请求),并主要负责执行用户请求的节点设备。
存储设备:即存储节点或存储引擎节点(Storage Engine),主要完成事务(Transaction)及存储功能。换言之,存储设备是指数据库中用来处理存储数据记录,并完成事务执行与提交的节点。
在线交易处理(On-Line Transaction Processing,OLTP)***:在线交易处理***主要用来处理短时内用户发起的交易型事务,交易型事务的特征为:事务执行时间短、并发度高,因此对于在线交易处理***在单位时间内的吞吐量要求高。
在线分析处理(On-Line Analytical Processing,OLAP)***:在线分析处理***,主要用来处理复杂的分析型事务,分析型事务的特征为:事务执行时间长、对于资源的占用较多、处理的数据量大。
混合交易/分析处理(Hybrid Transaction/Analytical Processing,HTAP)***:这是一种混合在线交易和在线分析这两种负载的数据处理***,可看作是一种融合***,期望在一个***内既支持交易型事务也支持分析型事务。
混合交易/分析/流式计算处理(Hybrid Transaction/Analytical/Stream Processing,HTASP)***:是指基于HTAP***,引入流式计算能力的数据处理***。
抽取-转换-加载(Extract-Transform-Load,ETL):用来描述将数据从源端经过抽取和转换以加载至目的端的过程。ETL方式较常用在数据仓库,但ETL可操作的对象并不限于数据仓库。
Paxos算法:是一种基于消息传递且具有高度容错特性的共识(Consensus)算法。
Raft算法:是一种用于替代Paxos的共识算法。相比于Paxos算法来说,Raft算法的目标是提供更清晰的逻辑分工,使得Raft算法本身能被更好地理解,同时Raft算法的安全性更高,并能提供一些额外的特性。Raft算法能为在计算机集群之间部署有限状态机提供一种通用方法,并确保集群内的任意节点在某种状态转换上保持一致。
在介绍本申请实施例之前,需要引入一些数据库技术中的基本概念:
本申请实施例涉及的数据库存储有多个数据表,每个数据表可以用于存储数据记录。其中,该数据库可以为基于MVCC(Multi-Version Concurrency Control,多版本并发控制)的任一类型的数据库。在本申请实施例中,对该数据库的类型不作具体限定。需要说明的是,上述数据库中的数据基于状态属性,可以包括三种状态:当前态、过渡态和历史态,该三种状态合称为“数据的全态”,简称全态数据,全态数据中的各个不同状态属性,可以用于标识数据在其生命周期轨迹中所处的状态。
当前态(Current State):数据记录的最新版本的数据,是处于当前阶段的数据。处于当前阶段的数据的状态,称为当前态。
过渡态(Transitional State):不是数据记录的最新版本也不是历史版本,处于从当前态向历史态转变的过程中,处于过渡态的数据,称为半衰数据。
历史态(Historical State):数据记录在历史上的一个状态,其值是旧值,不是当前值。处于历史阶段的数据的状态,称为历史态。一个数据记录的历史态,可以有多个,反映了数据的状态变迁的过程。处于历史态的数据,只能被读取而不能被修改或删除。
需要说明的是,在MVCC机制下,数据的上述三种状态均存在,在非MVCC机制下,数据可以只存在历史态和当前态。在MVCC或封锁并发访问控制机制下,事务提交后的数据的新值处于当前态。以MVCC机制为例,当前活跃事务列表中最小的事务之前的事务生成的数据,其状态处于历史态。在封锁并发访问控制机制下,事务提交后,提交前的数据的值变为历史态的值,即数据记录的旧值处于历史态。而被读取的版本上尚有活跃事务(非最新相关事务)在使用,而由于最新相关事务修改了数据记录的值,其最新值已经处于一个当前态,被读取到的值相对当前态已经处于一个历史状态,因此,其数据状态介于当前态和历史态之间,所以称为过渡态。
例如,MVCC机制下,用户表(User表)的A账户余额从10元充值变为20元,然后消费了15元变为5元,此时金融B机构读取数据做检查事务一直进行中,A之后又充值20元变为25元,则25元为当前态数据,B正在读取到的5元为过渡态,其余的两个值20、10是历史上存在过的状态,都是历史态数据。
数据记录的可见性:数据记录的可见与否(即数据记录的可见性)是针对于事务而言的,某一条数据记录可能针对一些事务可见,针对一些事务不可见。在本申请实施例中,涉及到判断处于历史态的数据记录(即历史态数据)是否在待分析事务的查询时间段内可见,相关的可见性判断算法将在后续的实施例中说明,这里不做赘述。
在相关技术提供的HTAP***中包括OLTP集群和OLAP集群,OLTP集群用于处理短时用户交易型事务,OLAP集群用于处理复杂分析型事务。在OLAP集群和OLTP集群之间配置有数据迁移模块,用来将OLTP集群内最新的相关数据迁移到OLAP集群中,以保证OLAP集群和OLTP集群对数据分析和处理的一致性。由于在OLTP集群对数据记录进行特定变更之后,不提供对历史态数据的存储功能,即当一条数据记录从旧值修改为新值,旧值会被新值直接覆盖,此后OLAP集群如果收到一个对被修改的数据记录的分析型事务,需要通过数据迁移模块把OLTP集群中最新版本的数据记录同步到OLAP集群中,然后在同步后的数据记录上进行分析和处理,换言之,相关技术提供的HTAP***仅支持对最新数据提供在线交易和分析功能,无法针对时间轴上指定的任一时间点或时间段内的历史态数据提供在线分析功能。
有鉴于此,本申请实施例提供一种事务处理方法,针对HTAP***内存储的各个数据记录,提供在时间维度上全态数据的保存功能,即,在HTAP***内保存所有的当前态数据、过渡态数据和历史态数据,当一条数据记录被修改之后,最新版本的数据记录(当前态数据)和转变为历史版本的数据记录(历史态数据)都会被存储到存储集群的存储设备中,并且在HTAP***中还通过流式计算处理引擎,对外提供流式分析和处理功能,融合了流式计算处理引擎的HTAP***也称为HTASP***。
换言之,基于全时态数据的时间维度,提供一种融合流式计算处理引擎的HTASP***,在整个HTASP***中OLTP集群和OLAP集群能够使用同一份全时态数据完成交易或 分析,并且大大降低了由于数据迁移模块导致的OLTP集群和OLAP集群的耦合,使得OLTP集群和OLAP集群各自在处理事务时互相隔离、互不影响。
由于对每一条数据记录都增加了时间维度的相关信息,使得OLAP集群不止支持对最新版本的数据记录进行分析的功能,而且用户在自定义在线分析的实时程度参数之后,能够以记录级的细粒度来在历史态数据上进行分析,并且用户在自定义增量分析结果的时间窗和全量分析结果的计算周期之后,能够每隔一个计算周期进行一次全量分析,在每个计算周期内每隔一个时间窗进行一次增量分析,即提供并行计算和流式计算在最小粒度上的结合,实现了在同一份全时态数据上构建HTASP***并对外提供高质量的数据服务的功能。
需要说明的是,本申请实施例可应用于HTAP***、HTASP***等各类支持混合交易/分析的数据处理***,此外,本申请实施例还可应用于一种基于区块链技术的数据库***(以下简称为“区块链***”),上述区块链***在本质上属于一种去中心化式的分布式数据库***,采用共识算法保持区块链上不同计算设备所记载的账本数据一致,通过密码算法保证不同计算设备之间账本数据的加密传送以及不可篡改,通过脚本***来拓展账本功能,通过网络路由来进行不同计算设备之间的相互连接。
在区块链***中可以包括一条或多条区块链,区块链是一串使用密码学方法相关联产生的数据块,每一个数据块中包含了一批次网络交易的信息,用于验证其信息的有效性(防伪)和生成下一个区块。示意性地,在HTAP***下,由各个已完成的交易型事务的日志记录作为账本数据,并构建区块链***,在每个计算设备中各自存储一批次的网络交易信息(即日志记录),将网络交易信息上链之后能够实现任意时刻的历史交易信息(历史态数据)的可追溯性。
区块链***中计算设备之间可以组成点对点(Peer To Peer,P2P)网络,P2P协议是一个运行在传输控制协议(Transmission Control Protocol,TCP)协议之上的应用层协议。在区块链***中,任一计算设备可以具备如下功能:1)路由,计算设备具有的基本功能,用于支持计算设备之间的通信;2)应用,用于部署在区块链中,根据实际业务需求而实现特定业务,记录实现功能相关的数据形成账本数据,在账本数据中携带数字签名以表示数据来源,将账本数据发送至区块链***中的其他计算设备,供其他计算设备在验证账本数据来源以及完整性成功时,将账本数据添加至临时区块中,其中,应用实现的业务可以包括钱包、共享账本、智能合约等;3)区块链,包括一系列按照先后的时间顺序相互接续的区块,新区块一旦加入到区块链中就不会再被移除,区块中记录了区块链***中计算设备提交的账本数据。
在一些实施例中,每个区块中可以包括本区块存储交易记录的哈希值(本区块的哈希值)以及前一区块的哈希值,各区块通过哈希值连接形成区块链,另,区块中还可以包括有区块生成时的时间戳等信息。
图1是本申请实施例提供的一种HTAP***的实施环境架构图。参见图1,在HTAP***100中包括OLTP集群110和OLAP集群120。
OLTP集群110用于提供在线交易功能,在OLTP集群110中包括多个TP计算节点,该TP计算节点用于处理交易型事务,当交易型事务对数据记录修改完毕,会将产生的历史 态数据通过存储接口转储到分布式文件***130中。每个TP计算节点都配置有各自的数据库引擎,每个TP计算节点可以是单机设备或者一主两备集群设备等,本申请实施例不对TP计算节点的类型进行具体限定。
OLAP集群120用于提供在线分析功能,在OLAP集群120中包括多个AP计算节点,该AP计算节点用于处理分析型事务,该分析型事务包括但不限于:查询历史态数据或者在查询得到的历史态数据上进一步处理以得到某个分析结果,例如,金融场景下,该分析型事务T1为用户查询过去一个月内的账单流水总额,则这一分析型事务T1需要从分布式文件***130中查询用户过去一个月内进行交易记录,并对花费的金额进行求和(SUM),得到的账单流水总额就是该分析型事务T1的分析结果,又例如,智慧交通场景下,该分析型事务T2为用户查询过去一周内某市内每个行政区域中新增停车位的总量,则这一分析型事务T2需要以该市内的每个行政区域为单位,从分布式文件***130中查询每个行政区域内新增停车位的***记录,并对***的新增停车位的数量进行求和(SUM),得到的每个行政区域内新增停车位的总量所构成的集合就是分析型事务T2的分析结果。
OLTP集群110和OLAP集群120都能够通过存储接口访问到分布式文件***130,OLTP集群110、OLAP集群120两者均能够和分布式文件***130之间通过有线或无线通信方式进行直接或间接地连接,本申请实施例对此不进行具体限定。
分布式文件***130能够对OLTP集群110和OLAP集群120提供无限存储功能,分布式文件***130中包括多个存储节点即存储设备,可按照冷热等级对全时态数据进行分区,将冷热等级相同的数据划分到相同分区下的存储设备中进行存储,或者,将冷热等级相同的数据划分到相同分区下的存储设备的同一种存储介质中进行存储。例如,该分布式文件***130包括但不限于:HDFS(Hadoop Distributed File System,Hadoop分布式文件***)、Ceph(一种Linux***下的分布式文件***)、Alluxio(一种基于内存的分布式文件***)等。
在一些实施例中,由于TP计算设备提供在线交易功能,当任一交易型事务提交完成的时刻,在生成新的当前态数据的同时,也会生成与该当前态数据所对应的历史态数据,而由于历史态数据会占用较多存储空间,而历史态数据又具有保存价值,因此会将历史态数据转储到分布式文件***中对应的存储设备上面进行持久化存储,在对历史态数据转储完毕之后,TP计算节点可删除本次转储的历史态数据,以清理更多的存储空间。
在一些实施例中,当TP计算节点将历史态数据成功转储到存储设备上之后,还可将本次转储的历史态数据的元数据注册到元数据(Meta Data,MD)管理器中,便于存储设备基于该元数据管理器统计已储备的历史态数据的元信息。
在一些实施例中,用户负载到达HTAP***100后,数据库引擎对该用户负载(即用户事务,包括交易型事务或者分析型事务)进行解析,基于SQL路由(Structured Query Language Router,SQL Router,简称SR)层中提供的查询语句、查询操作的语义和元数据,将该用户负载路由到OLTP集群110或OLAP集群120中进行处理。例如,OLTP集群110主要提供对当前态数据的查询和修改服务,OLAP集群120则主要提供对历史态数据的查 询和分析服务(不支持修改或删除历史态数据)。其中,查询操作的语义是根据查询语句分析得到的操作意图,例如,WHERE子句的条件可以表示WHERE子句的意图。
在上述架构中,每个TP计算节点或AP计算节点(统称为计算节点)所对应的一个或多个数据库的数据库实例集合称为一个SET(集合),可选地,如果计算节点为单机设备,那么该单机设备的数据库实例为一个SET,如果计算节点为一主两备集群设备,那么该计算节点的SET为主机数据库实例以及两个备机数据库实例的集合,此时基于云数据库(Cloud Database)的强同步技术来保证主机的数据与备机的副本数据之间的一致性,可选地,每个SET支持线性扩容,以应付大数据场景下的业务处理需求。
在一些实施例中,OLTP集群110或OLAP集群120通过分布式协调***(例如ZooKeeper)实现对TP计算节点或AP计算节点的管理,例如通过ZooKeeper使得某一个TP计算节点或AP计算节点失效(也即是将该TP计算节点或AP计算节点从OLTP集群110或OLAP集群120中删除)。
在一些实施例中,上述HTAP***100可整体视为一个对外提供混合交易和分析功能的服务器,服务器可以是独立的物理服务器,或者是多个物理服务器构成的服务器集群或者分布式***,或者是提供云服务、云数据库、云计算、云函数、云存储、网络服务、云通信、中间件服务、域名服务、安全服务、CDN(Content Delivery Network,内容分发网络)、以及大数据和人工智能平台等基础云计算服务的云服务器。
在一些实施例中,用户负载是用户由终端向HTAP***100发起的用户事务,其中,用户侧的终端也称为用户设备、终端设备、用户终端、移动终端、智能终端、通信设备等。可选地,终端的设备类型包括:智能手机、平板电脑、笔记本电脑、台式计算机、智能音箱、智能手表、车载终端、智能家电、智能语音交互设备等,但并不局限于此。
以下,对本申请实施例的***架构下事务的逻辑流程进行介绍。
图2是本申请实施例提供的一种事务处理流程的逻辑框图,如200所示,在本申请实施例所适用的HTAP***中,HTAP***接收到用户发起的一个SQL语句,将该SQL语句输入到查询解析器(Query Parser)中,先后对该SQL语句进行词法分析(Lexical Analysis)、句法分析(Syntax Analysis)、时态翻译(Temporal Translation)和语义检查(Semantic Check)。在句法分析和时态翻译过程中,需要先对查询语句(SELECT Query)按照查询树进行重写,再将重写后的查询语句输入到优化器(Optimizer)中进行查询优化,接着从事务状态管理器(Transaction Status Manager)中对该查询语句的查询事务申请一个事务状态(Transaction Status),再将优化器输出的查询语句和事务状态管理器输出的事务状态一起输入到执行器(Executor)中进行查询执行。
在HTAP***的存储集群(例如分布式文件***)中,将当前态数据和历史态数据分别存储在不同的存储设备中。例如,将冷热等级较高(例如访问频率和修改频率都很高)的数据记录存储在当前态数据存储设备中,随着时间的推移,如果当前态数据存储设备中的部分数据记录的冷热等级下降,可将冷热等级下降的该部分数据记录通过ETL方式进行抽取、转换,并最终加载到历史态数据存储设备中进行持久化存储。可选地,历史态数据存储设备可对加载到的历史态数据进行压缩,比如对某一条数据记录的各个历史版本,仅 存储一部分历史版本的全量数据,剩余的另一部分历史版本以与上一个全量数据之间的增量数据的方式进行存储。
在上述处理流程中,每一条数据记录都从***到更新,完整的保留了按照时间维度排列的各个版本(包括当前版本和一个或多个历史版本),这些各个版本的数据反映了从***数据记录以来相关事务每一次对该数据记录进行的操作,通过在全时态数据的基础上搭建HTAP***或者HTASP***,能够只基于一套全时态数据实现HTAP***或者HTASP***,并且使得OLTP集群和OLAP集群在运行时基于用户设定的实时程度参数,来最大化地降低OLTP集群和OLAP集群的耦合程度,即最小化OLTP集群和OLAP集群之间的相互影响。
通过全时态数据,能够很方便地向HTAP***引入流式计算能力以转变成HTASP***,并且能够在记录级别对数据记录的各个版本按照时间轴进行细粒度地划分,从而以更小粒度来划分实时分析的大任务,提高更高水平的扩张能力。
以时间窗为基础,为整个HTAP***或者HTASP***提供全量与增量合并得到分析结果的执行模式,能够最大程度上节约计算资源,提高OLAP集群的分析处理能力。
在分布式数据库集群中,通过以数据块(block)为单位来对历史态数据进行转储,能够减少转储历史态数据而对OLTP集群所造成的影响,从而保证不影响OLTP集群的正常工作。
图3是本申请实施例提供的一种事务处理方法的实施环境示意图,如图3所示,示出了一种基于HTAP架构的实时分析***或实时分析模型,由于在该实时分析***中引入了全时态数据,原本传统HTAP是不支持在历史态数据上做分析的,且每次对当前态数据都需要进行全量分析(相当于一层计算模式),但本申请实施例提供的实时分析***,由于实现了历史态数据的转储,从而支持在历史态数据上做分析,且无需每次对当前态数据都进行全量分析,而是能够转换成之前在某个时间点计算的全量分析结果作为基线,结合从该时间点到实时分析事务所指定的时间点(如果用户未指定,可根据***的实时程度参数来计算得到)计算的增量分析结果,合并得到最终的最终分析结果,即提供了一种两层实时分析模型。
在该实时分析***中,包含计算层310和分层分布式共享存储***320,在计算层310中包括OLTP集群311和OLAP集群312,OLTP集群311通常针对在当前态数据和过渡态数据上发起的短时交易型事务提供在线交易功能,OLAP集群312通常针对在历史态数据上发起的复杂分析型事务提供在线分析功能。OLTP集群311和OLAP集群312两者均能够访问到分层分布式共享存储***320中存储的全时态数据,示意性地,按照全时态数据的冷热等级,将***内所有的数据划分为如下几种分区:热数据、温数据、冷数据等。在后续实施例中将对冷热等级的划分方式进行详细介绍,这里不做赘述。
在一些实施例中,如果到达OLAP集群312的分析型事务不涉及到热数据,那么在该分析型事务的执行过程中无需与OLTP集群311产生任何交互。如果到达OLAP集群312的分析型事务涉及到热数据,并且这部分最新修改的热数据尚未转储到分层分布式共享存 储***320中,那么OLAP集群312则根据该分析型事务的业务需求,以时态为条件(即查询时间段)从OLAP集群311中查询尚未转储的热数据。
图4是本申请实施例提供的一种事务处理方法的流程图。参见图4,该实施例适用于HTAP***、HTASP***或者区块链***等支持实时分析的数据处理***,该事务处理方法由OLAP集群中的任一计算设备执行,该实施例包括下述步骤:
401、计算设备从待分析事务的查询时间段中确定第一时刻,该第一时刻为对历史态数据最近一次完成转储的时刻,历史态数据为符合待分析事务的数据查询条件的数据。
其中,该待分析事务是任一种可能的分析型事务,例如,该分析型事务包括但不限于:查询事务,只读事务,或者在查询结果上进行分析或处理的事务等。例如,金融场景下一种示例性的分析型事务为:查询用户过去一个月内的账单流水总额;又例如,智慧交通场景下一种示例性的分析型事务为:查询过去一周内某市内每个行政区域中新增停车位的总量。
其中,该查询时间段是由一个开始时刻和一个结束时刻所构成的时间区间。该查询时间段的开始时刻或结束时刻可由用户请求来自定义,如果用户请求中未定义开始时刻或结束时刻,则由数据库引擎来对待分析事务配置该查询时间段。
其中,该第一时刻是指所有符合该待分析事务的数据查询条件的历史态数据最近一次完成转储的时刻,该数据查询条件是解析该待分析事务的SQL语句得到的,上述转储过程是指:将OLTP集群产生的该历史态数据迁移到存储集群的存储设备上进行存储。
在一些实施例中,用户在终端上登录应用客户端,触发应用客户端生成用户请求,例如,金融场景下,用户请求为请求查询用户在过去一个月内的账单流水总额,又例如,智慧交通场景下,用户请求为请求查询过去一周内某市内每个行政区域中新增停车位的总量,本申请实施例不对用户请求的类型进行具体限定。应用客户端在生成用户请求之后,可调用API(Application Programming Interface,应用程序编程接口)将该用户请求发送至计算设备,比如,该API可以是MySQL API(一种关系型数据库***提供的API)。
在一些实施例中,计算设备接收应用客户端的任一用户请求,解析该用户请求的头字段,当该头字段指示该用户请求为分析型事务时,解析该用户请求的数据字段,得到该用户请求中携带的SQL语句(例如SELECT Query语句等),将该SQL语句到查询解析器中,通过该查询解析器对该SQL语句进行词法分析、句法分析、时态翻译和语义检查,得到该SQL语句的数据查询条件和分析语义(比如,求和、求平均、排序等),另外,如果SQL语句中指定了查询时间段的开始时刻,则获取该开始时刻,如果SQL语句中指定了查询时间段的结束时刻,则获取该结束时刻。接着,基于该SQL语句构建对应的该待分析事务,计算设备为该待分析事务创建一个进程或线程,或者复用已创建的某个进程或线程,并通过该进程或线程来处理该待分析事务。
示意性地,当该待分析事务指定了该查询时间段的开始时刻时,则以待分析事务指定的开始时刻为准;当该待分析事务未指定该查询时间段的开始时刻时,在一种可能的实现方式中,以待分析事务所查询的数据的最新版本的生效时刻为准,也即将符合该数据查询条件的数据的最新版本的生效时刻确定为该开始时刻;当涉及到查询多个数据的时候,对 每个数据都以各自最新版本的生效时刻来选择对应的可见版本进行分析即可,在一种可能的实现方式中,以待分析事务所查询的数据的***版本的生效时刻为准,也即将符合该数据查询条件的数据的***版本的生效时刻确定为该开始时刻;当涉及到查询多个数据的时候,对每个数据都以各自***版本的生效时刻来选择对应的可见版本进行分析即可,本申请实施例对开始时刻的确定方式不进行具体限定。
示意性地,当该待分析事务指定了该查询时间段的结束时刻时,则以待分析事务指定的结束时刻为准;当该待分析事务未指定该查询时间段的结束时刻时,在一种可能的实现方式中,获取数据库***内的实时程度参数,该实时程度参数用于表征数据库***内的分析型事务处理的数据所支持的最大延时,换言之,该数据库***可允许使用多久之前的历史态数据来进行分析,比如,设定实时程度参数为1小时,则代表数据库***允许以1小时之前的历史态数据来进行分析。在一种可能的实现方式中,该实时程度参数由管理用户来进行自定义设置,如果管理用户没有设置过实时程度参数,则采用数据库***的默认值(例如1秒),其中该默认值也是可以进行修改的,本申请实施例对此不进行具体限定。在获取到实时程度参数之后,基于该实时程度参数来确定该结束时刻,在一种可能的实现方式中,将当前时刻之前距离该当前时刻达到实时程度参数的时刻确定为该结束时刻,本申请实施例不对结束时刻的确定方式进行具体限定。
在基于上述方式确定了开始时刻和结束时刻之后,即可得到从开始时刻到结束时刻所构成的查询时间段。在一种可能的实现方式中,该查询时间段是一个闭区间,即该查询时间段既包括左端点的开始时刻也包括右端点的结束时刻,或者,该查询时间段是一个左闭右开的时间区间,即该查询时间段包括左端点的开始时刻,但不包括右端点的结束时刻。其中,左闭右开的时间区间的特定在于:对于开始时刻来说,恰好在开始时刻生效的数据是可见的,对于结束时刻来说,恰好在结束时刻生效的数据是不可见的,这一点是用于判断辅助数据的可见性。在一种可能的实现方式中,可以由用户指定该查询时间段是否包含右端点的结束时刻,如果指定包含该结束时刻,则该查询时间段为闭区间,如果指定不包含该结束时刻,则该查询时间段为左闭右开的时间区间,如果用户未指定是否包含该结束时刻,可默认包含该结束时刻,即默认该查询时间段为闭区间。
在一些实施例中,计算设备在确定该查询时间段和该数据查询条件之后,由于OLTP集群会周期性地将自身交易型事务所产生的历史态数据迁移到存储集群的存储设备中进行持久化存储,上述迁移历史态数据的过程就是对历史态数据进行转储的过程,由于每次进行转储时通常会产生相关的记录,比如在存储集群的元信息管理器中记载本次转储的历史态数据有哪些,或者专门维护一张表来记录每次转储的完成时间,因此很容易能够查询到所有符合该待分析事务的数据查询条件的历史态数据最近一次完成转储的时刻(即第一时刻)。
402、计算设备基于该历史态数据,获取全量分析结果,该全量分析结果为基于该待分析事务的分析语义对该历史态数据进行处理所得的结果。
在一些实施例中,以第一时刻为界限,可将查询时间段划分为两个子区间,从开始时刻到第一时刻构成的第一子区间,和从第一时刻到结束时刻构成的第二子区间,将第一子 区间的全量分析结果和第二子区间的增量分析结果进行合并,即可得到最终的最终分析结果。
在一些实施例中,由于数据库***内可能会缓存有从开始时刻到第一时刻之间的第二时刻下的中间分析结果,根据可见性判断算法,此时只需要将已缓存的中间分析结果和第二子区间的增量分析结果进行合并,即可得到最终分析结果,即无需重复计算出第一时刻下的全量分析结果。
在一种可能的实现方式中,基于该历史态数据获取全量分析结果的方式可以是查询是否缓存对第二时刻可见的第一历史态数据的中间分析结果,该第二时刻为该查询时间段的开始时刻到该第一时刻之间的任一时刻,接着,在已缓存该中间分析结果的情况下,将该中间分析结果确定为该全量分析结果;在未缓存该中间分析结果的情况下,生成对该第一时刻可见的第二历史态数据的该全量分析结果。其中,关于可见性判断算法、生成全量分析结果的方式将会在下一个实施例中详细说明,这里不再赘述。
在上述过程中,通过判断是否缓存了中间分析结果,能够在已缓存有中间分析结果的情况下,避免重复计算出第一时刻下的全量分析结果,因此能够大大节约计算设备的计算资源。
403、计算设备基于在该历史态数据对应的当前态数据上发生的特定变更,获取增量分析结果,该特定变更为在该第一时刻起至该查询时间段的结束时刻之间发生的变更,该增量分析结果为基于该分析语义对该特定变更得到的数据进行处理所得的结果。
由于第一时刻是对应的历史态数据最近一次完成转储的时刻,那么代表第一时刻起到结束时刻之间的数据都尚未从OLTP集群转储到存储集群中,因此,需要先从OLTP集群中获取到对应特定变更产生的日志,对特定变更产生的日志进行回放,获取到对应特定变更得到的数据,接着,比较主键标识相同的历史态数据和特定变更得到的数据,并按照待分析事务的SQL语句的分析语义对比较结果进行处理,生成该增量分析结果。其中,该增量分析结果只代表与上一次全量分析结果之间发生的变化,而并非是新一次计算所得的全量分析结果。
在上述过程中,通过对特定变更得到的数据仅获取增量分析结果,而无需获取全量分析结果,由于增量分析结果所需分析的数据量较少、且耗时较少,因此能够大大节约计算设备的计算开销和处理资源。
404、计算设备基于该全量分析结果和该增量分析结果,获取该待分析事务的最终分析结果。
在一些实施例中,该增量分析结果包括特定变更前的分析结果和特定变更后的分析结果,因此,在合并全量分析结果和增量分析结果时,需要将该全量分析结果减去该特定变更前的分析结果,得到第一分析结果;将该第一分析结果加上该特定变更后的分析结果,得到该最终分析结果。
在上述过程中,通过全量计算和增量计算结合以获取最终分析结果的方式,能够将传统的单层计算模型转换为两层实时分析模型,能够适配于基于全时态数据库的HTAP***或HTASP***,且能够节约整个***内的计算资源。
上述所有可选技术方案,能够采用任意结合形成本公开的可选实施例,在此不再一一赘述。
本申请实施例提供的方法,通过针对待分析事务确定查询时间段,并以第一时刻作为查询时间段内的一个时间点分界线,对第一时刻之前的历史态数据计算出一个全量分析结果,对第一时刻之后特定变更得到的数据计算出一个增量分析结果,将全量分析结果和增量分析结果两者结合,最终得到最终分析结果,使得针对全时态的时间轴上的任一个查询时间段,都能够提供在线实时分析和处理服务,而并不局限于仅对最新数据提供分析功能,从而大大提升了HTAP***或HTASP***等数据库***的事务处理性能。
通过全量计算和增量计算结合以获取最终分析结果的方式,能够将传统的单层计算模型转换为两层实时分析模型,能够适配于基于全时态数据库的HTAP***或HTASP***,且能够节约整个***内的计算资源。
在上述实施例中,总体介绍了HTAP***或HTASP***下,某一时刻新到一个分析型的待分析事务,AP计算节点是如何对待分析事务生成最终分析结果的,在本申请实施例中,将结合对数据库命令的扩展、冷热等级和全时态数据的关联、基于冷热等级的存储模型、可见性判断算法等,分别详细介绍实时分析模型和流式计算模型。
以下,将对基于全时态数据库搭建的HTAP***或HTASP***下,数据库命令的扩展方式进行说明。
(1.1)设置实时程度参数的命令
其中,该实时程度参数用于表征数据库***内的分析型事务处理的数据所支持的最大延时,换言之,该数据库***可允许使用多久之前的历史态数据来进行分析,比如,设定实时程度参数为1小时,则代表数据库***允许以1小时之前的历史态数据来进行分析。
上述设置实时程度参数的命令,使得管理用户能够根据自身的SLA(Service Level Agreement,服务级别协议)需求来确定当前数据库进行在线分析的程度(即按需设置当前数据库的实时程度参数)。
在一种可能的实现方式中,实时程度参数的单位包括:秒、分钟或小时,此外,当实时程度参数设置为0时表示采用实时分析结果,需要感知当前交易***(即OLTP集群)的最新数据的变化,如果管理用户没有设置实时程度参数,可为当前数据库将实时程度参数配置为默认值,该默认值为大于或等于0的任一数值,例如,默认值为1s(秒)。
示意性地,以REAL_TIME_ANALYZING_DEGREE来表示实时程度参数,则查看当前数据库的实时程度参数的命令如下述代码所示:
SHOW REAL_TIME_ANALYZING DEGREE;
MySQL[test]>SHOW REAL_TIME_ANALYZING_DEGREE;
+---------------------------------------------------------------------------------------------------+
|REAL_TIME_ANALYZING_DEGREE|memo|
+---------------------------------------------------------------------------------------------------+
|1s|Allow 1 second latency between analyzing scan data with current newest data version(允许分析扫描数据和当前最新数据版本之间存在1秒延迟)|
+---------------------------------------------------------------------------------------------------+
1 row in set(0.00sec)
上述代码示出了一种针对当前数据库的实时程度参数的查看命令,能够看出,所显示的当前用户设置的实时程度参数为1秒,代表实时粒度与最新数据延迟1秒,也即,AP计算节点(即计算设备)能够从与最新数据相差1秒的历史版本或者过渡态版本的数据来进行在线分析。
示意性地,以REAL_TIME_ANALYZING_DEGREE来表示实时程度参数,则设置当前数据库的实时程度参数的命令如下述代码所示:
SET REAL_TIME_ALALYZING DEGREE=5s;
上述代码示出了一种针对当前数据库的实时程度参数的设置命令,能够看出,这条设置命令的含义是:允许比最新数据延迟5秒的数据版本来作为AP计算节点进行分析的数据版本。
以下是针对上述设置命令的测试代码:
MySQL[test]>SET REAL_TIME_ANALYZING_DEGREE=5s;
0 row in set(0.00sec)
通常情况下,管理用户不会设置太大的实时程度参数,因此实时程度参数越大,代表AP计算设备进行分析的数据版本与最新版本之间的延时越大,因此,本申请实施例以秒级单位的实时程度参数为例进行说明。但管理用户可根据自身SLA需求,修改为分钟单位、小时单位等粒度,再让OLAP集群来确定本次查询所需使用的所有数据版本。
(1.2)设置时间窗大小的命令
时间窗大小是指:在AP计算节点或流式计算节点上运行的流式计算引擎(或流式计算模型)以多大的步长来计算增量分析结果。
换言之,修改时间窗大小的命令主要是用来调节最小划分的时间窗大小,流式计算引擎会按照时间窗大小,来启动对于每个时间窗内的各个数据版本的划分与增量分析结果的计算。
在一种可能的实现方式中,时间窗大小的单位包括:秒、分钟或小时,用户可自定义设置一个时间窗大小,但对于数据库引擎来说,会以1秒为最小的时间窗单元,从而将用户设置的时间窗大小划分为多个秒级的时间窗单元,换言之,用户设置的时间窗大小为最小的时间窗单元的整数倍,如果管理用户没有设置时间窗大小,可为当前数据库将时间窗大小配置为默认值,该默认值为大于0的任一数值,例如,默认值为1s(秒)。需要说明的是,当管理用户设置的时间窗大小大于***最小的时间窗单元时,***会按照最小的时间窗单元来进行数据的划分,但是只会在达到管理用户指定的时间窗大小时才进行增量分析结果的计算。
示意性地,以TIME_WINDOW来表示时间窗大小,则查看当前数据库的时间窗大小的命令为SHOW TIME_WINDOW,如下述代码所示:
MySQL[test]>SHOW TIME_WINDOW;
+---------------------------------------------------------------------------------------------------+
|REAL_TIME_ANALYZING DEGREE|memo|
+---------------------------------------------------------------------------------------------------+
|5s|use time range as time window to divide different transaction into different time window(使用时间窗来将不同的交易划分至不同的时间窗口)|
+---------------------------------------------------------------------------------------------------+
1 row in set(0.00sec)
示意性地,以TIME_WINDOW来表示时间窗大小,则设置当前数据库的时间窗大小的命令为SET TIME_WINDOW,如下述代码所示:
MySQL[test]>Set TIME_WINDOW=5s;
0 row in set(0.00sec)
(1.3)设置全量分析结果的计算周期的命令
全量分析结果的计算周期也即全量分析结果的计算频率,该计算周期用来定义在流式计算引擎中每隔多长时间计算一次全量分析结果。
在一种可能的实现方式中,管理用户可根据业务特点定义一个全量分析结果的计算周期,通过定期发起一次全量分析结果的计算,能够减少在每一次生成最终分析结果时,由于全量分析结果太旧,导致中间需要合并的增量分析结果太多,从而导致计算性能下降的情况发生。换言之,能够保证每次生成最终分析结果时,与上一次的全量分析结果之间需要合并的增量分析结果是适配于业务特点的。
在一种可能的实现方式中,全量分析结果的计算周期的单位包括:秒、分钟或小时,用户可自定义设置一个全量分析结果的计算周期,但对于数据库引擎来说,会以1秒为最小的时间窗单元,从而将用户设置的全量分析结果的计算周期划分为多个秒级的时间窗单元,换言之,用户设置的全量分析结果的计算周期为最小的时间窗单元的整数倍,如果管理用户没有设置全量分析结果的计算周期,可为当前数据库将全量分析结果的计算周期配置为默认值,该默认值为大于0的任一数值,例如,默认值为1分钟,全量分析结果的计算周期的默认值通常会比增量分析结果的时间窗大小的默认值要大,这是因为***不希望频繁进行全量分析结果的计算。当然,管理用户可根据自身的业务情况来调整设置的全量分析结果的计算周期(当前也支持调整默认值),在一种可能的实现方式中,还可根据性能监控模块来感知当前的全量分析结果的计算周期是否会导致资源使用的波动,从而来自适应调节全量分析结果的计算周期,即,当前***较为繁忙时,调大全量分析结果的计算周期,即降低全量分析结果的计算频率,当前***较为空闲时,调小全量分析结果的计算周期,即提高全量分析结果的计算频率。需要说明的是,当管理用户设置的全量分析结果的计算周期大于***最小的时间窗单元时,***会按照最小的时间窗单元来进行数据的划分,但是只会在达到管理用户指定的全量分析结果的计算周期时才进行全量分析结果的计算。
示意性地,以FULL_ANALAZY_RESULT_PERIOD来表示全量分析结果的计算周期,则查看当前数据库的全量分析结果的计算周期的命令为SHOW FULL_ANALAZY_RESULT_PERIOD,如下述代码所示:
MySQL[test]>SHOW FULL_ANALAZY_RESULT_PERIOD;
+---------------------------------------------------------------------------------------------------+
|REAL_TIME_ANALYZING DEGREE|memo|
+---------------------------------------------------------------------------------------------------+
|5s|use time range as time window to divide different transaction into different time window(使用时间窗来将不同的交易划分至不同的时间窗口)|
+---------------------------------------------------------------------------------------------------+
1 row in set(0.00sec)
示意性地,以FULL_ANALAZY_RESULT_PERIOD来表示全量分析结果的计算周期,则设置当前数据库的全量分析结果的计算周期的命令为SET FULL_ANALAZY_RESULT_PERIOD,如下述代码所示:
MySQL[test]>Set FULL_ANALAZY_RESULT_PERIOD=1m;设置全量分析结果的计算周期为1分钟;
0 row in set(0.00sec)
(1.4)设置冷热等级的划分标准的命令
其中,冷热等级用于表征在历史时间段内对在历史态数据上发生操作的频繁程度,由于该发生的操作包括但不限于读操作即查询或访问操作和写操作即修改或更新操作,因此,可基于历史时间段内对历史态数据的访问次数以及对历史态数据对应的当前态数据的修改次数,来综合确定出每个版本的历史态数据的冷热等级。
在一种可能的实现方式中,管理用户可根据自身的业务***情况来定义冷热等级的划分标准,即,管理用户自定义将什么样的数据划分到哪一种冷热等级。在一个示例性,按照发生操作的频繁程度从高到低,包括如下几类冷热等级:热数据、温数据、冷数据和归档数据,并分别设定每种冷热等级的划分标准,并按照冷热等级对全时态数据进行分区,将冷热等级相同的数据划分到相同分区下的存储设备中进行存储,或者,将冷热等级相同的数据划分到相同分区下的存储设备的同一种存储介质中进行存储,从而达到最好的数据库使用效果与最低廉的数据库使用成本。
示意性地,以HOT_COLD_DATA_LEVEL来表示冷热等级的划分标准,则查看当前数据库的冷热等级的划分标准的命令为SHOW HOT_COLD_DATA_LEVEL,意思是当某一时态的数据在规定时间段的访问次数小于指定值时,将会转储到下一个级别的存储设备或存储介质中,如下述代码所示:
+---------------------------------------------------------------------------------------------------+
|REAL_TIME_ANALYZING DEGREE.|memo|
+---------------------------------------------------------------------------------------------------+
|{{|
|
|{hot_data,1day},|一天内的数据为热数据.|
|{change_times,100},|一天内的数据被修改的次数.|
|{access_times,100},|一天内的数据被访问的次数.|
|{storage_path,“/data/ssd_disk/”},|满足上述条件的数据将被存储在这个路径下,通过目录路径对应不同速度设备.|
|}|
|
|…|其他级别的数据,格式定义与上述热数据描述一致,不再一一枚举。|
+---------------------------------------------------------------------------------------------------+
1 row in set(0.00sec)
基于上述示例,可知管理用户能够从两个维度指定冷热等级的划分标准,即,基于一定周期内(如数小时、1天、1周、1月、1年等,最小到1秒)对于某一条数据记录的修改和/或访问次数,来确定本条数据记录是否需要转储到下一个级别的存储设备或存储介质中。
示意性地,以HOT_COLD_DATA_LEVEL来表示冷热等级的划分标准,则设置当前数据库的冷热等级的划分标准的命令为SET HOT_COLD_DATA_LEVEL;,用于由管理用户根据自身业务需求来自定义转储标准。
在一个示例性场景中,按照上述格式可修改冷热等级的划分标准,详细命令略,如果管理用户并未设置冷热等级的划分标准,可提供一种默认的冷热等级的划分标准,例如,按照如下方式划分为4个冷热等级:热数据(1天内频繁修改的数据,例如1天内修改次数>100次)、温数据(1月内被修改次数一般,但会被访问,例如10次<1月内修改次数<50次,同时1月内访问次数>50次)、冷数据(1年内数据不会被修改,但是会被访问,例如1年内访问次数<10次)以及归档数据(超过1年没有被访问的数据)。上述默认的冷热等级划分标准也可被管理用户进行修改,本申请实施例对此不进行具体限定。
需要说明的是,上述(1.1)-(1.4)的所有命令,都能够通过数据库启动参数配置文件中进行定义,并且,也能够在数据库运行过程中,由管理用户根据业务特点和实际资源情况来设置更为合适的值,上述所有参数都人工动态调整或者***自适应调整,不需要停止数据库的服务。
以下,将介绍在某一种可能的冷热等级的划分标准下,历史态数据与冷、温、热数据的关联关系:
图5是本申请实施例提供的一种历史态数据与冷热温数据的关系示意图,如500所示,由于冷热等级的划分标准是由在一段目标时间段内发生的访问次数和/或修改次数来衡量的,使得在全时态数据库下,自然实现了历史态数据与冷数据和温数据的对应关系。即,由于热数据是指短期内(如1天)被频繁修改的数据记录,这代表着热数据对应于当前态数据和过渡态数据,对冷数据和温数据相对来说几乎不会被修改,只会被访问,因此温数据对应于历史态数据,冷数据对应于更旧的历史态数据。
在一种可能的实现方式中,随着时间的推移,过渡态数据也会被转储为历史态数据。比如,全时态数据库会周期性启动转储算法,将过渡态数据从热数据对应的存储设备转储到温数据对应的存储设备(用于存储历史态数据),当过渡态数据转储完毕后,热数据对应的存储设备可直接丢弃已转储完毕的过渡态数据,以节约更多的存储空间。
在一种可能的实现方式中,当前态数据和过渡态数据都对应于热数据,而对于历史态数据,可根据用户特征来定义多个冷热级别,以满足自身业务特定的需求,比如,对历史态数据,划分为3个冷热级别:温数据、冷数据和归档数据。当任一数据一旦被转储为历史态数据之后,将会变为只读数据,即***不允许修改历史态数据,即防止篡改历史态数据,以保护历史态数据的安全性。
在一种可能的实现方式中,也可以将修改频率较高的当前态数据和过渡态数据对应于热数据,将修改频率较低的当前态数据和过渡态数据对应于温数据,将历史态数据划分为冷数据和归档数据,上述图5仅仅是给出一种将全时态数据和冷热等级下的数据之间的关联关系,但并不局限于图5所示的关联关系。
在一种可能的实现方式中,当前态数据和过渡态数据,都对应于现在活跃的数据记录,当数据记录不断发生修改,就会持续产生最新版本的数据记录;当数据记录一段时间段内(目标时间段内)持续没有被修改或被访问,就会被转储为历史态数据(如冷数据),如果在冷数据的等级下,仍然持续没有被访问,那么数据记录会因为最新的历史态数据的不断老化,会被转储成更冷的数据(如归档数据)。
在一种可能的实现方式中,对全时态数据库中所有的当前态数据,增加一个修改标记信息来统计修改次数,此外,对所有的当前态数据、过渡态数据和历史态数据,都增加一个访问标记信息来统计访问次数,从而能够方便地判定每条数据记录是否需要被转储到下一个级别的存储设备上。
以下,将介绍数据状态转换与主从副本机制相结合后,如何运用在HTAP或HTASP***中的原理。
图6是本申请实施例提供的一种主从副本机制内增加实时分析从副本的原理性示意图,如600所示,在分布式***的存储集群(即存储引擎)中,为了防止某一存储设备故障而导致存储设备上的所有数据不可用,通常会引入主从副本机制,即,对每一份数据(即主副本)都保存有至少一个从副本,以保证即使主副本宕机,仍然可以在从副本上提供相关数据服务。在本申请实施例中以一主两备机制为例,对每个主副本都保存有两个从副本,主从副本之间通过Paxos或Raft等一致性协议来保证数据一致性,从而来提升整个存储集群的数据可靠性。在Paxos或Raft等一致性协议下,会将数据表中的数据记录划分为一个个较小的管理单元,比如数据块block、数据区域region、数据页page等,本申请实施例对此不进行具体限定。
而在本申请实施例中,会在存储集群的每一组主从副本中,额外引入一个实时分析从副本,主从副本会按照传统方式来读写数据,并将相关数据都写到热数据对应的存储设备中,而实时分析从副本则只负责同步数据,但不会参与到主副本的选举流程中,这是由于 某些一致性协议可能会定期重新选举主副本导致的选举流程,并且实时分析从副本也不会参与一致性协议下的任何投票流程(即不参与一致性协议下的多数派达成)。
实时分析从副本所在的节点设备,用于提供历史态数据的转储功能,换言之,能够将历史态数据的转储功能从主副本迁移到实时分析从副本,这样能够减去主副本所在节点设备的计算压力,从而彻底解耦OLTP集群和OLAP集群之间的相互影响。
在一种可能的实现方式中,实时分析从副本与主副本或其他两个从副本之间可具有较大的延迟,通过一致性协议进行日志的追赶,使得最终实时分析从副本能够获取主副本上所有版本的所有数据,使得对于现在的OLTP集群的干扰最低。
在一种可能的实现方式中,实时分析从副本并不一定每次都从主副本来拉取数据,而是可根据主从副本各自的忙闲情况,选择最有利的副本(如最空闲的副本)来进行同步,从而使得实时分析副本的同步任务对OLTP集群的影响最小化。
在一种可能的实现方式中,实时分析从副本将某些访问频率或修改频率较低的数据转储为历史态数据之后,可周期性地清理掉已转储完毕的数据(主副本和其他从副本也可以同步清理掉),比如,对1天内修改次数小于50次且访问次数小于100次的数据,则转储到历史态数据对应的存储设备中,然后直接删除转储完毕的数据。
以下,将对本申请实施例的冷热数据的存储模型进行介绍。
图7是本申请实施例提供的一种冷热数据的存储模型的原理性示意图,如700所示,由于上面介绍过,针对全时态数据库中数据表,增加了冷热等级的划分标准HOT_COLD_DATA_LEVEL这一参数,以下,将以划分为热数据、温数据、冷数据、归档数据这4种冷热等级为例,介绍如何基于4种冷热等级来存储对应的数据。
在参数初始化阶段,管理用户可利用上述(1.4)中提供的相关命令,在集群建立的时候按照自身的业务特点来设置热数据、温数据、冷数据、归档数据各自的划分标准,需要说明的是,本申请实施例并不局限于划分为4个等级,且并不限定每个等级的划分标准,管理用户可细分为任意数量的等级,并设置任意的划分标准。
在一种可能的实现方式中,在设定了冷热等级的划分标准的基础上,后续在创建数据表时,***会自动将数据表按照数据的冷热等级来进行分区,将数据表划分到对应分区下的存储设备中进行存储。此外,存储设备的后台线程也会定期判断数据的冷热等级是否发生变化,当冷热等级发生变化时,需要将数据转储到相应的分区的其他存储设备中。
在一种可能的实现方式中,在判断数据的冷热等级是否发生变化时,按照不同的划分标准,分别启动对于不同分区下存储的数据的后台存储处理任务,例如,扫描每个分区存储的全部数据,判断数据记录的各个版本是否符合转储到下一个存储级别的条件,如果符合转储的条件,需要将符合的各个版本的数据记录移动到对应的存储分区当中,其中,全时态数据会保留所有数据记录的所有版本,并且历史版本是不可修改的,历史版本的数据记录会逐步从热数据分区被迁移到温数据分区、冷数据分区,最终会被迁移到归档数据分区(仅保存但通常不会被访问)。
在一种可能的实现方式中,在***运行时,管理用户可随时更改或者***自动动态调整冷热等级的划分标准,无需进行停机。当冷热等级的划分标准发生变化时,存储设备的 后台线程会根据负载情况来对自身的分区情况进行调整。例如,增加一个分区时,先添加相应的新分区,然后对相邻分区的数据进行调整;又例如,减少一个分区时,虽然减少了一个已有分区,但是并不会删除被减少的分区中存储的数据,因此需要将被删除的分区内的数据迁移到相邻的分区中之后,再删除被指定的分区;又例如,修改一个分区时,后台线程会将不符合当前分区的划分标准的数据迁移到相邻分区中,例如将温数据分区的划分标准从访问次数小于100次修改为访问次数小于90次,那么在温数据分区中,原本访问次数在90~100之间的数据将需要迁移到热数据分区中。
以下,对数据记录的可见性判断算法进行介绍。
由于全时态数据库中对每个数据记录都引入了时间维度,因此OLAP集群中计算设备的查询计算模型也需要进行重构,以提供本申请实施例的高效、低资源使用率的实时分析计算能力,并且支持提供一个基于时间窗粒度的流式计算引擎的基础查询执行模型。
关系型数据库通常提供的数据服务,都会转变为对于用户提交的SQL语句的处理,由于历史态数据不允许被修改或删除,是只读数据,因此本申请实施例涉及的计算语句只有查询(Select)语句,其他的增删改语句都只需要按照SQL:2011时态标准中关于数据提取的语义实现即可。因此,由于Select语句一定会涉及到判断数据记录的可见性,因此下面将描述数据记录的可见性判断算法。
图8是本申请实施例提供的一种全时态的数据记录的存储形态原理图,如800所示,横轴代表时间轴,从左到右代表时间流逝,以数据库中存储的记录1、记录2和记录3为例,说明从T1~T4这四个时刻下各条记录的可见版本。
记录1(Record 1,R1):R1是在时间轴的最左侧***数据库的,***时的***版本为V1,在T1之前的某个时刻R1被修改,因此第一条线段代表R1.V1(R1的V1版本)的生命周期,该生命周期代表了对应版本的有效时间,这一有效时间可基于用户定义的有效时间来决定,也可以基于数据的事务时间来决定。接下来的三条线段分布代表R1.V2、R1.V3、R1.V4各自的生命周期。
分析可知,在T1时刻,R1的可见版本为V2;在T2时刻,V2版本正好结束,由于时间连续性,V3版本的生命周期的左端值等于V2版本的生命周期的右端值,因此T2时刻,R1的可见版本为V3;在T3时刻,仍然处于V3的生命周期内,因此R1的可见版本为V3;在T4时刻,处于V4的生命周期内,因此R1的可见版本为V4。
记录2(Record 2,R2):R2在整个时间轴上表现为一条射线,代表R2在时间轴的最左侧***数据库之后,没有被修改过,一直处于R2.V1版本。
分析可知,在T1、T2、T3和T4时刻下,R2的可见版本均为V1。
记录3(Record 3,R3):R3在T1时刻之后***数据库,因此T1时刻时还没被***,T1时刻不可见,然后在T1时刻和T2时刻之间的某个时刻被删掉了,因此在T2时刻之后已被删除,T2时刻之后也不可见。
分析可见,在T1、T2、T3和T4时刻下,R3均不可见。
在一些实施例中,数据记录的每个版本的生命周期采用一个左闭右开的时间区间来标识,例如,R1.V1[2020-08-09 12:30:30,2020-08-19 22:15:30)表示记录R1在2020-08-09 12:30:30***,并在2020-08-19 22:15:30失效。
下面对单一时刻(即某一时间点)的数据可见性判断算法进行说明:
在T1时刻,假设一个查询事务来扫描数据,在T1时刻对该查询事务来说,可见的数据记录为R1.V2和R2.V1,由于记录R3在T1时刻尚未***,因此记录R3还不存在,自然也不可见。
在T2时刻,假设一个查询事务来扫描数据,在T2时刻对该查询事务来说,可见的数据记录为R1.V3和R2.V1,由于记录R3在T2时刻之前已被删除,因此记录R3数据不存在,自然也不可见。需要说明的是,由于数据记录的每个版本的生命周期具有左开右闭的区间属性,因此T2时刻下R1.V3可见。
在T3时刻,假设一个查询事务来扫描数据,在T3时刻对该查询事务来说,可见的数据记录为R1.V3和R2.V1,记录R3已经被删除因此不可见。
在T4时刻,假设一个查询事务来扫描数据,在T4时刻对该查询事务来说,可见的数据记录为R1.V4和R2.V1,记录R3已经被删除因此不可见。
通过上述单一时刻的数据可见性判断算法,可知每个时间点上的查询事务只能看到某条数据记录的一个版本,但有一些查询事务并非是针对时间点发起的,而是有可能指定了一个查询时间段,因此,下面对某一时间段(即查询时间段)的数据可见性判断算法进行说明:
假设某一查询时间段划分为如下的4个连续的时间段:[min_time,T1),[T1,T3),[T3,T2),[T2,current_time],如果某一个查询事务指定了一个查询时间段(或称为有效时间区间、事务时间区间等),那么针对该查询时间段,数据记录的可见性判断算法如下。
示意性地,假设查询事务的SELECT Query语句为:
select sum(a)from T1 FOR SYSTEM_TIME FROM time1 TO time2 where a>1;
其中,time1是指上述查询时间段的左端值(即min_time),time2是指上述查询时间段的右端值(即current_time)。
对于单个时间段来说,其可见性判断算法表达为:确定单个时间段的右端值,在该单个时间段的右端值上按照单一时刻的数据可见性判断算法,来判断出相对该单个时间段可见的数据版本。
对于单个时间段拆分为多个时间段组合的情况,比如将单个时间段[min_time,current_time]拆分为[min_time,T1)、[T1,T3)、[T3,T2)和[T2,current_time]这4个时间段,也即[min_time,current_time]->[min_time,T1),[T1,T3),[T3,T2),[T2,current_time]。
此时,对该多个时间段中最左端的时间段,按照单一时间段的可见性判断算法来判断出可见的数据版本;对该多个时间段中最右端的时间段,查找到每个数据记录最新可见的记录版本值,并找到最左端的记录版本值,再对这两个版本的数据记录执行一个多层次查询结果的修正算法,来保证查询结果的正确性;在最右端的时间段为左闭右开区间的情况下,如果上述最右端的时间段中看到的版本与上述最左端的时间段中看到的版本一致,则 跳过本数据记录,视本数据记录为不可见,在最右端的时间段为闭区间的情况下,这个时候以最右端的时间段的右端值下的可见版本作为本条记录的可见值。
在上述相关内容的支持基础上,在本申请实施例中将详细介绍HTAP***或HTASP***中OLAP集群中的计算设备如何基于实时分析模型来提供高效的实时分析性能。
图9是本申请实施例提供的一种事务处理方法的流程图。参见图9,该实施例适用于HTAP***、HTASP***或者区块链***等支持实时分析的数据处理***,该事务处理方法由OLAP集群中的任一计算设备执行,该实施例包括下述步骤:
901、计算设备解析用户请求,得到该用户请求对应的待分析事务的数据查询条件、分析语义和查询时间段。
在一些实施例中,计算设备接收应用客户端的任一用户请求,解析该用户请求的头字段,当该头字段指示该用户请求为分析型事务时,解析该用户请求的数据字段,得到该用户请求中携带的SQL语句(例如SELECT Query语句等),将该SQL语句到查询解析器中,通过该查询解析器对该SQL语句进行词法分析、句法分析、时态翻译和语义检查,得到该SQL语句的数据查询条件和分析语义(比如,求和、求平均、排序等),另外,如果SQL语句中指定了查询时间段的开始时刻,则获取该开始时刻,如果SQL语句中指定了查询时间段的结束时刻,则获取该结束时刻。接着,基于该SQL语句构建对应的该待分析事务,计算设备为该待分析事务创建一个进程或线程,或者复用已创建的某个进程或线程,并通过该进程或线程来处理该待分析事务。
示意性地,当该待分析事务指定了该查询时间段的开始时刻时,则以待分析事务指定的开始时刻为准;当该待分析事务未指定该查询时间段的开始时刻时,以待分析事务所查询的数据的最新版本的生效时刻为准,也即将符合该数据查询条件的数据的最新版本的生效时刻确定为该开始时刻;当涉及到查询多个数据的时候,对每个数据都以各自最新版本的生效时刻来选择对应的可见版本进行分析即可,以待分析事务所查询的数据的***版本的生效时刻为准,也即将符合该数据查询条件的数据的***版本的生效时刻确定为该开始时刻,当涉及到查询多个数据的时候,对每个数据都以各自***版本的生效时刻来选择对应的可见版本进行分析即可,本申请实施例对开始时刻的确定方式不进行具体限定。
示意性地,当该待分析事务指定了该查询时间段的结束时刻时,则以待分析事务指定的结束时刻为准,当该待分析事务未指定该查询时间段的结束时刻时,获取数据库***内的实时程度参数,该实时程度参数用于表征数据库***内的分析型事务处理的数据所支持的最大延时,换言之,该数据库***可允许使用多久之前的历史态数据来进行分析,比如,设定实时程度参数为1小时,则代表数据库***允许以1小时之前的历史态数据来进行分析。在一种可能的实现方式中,该实时程度参数由管理用户来进行自定义设置,如果管理用户没有设置过实时程度参数,则采用数据库***的默认值(例如1秒),其中该默认值也是可以进行修改的,本申请实施例对此不进行具体限定。在获取到实时程度参数之后,基于该实时程度参数来确定该结束时刻,可选地,将当前时刻之前距离该当前时刻达到实时程度参数的时刻确定为该结束时刻,本申请实施例不对结束时刻的确定方式进行具体限定。
在基于上述方式确定了开始时刻和结束时刻之后,即可得到从开始时刻到结束时刻所构成的查询时间段,该查询时间段是一个闭区间,即该查询时间段既包括左端点的开始时刻也包括右端点的结束时刻,或者,该查询时间段是一个左闭右开的时间区间,即该查询时间段包括左端点的开始时刻,但不包括右端点的结束时刻,其中,左闭右开的时间区间的特定在于:对于开始时刻来说,恰好在开始时刻生效的数据是可见的,对于结束时刻来说,恰好在结束时刻生效的数据是不可见的,这一点是用于判断辅助数据的可见性。由用户指定该查询时间段是否包含右端点的结束时刻,如果指定包含该结束时刻,则该查询时间段为闭区间,如果指定不包含该结束时刻,则该查询时间段为左闭右开的时间区间,如果用户未指定是否包含该结束时刻,可默认包含该结束时刻,即默认该查询时间段为闭区间。
902、计算设备从该查询时间段中确定第一时刻,该第一时刻为对历史态数据最近一次完成转储的时刻,该历史态数据为符合该待分析事务的数据查询条件的数据。
在一些实施例中,计算设备在确定该查询时间段和该数据查询条件之后,由于OLTP集群会周期性地将自身交易型事务所产生的历史态数据迁移到存储集群的存储设备中进行持久化存储,上述迁移历史态数据的过程就是对历史态数据进行转储的过程,由于每次进行转储时通常会产生相关的记录,比如记录一下上一次同步的日志的位点(Checkpoint),或者在存储集群的元信息管理器中记载本次转储的历史态数据有哪些,或者专门维护一张表来记录每次转储的完成时间,因此很容易能够查询到所有符合该待分析事务的数据查询条件的历史态数据最近一次完成转储的时刻(即第一时刻)。
需要说明的是,上述对历史态数据进行转储的过程,可基于之前实施例中涉及的在主从副本机制上增加的实时分析从副本,从而能够降低历史态数据转储任务对OLTP集群内交易性能的影响。
903、计算设备查询是否缓存对第二时刻可见的第一历史态数据的中间分析结果,该第二时刻为该查询时间段的开始时刻到该第一时刻之间的任一时刻。
在一些实施例中,以第一时刻为界限,可将查询时间段划分为两个子区间,从开始时刻到第一时刻构成的第一子区间[开始时刻,第一时刻),和从第一时刻到结束时刻构成的第二子区间[第一时刻,结束时刻],将第一子区间的全量分析结果和第二子区间的增量分析结果进行合并,即可得到最终的最终分析结果。换言之,将一个查询时间段的分析工作,划分为针对两个子区间(即两个时间段)的分析工作。需要说明的是,本申请实施例以第二子区间为闭区间为例进行说明,但可由用户指定该查询时间段是否包含右端值的结束时刻,如果指定包含该结束时刻,则该第二子区间为闭区间,如果指定不包含该结束时刻,则该第二子区间为左闭右开区间,如果用户未指定是否包含该结束时刻,可默认包含该结束时刻,即默认第二子区间为闭区间。
在一些实施例中,由于数据库***内可能会缓存有从开始时刻到第一时刻之间的第二时刻下的中间分析结果,根据可见性判断算法,如果查询时间段被划分为超过两个时间段,那么只需要考察最左端的时间段和最右端的时间段,就能够判断出数据记录的可见版本,从而在可见版本上进行分析和处理。因此,如果以第一时刻和第二时刻为界限,将查询时 间段划分为三个子区间:[开始时刻,第二时刻)、[第二时刻,第一时刻)、[第一时刻,结束时刻],只需要将已缓存的[开始时刻,第二时刻)的中间分析结果和第二子区间[第一时刻,结束时刻]的增量分析结果进行合并,即可得到最终分析结果,即无需重复计算出第一时刻下的全量分析结果。
在一些实施例中,计算设备在查询该中间分析结果时,根据该待分析事务的数据查询条件,确定出之前的历史事务是否针对相同的数据查询条件进行过分析,如果未命中之前任一历史事务的数据查询条件,进入下述步骤904-B;如果命中之前任一历史事务的数据查询条件,则从缓存区中以被命中的该历史事务的事务ID(Identification,标识)为索引,查询是否缓存有与该索引对应的全量分析结果,如果命中任一已缓存的全量分析结果,将被命中的全量分析结果确定为该中间分析结果,进入下述步骤904-A,如果未命中所有缓存的全量分析结果,则确定未缓存该中间分析结果,进入下述步骤904-B。
在一些实施例中,在查询是否缓存对第二时刻可见的第一历史态数据的中间分析结果之后,在已缓存该中间分析结果的情况下,执行下述步骤904-A;在未缓存该中间分析结果的情况下,执行下述步骤904-B。
在上述过程中,通过判断是否缓存了中间分析结果,能够在已缓存有中间分析结果的情况下,避免重复计算出第一时刻下的全量分析结果,因此能够大大节约计算设备的计算资源。
904-A、在已缓存该中间分析结果的情况下,计算设备将该中间分析结果确定为全量分析结果。
也即是说,如果缓存了从开始时刻到第一时刻之间任一第二时刻的中间分析结果,那么直接将以该中间分析结果作为全量分析结果,而无需对全量分析结果进行冗余计算,能够大大节约计算设备的计算资源。
904-B、在未缓存该中间分析结果的情况下,计算设备生成对该第一时刻可见的第二历史态数据的全量分析结果。
在一些实施例中,如果未缓存该中间分析结果,那么计算设备需要运用单一时间段可见性判断算法,从该历史态数据中,判断出对第一子区间[开始时刻,第一时刻)可见的第二历史态数据,也即是说,仅针对第一子区间[开始时刻,第一时刻)的右端值第一时刻,执行单一时刻的可见性判断算法,从而确定出该第二历史态数据。
在从各个版本的历史态数据中,确定出第二历史态数据之后,基于上述步骤901中解析得到的分析语义,对该第二历史态数据进行该分析语义所对应的处理,得到该全量分析结果。比如,分析语义为对某个字段值求平均值,那么需要对各个第二历史态数据对应字段的Value值求平均值,又例如,分析语义为对某个字段值求和,那么需要将各个第二历史态数据对应字段的Value值相加,以获取相加得到的和值。
在上述步骤903-904A或904B中,计算设备基于该历史态数据,获取全量分析结果,该全量分析结果为基于该待分析事务的分析语义对该历史态数据进行处理所得的结果,从而能够将步骤904-A或904-B的全量分析结果与下述步骤905的增量分析结果合并,得到最终分析结果,即将单层计算模型转换为全量+增量的双层计算模型。
在一些实施例中,由于HTAP***或HTASP***的存储集群可按照冷热等级的划分标准,划分为对应的分区,并在各个分区内的存储设备中存储对应冷热等级的历史态数据。换言之,冷热等级不同的历史态数据存储在不同的存储设备或者同一存储设备的不同存储介质中。因此,计算设备在读取该第二历史态数据时,先确定该第二历史态数据的冷热等级,该冷热等级用于表征在历史时间段内对在该第二历史态数据上发生操作的频繁程度;再从与该冷热等级对应的存储设备中,读取该第二历史态数据。
在一些实施例中,计算设备在确定该第二历史态数据的冷热等级时,获取该历史时间段内对该第二历史态数据的访问次数,获取该历史时间段内对该第二历史态数据对应的当前态数据的修改次数,基于该访问次数和该修改次数,确定该冷热等级。其中,该历史时间段是指管理用户设定的当前数据库下冷热等级的划分标准中需要判断的时间段(例如1天、1月、1年等)。
在一种可能的实现方式中,由于在介绍上述冷热等级的划分标准时,提到过对全时态数据库中所有的当前态数据,增加一个修改标记信息来统计修改次数,此外,对所有的当前态数据、过渡态数据和历史态数据,都增加一个访问标记信息来统计访问次数,因此,只要对每个第二历史态数据(实际上是在第一时刻下该历史态数据的可见版本),查询该第二历史态数据对应的当前态数据的修改标记信息,即可得到该修改次数,查询该第二历史态数据自身的访问标记信息,即可得到该访问次数。
在一种可能的实现方式中,在获取到访问次数和修改次数之后,基于当前数据库的冷热等级的划分标准,查看该访问次数和修改次数落入到哪一种冷热等级的划分标准下的取值区间中,即可确定出该第二历史态数据的冷热等级。
905、计算设备基于在该历史态数据对应的当前态数据上发生的特定变更,获取增量分析结果,该特定变更为在该第一时刻起至该查询时间段的结束时刻之间发生的变化,该增量分析结果为基于该分析语义对该特定变更得到的数据进行处理所得的结果。
由于第一时刻是对应的历史态数据最近一次完成转储的时刻,那么代表第一时刻起到结束时刻之间的数据都尚未从OLTP集群转储到存储集群中,因此,需要先从OLTP集群中获取到对应特定变更产生的日志,对特定变更产生的日志进行回放,获取到对应特定变更得到的数据,接着,比较主键标识相同的历史态数据和特定变更得到的数据,并按照待分析事务的SQL语句的分析语义对比较结果进行处理,生成该增量分析结果。其中,该增量分析结果只代表与上一次全量分析结果之间发生的变化,而并非是新一次计算所得的全量分析结果。
在上述过程中,通过对特定变更得到的数据仅获取增量分析结果,而无需获取全量分析结果,由于增量分析结果所需分析的数据量较少、且耗时较少,因此能够大大节约计算设备的计算开销和处理资源。
在一些实施例中,该增量分析结果包括特定变更前的分析结果和特定变更后的分析结果,计算设备在比较主键标识相同的历史态数据和特定变更得到的数据的时候,需要先对主键标识相同的历史态数据和特定变更得到的数据,确定发生特定变更的字段,需要说明的是,这里主键标识相同的历史态数据和特定变更得到的数据,是指针对具有该主键标识 的数据记录在不同时刻进行特定变更时产生的不同版本,比如,对记录R1的历史态V1版本和最新特定变更得到的V4版本,确定发生特定变更的字段。
在确定发生特定变更的字段的基础上,如果该数据查询条件不涉及该发生特定变更的字段,代表待分析事务所查询的字段没有变化,因此可跳过具有该主键标识的数据,比如,如果记录R1的V1版本和V4版本之间发生变化的是字段c1,而其余字段没有变化,此时数据查询条件所查询的是字段c2,代表第一时刻到结束时刻之间没有增量变化,那么直接跳过记录R1即可;如果该数据查询条件涉及该发生特定变更的字段,那么需要分别计算一个特定变更前的分析结果和一个特定变更后的分析结果,也即是说,对该第一时刻可见的第二历史态数据,生成该特定变更前的分析结果,对该结束时刻可见的特定变更得到的目标数据,生成该特定变更后的分析结果,例如,如果记录R1的V1版本和V4版本之间发生变化的是字段c1,恰好数据查询条件所查询的也是字段c1,那么记录R1需要参与到增量计算中,因此,在统计到所有需要参与增量计算的记录之后,需要对各个记录的字段c1的旧值计算特定变更前的分析结果,对各个记录的字段c1的新值计算特定变更后的分析结果。
906、计算设备基于该全量分析结果和该增量分析结果,获取该待分析事务的最终分析结果。
在一些实施例中,该增量分析结果包括特定变更前的分析结果和特定变更后的分析结果,因此,在合并全量分析结果和增量分析结果时,需要将该全量分析结果减去该特定变更前的分析结果,得到第一分析结果;将该第一分析结果加上该特定变更后的分析结果,得到该最终分析结果。
在上述过程中,通过全量计算和增量计算结合以获取最终分析结果的方式,能够将传统的单层计算模型转换为两层实时分析模型,能够适配于基于全时态数据库的HTAP***或HTASP***,且能够节约整个***内的计算资源。
上述所有可选技术方案,能够采用任意结合形成本公开的可选实施例,在此不再一一赘述。
本申请实施例提供的方法,通过针对待分析事务确定查询时间段,并以第一时刻作为查询时间段内的一个时间点分界线,对第一时刻之前的历史态数据计算出一个全量分析结果,对第一时刻之后特定变更得到的数据计算出一个增量分析结果,将全量分析结果和增量分析结果两者结合,最终得到最终分析结果,使得针对全时态的时间轴上的任一个查询时间段,都能够提供在线实时分析和处理服务,而并不局限于仅对最新数据提供分析功能,从而大大提升了HTAP***或HTASP***等数据库***的事务处理性能。
通过全量计算和增量计算结合以获取最终分析结果的方式,能够将传统的单层计算模型转换为两层实时分析模型,能够适配于基于全时态数据库的HTAP***或HTASP***,且能够节约整个***内的计算资源。
在上述实施例中,详细介绍了HTAP***或HTASP***等数据库***的事务处理流程,而在本申请实施例中,将结合一个具体的数据查询语句,来详细描述OLAP集群中的计算设备是如何通过实时分析模型来获取最终分析结果的。
图10是本申请实施例提供的一种实时分析模型的原理性流程图,如1000所示,将介绍在实时程度参数REAL_TIME_ANALYZING DEGREE设置为1秒的情况下,***某一时刻进来一条数据查询语句(即待分析事务为查询事务),计算设备会如何对该数据查询语句进行处理。假设该数据查询语句为:
select sum(a)from T1 where a>1;
在一些实施例中,用户可在该数据查询语句中增加对应的时间语法来限定查询的目标数据集,假设用户和一些通常的分析查询一样没有对设置任何时间态信息,那么本条数据查询语句的在线实时分析流程(即实时计算算法)包括:
步骤1、开始执行分析查询。
在步骤1中,需要针对该查询事务进行事务时间范围的增加,例如,首先,确定实时程度参数REAL_TIME_ANALYZING_DEGREE的值,即确定用户对于OLAP集群的实时性要求(假设REAL_TIME_ANALYZING_DEGREE设置为1秒),即可将该数据查询语句修改为:
select sum(a)from T1 FOR SYSTEM_TIME FROM min_time TO current_time–REAL_TIME_ANALYZING_DEGREE where a>1;
其中,min_time是指查询时间段的开始时刻,current_time是指当前时刻,current_time–REAL_TIME_ANALYZING_DEGREE是指查询时间段的结束时刻。
步骤2、判断是否在warm层有一个中间计算(即中间分析结果),如果warm层有中间计算,进入步骤4;如果warm层没有中间计算,进入步骤3。
在开始执行查询分析前,先判断一下之前是否有保存的warm层的中间计算结果(即中间分析结果),这里warm层是指温数据所在的存储分区,即在存储集群中温数据所在分区下的存储设备中查询是否缓存该中间分析结果。
需要说明的是,基于上述4层冷热等级的划分方式,通常实时分析的查询事务发生在hot和warm层(即大部分分析型事务都需要访问热数据和/或温数据),cold和archive层数据主要是用来分离存储过旧的历史态数据(即分析型事务通常不会访问冷数据和/或归档数据,这些冷数据和/或归档数据只是为了分离出来以减轻存储压力)。
步骤3、分两段查询执行,进入步骤7。
如果warm层中没有中间分析结果,代表之前的查询事务没有执行过对应数据的分析任务,或者之前的查询事务虽然执行了对应数据的分析任务,但是用户没有选择保存中间分析结果,因此,需要重新生成一个warm层的全量分析结果。
换言之,将上述数据查询语句中的单个查询时间段[min_time,current_time–REAL_TIME_ANALYZING_DEGREE]转换为如下两段时间段:[min_time,time1)和[time1,current_time–REAL_TIME_ANALYZING_DEGREE],其中,time1为由上述实施例中涉及的实时分析从副本通过数据转储程序,将主副本或其他从副本的实时数据通过后台线程转储到存储集群对应分区的存储设备上的最新时间段,也即,time1是指第一时刻(对符合该待分析事务的数据查询条件的历史态数据最近一次完成转储的时刻)。需要说明的 是,实时分析从副本中存储的数据中可能包括部分warm层的温数据(因为可能有一部分warm数据尚未转储完毕,但是并不影响对查询事务的分析计算)。
步骤4、分三段查询执行,进入步骤5。
如果warm层中有中间分析结果,那么将上述数据查询语句中的单个查询时间段[min_time,current_time–REAL_TIME_ANALYZING_DEGREE]转换为如下三段时间段:[min_time,timew),[timew,time1)和[time1,current_time–REAL_TIME_ANALYZING_DEGREE],根据之前介绍的多个时间段的可见性判断算法,最终只需要对最左端的时间段[min_time,timew)和最右端的时间段[time1,current_time–REAL_TIME_ANALYZING_DEGREE]这两个时间段进行计算。
步骤5、直接获取之前保存的warm层的全量分析结果(即中间分析结果),进入步骤6。
步骤6、计算增量部分的增量分析结果,进入步骤8。
换言之,计算[time1,current_time–REAL_TIME_ANALYZING_DEGREE]这部分特定变更得到的数据的结果集,增量计算所操作的该特定变更得到的数据通常处于hot层(通常是热数据),当然,如果用户对OLAP集群设置的实时程度参数REAL_TIME_ANALYZING_DEGREE比较大,说明用户对实时性要求不高,那么可能会导致增量计算所操作的该特定变更得到的数据落入到warm层中,此时对该特定变更得到的数据的增量计算将完全在warm层中能够完成,而无需与OLTP集群中hot层的相关TP计算节点进行交互,相当于本查询事务对于hot层的TP业务没有任何影响,能够完全解耦。
步骤7、并行计算warm层的全量分析结果和hot层的增量分析结果,进入步骤8。
根据步骤3所介绍的,可并行启动对上述两个时间段各自的分析结果的计算流程,并行计算相较于串行计算具有更高的计算效果。
首先,在warm层计算[min_time,time1)这一时间段的全量分析结果,并且,在计算得到该全量分析结果之后,如果用户设置保存该全量分析结果,那么将会在warm层中保存该全量分析结果,以备后续同类的查询事务参考,即将本次的全量分析结果缓存,以作为后续同类的查询事务在步骤2所查询的中间分析结果。可选地,warm层中可周期性地重新发起一次全量计算,避免由于长时间不进行全量计算,导致上一次的全量计算太旧,因此存在太多的数据特定变更导致所需要计算的数据集太大,所导致的查询性能降低的情况。
其次,在hot层计算[time1,current_time–REAL_TIME_ANALYZING_DEGREE]这一时间段内特定变更得到的数据的结果集(即增量分析结果)。
步骤8、将warm层的全量分析结果和hot层的增量分析结果进行合并,得到最终分析结果。
换言之,将worm层的全量分析结果和hot层特定变更得到的数据的结果集(即增量分析结果)进行合并,得到最终分析结果。
步骤9、返回本次查询结果,即返回合并得到的最终分析结果。
步骤10、实时分析结束。
在本申请实施例中,通过对worm层的全量分析结果和hot层的增量分析结果进行合并,提供一种两段式的实时计算模型,能够最大程度的降低OLTP集群和OLAP集群之间的耦合程度,即,OLAP集群在进行实时分析时,只需要通过实时分析从副本来扫描OLTP集群hot层中发生特定变更的数据即可,并且,如果用户不需要很高的实时性,有可能warm层的温数据就能够满足实时分析的计算要求,此时就无需访问hot层中的热数据,这种情况下对OLTP集群是完全解耦的,此外,由于warm层中缓存的中间计算结果是能够复用的,因此能够最大程度的节约计算资源。
在上述实施例的步骤7和步骤8中,涉及到对特定变更得到的数据计算增量分析结果,以及最终将全量分析结果和增量分析结果进行合并,在本申请实施例中,将详细描述这部分增量分析结果的计算方式,和全量+增量分析结果的合并方式,下面进行说明。
在实时计算模型中,将实时计算任务分解为全量+增量两层计算模式,通过在warm层生成一个全量分析结果,同时支持对该全量分析结果进行缓存以达到多次复用,来降低计算资源因为重复计算造成的浪费。同时,也因为将特定变更得到的数据这部分的计算量控制在一个合理的程度,使得整个***的实时分析能力对用户呈现出最优表现,提高了用户体验。
基于之前的针对多个时间段的可见性判断算法,可知worm层的全量计算的时间段是指多个时间段中最左端的时间段,hot层的增量计算的时间段是指多个时间段中最右端的时间段。
首先,计算或者获取到warm层的全量分析结果,如果已缓存有中间分析结果,则直接获取该中间分析结果为全量分析结果,如果没有缓存中间分析结果,那么需要重新计算出一个全量分析结果。
接着,对于特定变更得到的数据来说,由于需要分别计算特定变更前的分析结果和特定变更后的分析结果,特定变更前的分析结果是指各个记录的旧值,由于增量计算的时间段的左端值是第一时刻,因此上述旧值就是指在第一时刻的可见版本的相关字段值,特定变更后的分析结果是指各个记录的新值,由于增量计算的时间段的右端值是结束时刻(即当前时刻减去实时程度参数所得的时刻),因此上述新值就是指在结束时刻的可见版本的相关字段值。需要说明的是,为了使得主键标识相同的数据记录的各个新旧版本相对于,可在计算时对每个版本都添加标识信息row_id,使得对同一条数据记录进行多次特定变更所得的各个版本都具有相同的row_id。对特定变更得到的数据中的任一条记录,确定该记录在第一时刻的可见版本和在结束时刻的可见版本,并确定两个可见版本之间发生特定变更的字段,然后执行如下操作:
(A)如果发生特定变更的字段,在查询事务的数据查询语句中不使用,忽略本条记录;
(B)如果发生特定变更的字段,和查询事务的数据查询语句相关,那么,针对每一条记录的旧值重新计算一个特定变更前的分析结果;针对每一条记录的新值重新计算一个特定变更后的分析结果。
最后,将warm层的全量分析结果和(B)中对新旧值分别计算的特定变更前的分析结果和特定变更后的分析结果进行合并,也即,从warm层的全量分析结果中将(B)中特定 变更前的分析结果减去,然后在对减去所得的分析结果加上(B)中特定变更后的分析结果,得到最终的最终分析结果,此时特定变更合并算法结束。
在一些实施例中,在上述HTAP***的基础上,还可提供高效的流式实时分析的计算能力,以将HTAP***转换为HTASP***,从而达到最快速、最高效的完成实时分析计算,以将对***内部计算资源的使用降到最低,从而提高***的资源利用率。
示意性地,针对上述待分析事务,在流式计算引擎中,如果开启流式计算模式,假设待分析事务的最终分析结果是流式计算中的第一个全量分析结果,那么,从该待分析事务的查询时间段的结束时刻开始,每间隔第一时长(即是指之前实施例涉及的时间窗大小),获取一次该第一时长内发生特定变更的特定增量分析结果;缓存获取到的各个特定增量分析结果。每间隔第二时长(即是指之前实施例涉及的全量分析结果的计算周期),基于上一次的全量分析结果与该第二时长内产生的各个特定增量分析结果,获取本次的全量分析结果。
下面,将对增量+全量的流式计算模型进行介绍。
图11是本申请实施例提供的一种流式计算模型的数据记录的存储形态的原理性示意图,如1100所示,对每一条数据记录,随着事务对数据记录的特定变更操作,会保存整个时间轴上特定变更得到的各个版本,因此,通过一个个的时间窗,能够在时间轴上将数据记录的各个版本框在各自所属的时间窗内,其中,时间窗大小即第一时长可由之前实施例涉及的扩展命令来进行设置和查看,对每个时间窗内数据记录的可见版本进行增量计算,得到各个增量分析结果,此外,可基于全量分析结果的计算周期即第二时长,能够每间隔该计算周期划分一条时间线,并合并相邻的两条时间线中的各个时间窗内的所有各个增量分析结果,最终得到本次的全量分析结果。全量分析结果和增量分析结果的计算方式与之前实施例中的计算方式相同,这里不再赘述。
图12是本申请实施例提供的一种HTASP***的实施环境架构图,如图12所示,在该实时分析***中,包含计算层1210和分层分布式共享存储***1220,在计算层1210中包括OLTP集群1211、OLAP集群1212和流式计算集群1213,OLTP集群1211通常针对在当前态数据和过渡态数据上发起的短时交易型事务提供在线交易功能,OLAP集群1212通常针对在历史态数据上发起的复杂分析型事务提供在线分析功能。其中,流式计算集群1213可集成在OLAP集群1212内部,比如,对OLAP集群1212中的部分或全部AP计算节点加载流式计算引擎,以同时提供常规分析功能和流式分析功能。流式计算集群1213用于基于上述流式计算模型,定期进行增量计算和全量计算,从而源源不断输出连续时间段内的全量分析结果和实时分析结果,使得整个HTASP***具有很高的事务分析性能。
OLTP集群1211、OLAP集群1212和流式计算集群1213三者均能够访问到分层分布式共享存储***1220中存储的全时态数据,示意性地,按照全时态数据的冷热等级,将***内所有的数据划分为如下几种分区:热数据、温数据、冷数据、归档数据等,之前已经对冷热等级的划分标准进行过介绍,这里不再赘述。
由于在流式计算模型中,是定期进行全量分析结果的获取,并在相邻的两次全量计算之前,以时间窗为单元来不断进行增量分析结果的获取,但由于需要不断进行全量+增量的 流式计算,因此,针对用户指定了进行流式计算的分析型事务才提供流式计算功能,例如,用户在数据查询语句中添加Stream字段以指定开始流式计算。
在一些实施例中,可由***宏观分析各个用户发起的查询事务的规律,如果用户每间隔一段时间就会发起某一类相似的查询事务(比如每个月的还款日之前查询本月总的账单流水),那么可按照时间维度,根据用户发起的历史查询事务和历史查询事务所查询的数据特征,来创建针对该用户的查询业务的流式计算模型。在流式计算模型中,通过全量分析结果的计算周期将全时态数据进行第一层次的划分,在每个计算周期内,对各个特定变更得到的数据,按照查询的数据特征进一步细化出多个时间窗来进行增量分析结果的计算,从而提升计算能力的水平扩展性。需要说明的是,如果一个计算周期内的数据特定变更不多的话,那么可对本计算周期划分较少的时间窗或者不划分时间窗,而是直接对两个计算周期内的所有特定变更得到的数据进行增量计算。
示意性地,流式计算模型包括如下步骤:
步骤1、启动流式计算。
例如,第一个时间窗为实时计算的最左端的时间段,即流式计算的发起事务所指定的查询时间段中最左端的时间段,此时对该发起事务会计算出第一个启示的全量分析结果。
步骤2、对后续的每一个时间窗,判定可见的数据特定变更集合(即本时间窗内的特定变更操作所得的数据构成的集合),并对数据特定变更集合计算增量分析结果,但不做合并操作。
步骤3、每当到达全量分析结果的计算周期时,基于上一次的全量分析结果,和从上一次全量分析结果到当前时刻之间产生的所有增量分析结果,依次进行合并,得到本次的全量分析结果。
步骤4、当用户针对正在进行流式计算的相关数据发起新到的查询事务时,首先,找到最近的一次全量分析结果作为基础,接着,对该上一次的全量分析结果,依次合并之后的所有完整的时间窗的增量分析结果,最后,再合并从最后一个时间窗开始到下发该查询事务的时刻之间发生特定变更得到的数据的增量分析结果(这部分数据的时间跨度小于一个时间窗,因此需要单独计算)。
步骤5、返回新到的查询事务的合并结果。
在本申请实施例中,流式分析算法能够将实时分析计算转变为以时间窗为单位的少量的增量计算,并且随着时间窗的后移,和将任意计算周期内的各个增量分析结果与上一次的全量分析结果合并,得到新一次的全量分析结果,从而能够以较为稳定可预估的常数执行时间,来完成用户已经构建了流式计算的实时分析结果,并且,流式计算的计算效率不会因为数据量的增大而发生变化,几乎能够保持在一个很小且稳定的执行时间内,同时可根据***资源的忙闲程度,与用户发起查询业务的周期变化,来自动调整时间窗和计算周期的大小,来进一步地优化HTASP***的资源使用效率。
在上述各个实施例中,分别介绍了HTAP***、HTASP***对分析型事务的处理流程,主要涉及全量+增量两层计算模型和流式计算模型,而在一些实施例中,在根据冷热等级的 划分标准,将全时态数据划分为不同冷热程度的数据之后,还可基于数据的冷热程度,来优化存储集群的存储模型。
图13是本申请实施例提供的一种冷热数据的访问量比例和数据量分布的统计图,如1300所示,可发现,一个月内8%的活跃数据量,承担了91%的访问量,而一年以外的冷数据的访问量只占用了全部访问量的1%。因此,为了更好地构建基于全时态数据库的HTAP***或HTASP***的分布式架构,使得***能够持续对外提供稳定的访问质量,可根据数据的冷热等级的不同,将具有不同访问频率的数据存储在不同级别的存储介质中,从而既能够达到用户的SLA性能要求,又能够降低数据库的存储成本,从而提供高品质、低成本的存储解决方案。
示意性地,针对热、温、冷、归档这4类数据采取如下的4层存储模型:
热数据:存储在增强型SSD(Solid State Disk,固态硬盘)磁盘中,并基于Raft一致性协议保存一个主副本和两个从副本,确保数据的高可用性和高可靠性;
温数据:存储在SSD磁盘中,并基于Raft一致性协议保存一个主副本和两个从副本,确保数据的高可用性和高可靠性;
冷数据:使用COS(Cloud Object Storage,云对象存储)方式进行云存储,SATA(Serial Advanced Technology Attachment,串行ATA即串口硬盘)盘是一种廉价的存储设备,可以纠删码的方式进行冗余存储,提高12个9的可靠性。其中,12个9的可靠性是指:由于在每条记录存储的时候增加了纠删码,使得记录不需要存储多个副本也能够保持高容错、不丢失的效果,由于只增加了一些冗余校验来减少历史态数据的容量综合,因此带来了99.9999999999%的可靠性,即指存在0.0000000001%的丢失数据的概率,并且即便磁盘出现坏道,也可以根据上述纠删码来计算其余部分,从而算出来出错部分的内容。
归档数据:使用CAS(Content-Addressable Storage,内容寻址存储)方式进行云存储,SATA盘是廉价的存储设备,能够以极低成本保存归档数据,并采用纠删码方式替代多副本方式,能够提高存储密度,并使用可编程关闭电源的硬盘。
基于上述存储模型,用于可根据自身业务特定,灵活设定冷热等级的划分标准HOT_COLD_DATA_LEVEL参数,从而为对应冷热等级的数据配置合适的存储介质。
在上述各个实施例中,基于全时态数据库构建了一种能够支持对任意时间段的在线分析功能的HTAP***,且在HTAP***的基础上提供流式计算能力即HTASP***,通过对全态数据的时间维度的充分利用,能够有效将OLTP集群和OLAP集群解耦,使得OLTP集群在主副本上对最新数据提供在线交易和分析功能,OLAP集群在实时分析从副本和历史态数据上提供在线分析和处理功能,使得两个集群针对同一份数据能够相互独立地计算和运行。此外,管理用户可根据自己的业务需求,确定在线分析的实时程度参数,从而既能够满足业务需求,又能够量化OLTP集群和OLAP集群之间允许的最大延时。
在一种可能的实现方式中,引入了全量+增量计算的实时分析模式,能够最大程度节约计算资源的使用,以达到最好的实时分析效果,并且,由于时间态是针对每一条记录级别的,从而能够以记录粒度的时间态,对各个记录增加时间窗维度的划分,以结合原有的 数据特征对外提供流式计算能力,从而达到记录级别的最小粒度的任务划分,提升了计算效率。
在一种可能的实现方式中,用户可自定义冷热等级的划分标准,以提升数据库***整体的服务能力,并通过多级不同存储设备不同存储介质的分区存储,降低了数据库***整体的使用成本,由于全时态数据每次修改都会产生一条对应的历史记录(即历史态数据),利用这一特性,如果一定时间内数据记录没有发生修改,只需要将历史态数据(冷数据)转储到更加廉价、低速的存储设备上,即能够实现在数据库***内既保存全量历史态数据,又能够将冷数据与热数据分离,保证了***处理性能的稳定性,同时数据库的使用成本能够以一个较低的增幅,来匹配数据的***式增长。
图14是本申请实施例提供的一种事务处理装置的结构示意图,请参考图14,该装置包括:
确定模块1401,用于从待分析事务的查询时间段中确定第一时刻,所述第一时刻为对历史态数据最近一次完成转储的时刻,所述历史态数据为符合所述待分析事务的数据查询条件的数据;
第一获取模块1402,用于基于该历史态数据,获取全量分析结果,该全量分析结果为基于该待分析事务的分析语义对该历史态数据进行处理所得的结果;
第二获取模块1403,用于基于在所述历史态数据对应的当前态数据上发生的特定变更,获取增量分析结果,所述特定变更为在所述第一时刻起至所述查询时间段的结束时刻之间发生的变更,所述增量分析结果为基于所述分析语义对所述特定变更得到的数据进行处理所得的结果;
第三获取模块1404,用于基于该全量分析结果和该增量分析结果,获取该待分析事务的最终分析结果。
本申请实施例提供的装置,通过针对待分析事务确定查询时间段,并以第一时刻作为查询时间段内的一个时间点分界线,对第一时刻之前的历史态数据计算出一个全量分析结果,对第一时刻之后特定变更得到的数据计算出一个增量分析结果,将全量分析结果和增量分析结果两者结合,最终得到最终分析结果,使得针对全时态的时间轴上的任一个查询时间段,都能够提供在线实时分析和处理服务,而并不局限于仅对最新数据提供分析功能,从而大大提升了HTAP***或HTASP***等数据库***的事务处理性能。
在一种可能实施方式中,基于图14的装置组成,该第一获取模块1402包括:
查询单元,用于查询是否缓存对第二时刻可见的第一历史态数据的中间分析结果,该第二时刻为该查询时间段的开始时刻到该第一时刻之间的任一时刻;
确定单元,用于在已缓存该中间分析结果的情况下,将该中间分析结果确定为该全量分析结果;
第一生成单元,用于在未缓存该中间分析结果的情况下,生成对该第一时刻可见的第二历史态数据的该全量分析结果。
在一种可能实施方式中,基于图14的装置组成,该第一生成单元包括:
确定子单元,用于确定该第二历史态数据的冷热等级,该冷热等级用于表征在历史时间段内对在该第二历史态数据上发生操作的频繁程度;
读取子单元,用于从与该冷热等级对应的存储设备中,读取该第二历史态数据;
处理子单元,用于基于该分析语义,对该第二历史态数据进行处理,得到该全量分析结果。
在一种可能实施方式中,该确定子单元用于:
获取该历史时间段内对该第二历史态数据的访问次数;
获取该历史时间段内对该第二历史态数据对应的当前态数据的修改次数;
基于该访问次数和该修改次数,确定该冷热等级。
在一种可能实施方式中,冷热等级不同的历史态数据存储在不同的存储设备或者同一存储设备的不同存储介质中。
在一种可能实施方式中,基于图14的装置组成,该第二获取模块1403包括:
获取单元,用于基于该特定变更产生的日志,获取该特定变更得到的数据;
第二生成单元,用于比较主键标识相同的历史态数据和特定变更得到的数据,生成该增量分析结果。
在一种可能实施方式中,该增量分析结果包括特定变更前的分析结果和特定变更后的分析结果,该第二生成单元用于:
对主键标识相同的历史态数据和特定变更得到的数据,确定发生特定变更的字段;
如果该数据查询条件不涉及该发生特定变更的字段,跳过具有该主键标识的数据;
如果该数据查询条件涉及该发生特定变更的字段,对该第一时刻可见的第二历史态数据,生成该特定变更前的分析结果;对该结束时刻可见的特定变更得到的目标数据,生成该特定变更后的分析结果。
在一种可能实施方式中,该第三获取模块1404用于:
将该全量分析结果减去该特定变更前的分析结果,得到第一分析结果;
将该第一分析结果加上该特定变更后的分析结果,得到该最终分析结果。
在一种可能实施方式中,该确定模块1401还用于:
当该待分析事务未指定该查询时间段的开始时刻时,将符合该数据查询条件的数据的最新版本的生效时刻确定为该开始时刻;
当该待分析事务未指定该查询时间段的结束时刻时,将当前时刻之前距离该当前时刻达到实时程度参数的时刻确定为该结束时刻,该实时程度参数用于表征数据库***内的分析型事务处理的数据所支持的最大延时。
在一种可能实施方式中,该第二获取模块1403还用于:
从该结束时刻开始,每间隔第一时长,获取一次该第一时长内发生特定变更的特定增量分析结果;
缓存获取到的各个特定增量分析结果。
在一种可能实施方式中,该第三获取模块1404还用于:
每间隔第二时长,基于上一次的全量分析结果与该第二时长内产生的各个特定增量分析结果,获取本次的全量分析结果。
上述所有可选技术方案,能够采用任意结合形成本公开的可选实施例,在此不再一一赘述。
需要说明的是:上述实施例提供的事务处理装置在处理事务时,仅以上述各功能模块的划分进行举例说明,实际应用中,能够根据需要而将上述功能分配由不同的功能模块完成,即将计算设备的内部结构划分成不同的功能模块,以完成以上描述的全部或者部分功能。另外,上述实施例提供的事务处理装置与事务处理方法实施例属于同一构思,其具体实现过程详见事务处理方法实施例,这里不再赘述。
图15是本申请实施例提供的一种计算设备的结构示意图,如图15所示,以计算设备为终端1500为例进行说明。在一种可能的实现方式中,该终端1500的设备类型包括:智能手机、平板电脑、MP3播放器(Moving Picture Experts Group Audio Layer III,动态影像专家压缩标准音频层面3)、MP4(Moving Picture Experts Group Audio Layer IV,动态影像专家压缩标准音频层面4)播放器、笔记本电脑或台式电脑。终端1500还可能被称为用户设备、便携式终端、膝上型终端、台式终端等其他名称。
通常,终端1500包括有:处理器1501和存储器1502。
在一种可能的实现方式中,处理器1501包括一个或多个处理核心,比如4核心处理器、8核心处理器等。处理器1501可以采用DSP(Digital Signal Processing,数字信号处理)、FPGA(Field-Programmable Gate Array,现场可编程门阵列)、PLA(Programmable Logic Array,可编程逻辑阵列)中的至少一种硬件形式来实现。在一些实施例中,处理器1501包括主处理器和协处理器,主处理器是用于对在唤醒状态下的数据进行处理的处理器,也称CPU(Central Processing Unit,中央处理器);协处理器是用于对在待机状态下的数据进行处理的低功耗处理器。在一些实施例中,处理器1501集成有GPU(Graphics Processing Unit,图像处理器),GPU用于负责显示屏所需要显示的内容的渲染和绘制。一些实施例中,处理器1501还包括AI(Artificial Intelligence,人工智能)处理器,该AI处理器用于处理有关机器学习的计算操作。
在一些实施例中,存储器1502包括一个或多个计算机可读存储介质,可选地,该计算机可读存储介质是非暂态的。存储器1502还包括高速随机存取存储器,以及非易失性存储器,比如一个或多个磁盘存储设备、闪存存储设备。在一些实施例中,存储器1502中的非暂态的计算机可读存储介质用于存储至少一个程序代码,该至少一个程序代码用于被处理器1501所执行以实现本申请中各个实施例提供的事务处理方法。
在一些实施例中,终端1500还可选包括有:***设备接口1503和至少一个***设备。处理器1501、存储器1502和***设备接口1503之间能够通过总线或信号线相连。各个***设备能够通过总线、信号线或电路板与***设备接口1503相连。具体地,***设备包括:射频电路1504、显示屏1505、摄像头组件1506、音频电路1507、定位组件1508和电源1509中的至少一种。
在一些实施例中,终端1500还包括有一个或多个传感器1510。该一个或多个传感器1510包括但不限于:加速度传感器1511、陀螺仪传感器1512、压力传感器1513、指纹传感器1514、光学传感器1515以及接近传感器1516。
本领域技术人员能够理解,图15中示出的结构并不构成对终端1500的限定,能够包括比图示更多或更少的组件,或者组合某些组件,或者采用不同的组件布置。
图16是本申请实施例提供的一种计算设备的结构示意图,该计算设备1600可因配置或性能不同而产生比较大的差异,该计算设备1600包括一个或一个以上处理器(Central Processing Units,CPU)1601和一个或一个以上的存储器1602,其中,该存储器1602中存储有至少一条计算机程序,该至少一条计算机程序由该一个或一个以上处理器1601加载并执行以实现上述各个实施例提供的事务处理方法。该计算设备1600还具有有线或无线网络接口、键盘以及输入输出接口等部件,以便进行输入输出,该计算设备1600还包括其他用于实现设备功能的部件,在此不做赘述。
在示例性实施例中,还提供了一种计算机可读存储介质,例如包括至少一条计算机程序的存储器,上述至少一条计算机程序可由终端中的处理器执行以完成上述各个实施例中的事务处理方法。例如,该计算机可读存储介质包括ROM(Read-Only Memory,只读存储器)、RAM(Random-Access Memory,随机存取存储器)、CD-ROM(Compact Disc Read-Only Memory,只读光盘)、磁带、软盘和光数据存储设备等。
在示例性实施例中,还提供了一种计算机程序产品或计算机程序,包括一条或多条程序代码,该一条或多条程序代码存储在计算机可读存储介质中。计算设备的一个或多个处理器能够从计算机可读存储介质中读取该一条或多条程序代码,该一个或多个处理器执行该一条或多条程序代码,使得计算设备能够执行以完成上述实施例中的事务处理方法。
本领域普通技术人员能够理解实现上述实施例的全部或部分步骤能够通过硬件来完成,也能够通过程序来指令相关的硬件完成,该程序存储于一种计算机可读存储介质中,上述提到的存储介质是只读存储器、磁盘或光盘等。
以上所述仅为本申请的可选实施例,并不用以限制本申请,凡在本申请的精神和原则之内,所作的任何修改、等同替换、改进等,均应包含在本申请的保护范围之内。
Claims (15)
- 一种事务处理方法,所述方法由计算设备执行,所述方法包括:从待分析事务的查询时间段中确定第一时刻,所述第一时刻为对历史态数据最近一次完成转储的时刻,所述历史态数据为符合所述待分析事务的数据查询条件的数据;基于所述历史态数据,获取全量分析结果,所述全量分析结果为基于所述待分析事务的分析语义对所述历史态数据进行处理所得的结果;基于在所述历史态数据对应的当前态数据上发生的特定变更,获取增量分析结果,所述特定变更为在所述第一时刻起至所述查询时间段的结束时刻之间发生的变更,所述增量分析结果为基于所述分析语义对所述特定变更得到的数据进行处理所得的结果;基于所述全量分析结果和所述增量分析结果,获取所述待分析事务的最终分析结果。
- 根据权利要求1所述的方法,所述基于所述历史态数据,获取全量分析结果包括:查询是否缓存对第二时刻可见的第一历史态数据的中间分析结果,所述第二时刻为所述查询时间段的开始时刻到所述第一时刻之间的任一时刻;在已缓存所述中间分析结果的情况下,将所述中间分析结果确定为所述全量分析结果;在未缓存所述中间分析结果的情况下,生成对所述第一时刻可见的第二历史态数据的所述全量分析结果。
- 根据权利要求2所述的方法,所述生成对所述第一时刻可见的第二历史态数据的所述全量分析结果包括:确定所述第二历史态数据的冷热等级,所述冷热等级用于表征在历史时间段内对在所述第二历史态数据上发生操作的频繁程度;从与所述冷热等级对应的存储设备中,读取所述第二历史态数据;基于所述分析语义,对所述第二历史态数据进行处理,得到所述全量分析结果。
- 根据权利要求3所述的方法,所述确定所述第二历史态数据的冷热等级包括:获取所述历史时间段内对所述第二历史态数据的访问次数;获取所述历史时间段内对所述第二历史态数据对应的当前态数据的修改次数;基于所述访问次数和所述修改次数,确定所述冷热等级。
- 根据权利要求3或4所述的方法,冷热等级不同的历史态数据存储在不同的存储设备或者同一存储设备的不同存储介质中。
- 根据权利要求1所述的方法,所述基于在所述历史态数据对应的当前态数据上发生的特定变更,获取增量分析结果包括:基于所述特定变更产生的日志,获取所述特定变更得到的数据;比较主键标识相同的历史态数据和特定变更得到的数据,生成所述增量分析结果。
- 根据权利要求6所述的方法,所述增量分析结果包括特定变更前的分析结果和特定变更后的分析结果,所述比较主键标识相同的历史态数据和特定变更得到的数据,生成所述增量分析结果包括:对主键标识相同的历史态数据和特定变更得到的数据,确定发生特定变更的字段;如果所述数据查询条件不涉及所述发生特定变更的字段,跳过具有所述主键标识的数据;如果所述数据查询条件涉及所述发生特定变更的字段,对所述第一时刻可见的第二历史态数据,生成所述特定变更前的分析结果;对所述结束时刻可见的特定变更得到的数据,生成所述特定变更后的分析结果。
- 根据权利要求7所述的方法,所述基于所述全量分析结果和所述增量分析结果,获取所述待分析事务的最终分析结果包括:将所述全量分析结果减去所述特定变更前的分析结果,得到第一分析结果;将所述第一分析结果加上所述特定变更后的分析结果,得到所述最终分析结果。
- 根据权利要求1所述的方法,所述方法还包括:当所述待分析事务未指定所述查询时间段的开始时刻时,将符合所述数据查询条件的数据的最新版本的生效时刻确定为所述开始时刻;当所述待分析事务未指定所述查询时间段的结束时刻时,将当前时刻之前距离所述当前时刻达到实时程度参数的时刻确定为所述结束时刻,所述实时程度参数用于表征数据库***内的分析型事务处理的数据所支持的最大延时。
- 根据权利要求1所述的方法,所述方法还包括:从所述结束时刻开始,每间隔第一时长,获取一次所述第一时长内发生特定变更的特定增量分析结果;缓存获取到的各个特定增量分析结果。
- 根据权利要求10所述的方法,所述方法还包括:每间隔第二时长,基于上一次的全量分析结果与所述第二时长内产生的各个特定增量分析结果,获取本次的全量分析结果。
- 一种事务处理装置,所述装置部署在计算设备上,所述装置包括:确定模块,用于从待分析事务的查询时间段中确定第一时刻,所述第一时刻为对历史态数据最近一次完成转储的时刻,所述历史态数据为符合所述待分析事务的数据查询条件的数据;第一获取模块,用于基于所述历史态数据,获取全量分析结果,所述全量分析结果为基于所述待分析事务的分析语义对所述历史态数据进行处理所得的结果;第二获取模块,用于基于在所述历史态数据对应的当前态数据上发生的特定变更,获取增量分析结果,所述特定变更为在所述第一时刻起至所述查询时间段的结束时刻之间发生的变更,所述增量分析结果为基于所述分析语义对所述特定变更得到的数据进行处理所得的结果;第三获取模块,用于基于所述全量分析结果和所述增量分析结果,获取所述待分析事务的最终分析结果。
- 一种计算设备,所述计算设备包括一个或多个处理器和一个或多个存储器,所述一个或多个存储器中存储有至少一条计算机程序,所述至少一条计算机程序由所述一个或多个处理器加载并执行以实现如权利要求1至权利要求11任一项所述的事务处理方法。
- 一种存储介质,所述存储介质中存储有至少一条计算机程序,所述至少一条计算机程序由处理器加载并执行以实现如权利要求1至权利要求11任一项所述的事务处理方法。
- 一种计算机程序产品,所述计算机程序产品包括至少一条计算机程序,所述至少一条计算机程序由处理器加载并执行以实现如权利要求1至权利要求11任一项所述的事务处理方法。
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US18/455,614 US20230418811A1 (en) | 2021-11-05 | 2023-08-24 | Transaction processing method and apparatus, computing device, and storage medium |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202111305207.6 | 2021-11-05 | ||
CN202111305207.6A CN115114344B (zh) | 2021-11-05 | 2021-11-05 | 事务处理方法、装置、计算设备及存储介质 |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US18/455,614 Continuation US20230418811A1 (en) | 2021-11-05 | 2023-08-24 | Transaction processing method and apparatus, computing device, and storage medium |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2023077971A1 true WO2023077971A1 (zh) | 2023-05-11 |
Family
ID=83325179
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2022/118941 WO2023077971A1 (zh) | 2021-11-05 | 2022-09-15 | 事务处理方法、装置、计算设备及存储介质 |
Country Status (3)
Country | Link |
---|---|
US (1) | US20230418811A1 (zh) |
CN (1) | CN115114344B (zh) |
WO (1) | WO2023077971A1 (zh) |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2023173753A (ja) * | 2022-05-26 | 2023-12-07 | 株式会社日立製作所 | 情報管理装置、情報管理方法およびプログラム |
CN117951141A (zh) * | 2022-10-21 | 2024-04-30 | 华为云计算技术有限公司 | 数据处理方法及装置 |
CN116821102B (zh) * | 2023-08-25 | 2023-11-17 | 腾讯科技(深圳)有限公司 | 数据迁移方法、装置、计算机设备和存储介质 |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101604326A (zh) * | 2009-07-16 | 2009-12-16 | 浙江大学 | 基于事件语义的土地利用数据更新与分析方法 |
CN109408330A (zh) * | 2018-10-15 | 2019-03-01 | 东软集团股份有限公司 | 日志分析方法、装置、终端设备及可读存储介质 |
US20190197524A1 (en) * | 2017-12-22 | 2019-06-27 | Idemia France | Collecting transaction history data on a terminal |
CN110196758A (zh) * | 2018-05-10 | 2019-09-03 | 腾讯科技(深圳)有限公司 | 数据处理方法和装置、存储介质及电子装置 |
Family Cites Families (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103544075B (zh) * | 2011-12-31 | 2017-07-07 | 华为数字技术(成都)有限公司 | 数据的处理方法和*** |
US10296508B2 (en) * | 2013-06-06 | 2019-05-21 | Sap Se | Systems and methods to manage online analytical and transactional processing for an in-memory columnar database |
CN106326317A (zh) * | 2015-07-09 | 2017-01-11 | ***通信集团山西有限公司 | 数据处理方法及装置 |
US10585873B2 (en) * | 2017-05-08 | 2020-03-10 | Sap Se | Atomic processing of compound database transactions that modify a metadata entity |
US11573964B2 (en) * | 2019-07-17 | 2023-02-07 | At&T Intellectual Property I, L.P. | Reducing database system query transaction delay |
CN110647579A (zh) * | 2019-08-16 | 2020-01-03 | 北京百度网讯科技有限公司 | 数据同步方法及装置、计算机设备与可读介质 |
CN112825069B (zh) * | 2019-11-21 | 2024-05-24 | 阿里巴巴集团控股有限公司 | 数据库数据的分析方法、设备、***及存储介质 |
CN113312330A (zh) * | 2020-02-27 | 2021-08-27 | 华为技术有限公司 | 一种数据处理方法、装置、设备及介质 |
CN111930848B (zh) * | 2020-09-17 | 2021-04-13 | 阿里云计算有限公司 | 数据分区存储方法、装置及*** |
CN112527894B (zh) * | 2020-11-27 | 2023-04-14 | 聚好看科技股份有限公司 | 一种数据库一致性校验方法及*** |
CN112463311B (zh) * | 2021-01-28 | 2021-06-08 | 腾讯科技(深圳)有限公司 | 事务处理方法、装置、计算机设备及存储介质 |
CN113206757B (zh) * | 2021-04-25 | 2022-04-19 | 烽火通信科技股份有限公司 | 流式同步网管配置全量数据和增量数据的方法及电子设备 |
CN113535656B (zh) * | 2021-06-25 | 2022-08-09 | 中国人民大学 | 数据访问方法、装置、设备及存储介质 |
-
2021
- 2021-11-05 CN CN202111305207.6A patent/CN115114344B/zh active Active
-
2022
- 2022-09-15 WO PCT/CN2022/118941 patent/WO2023077971A1/zh unknown
-
2023
- 2023-08-24 US US18/455,614 patent/US20230418811A1/en active Pending
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101604326A (zh) * | 2009-07-16 | 2009-12-16 | 浙江大学 | 基于事件语义的土地利用数据更新与分析方法 |
US20190197524A1 (en) * | 2017-12-22 | 2019-06-27 | Idemia France | Collecting transaction history data on a terminal |
CN110196758A (zh) * | 2018-05-10 | 2019-09-03 | 腾讯科技(深圳)有限公司 | 数据处理方法和装置、存储介质及电子装置 |
CN109408330A (zh) * | 2018-10-15 | 2019-03-01 | 东软集团股份有限公司 | 日志分析方法、装置、终端设备及可读存储介质 |
Also Published As
Publication number | Publication date |
---|---|
CN115114344B (zh) | 2023-06-23 |
CN115114344A (zh) | 2022-09-27 |
US20230418811A1 (en) | 2023-12-28 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN111338766B (zh) | 事务处理方法、装置、计算机设备及存储介质 | |
US10860612B2 (en) | Parallel replication across formats | |
WO2023077971A1 (zh) | 事务处理方法、装置、计算设备及存储介质 | |
TWI759789B (zh) | 電腦實現的方法、非暫態的電腦可讀儲存媒體及儲存系統 | |
Bailis et al. | Bolt-on causal consistency | |
TWI766309B (zh) | 日誌結構儲存系統 | |
EP4254183A1 (en) | Transaction processing method and apparatus, computer device, and storage medium | |
CN111597015B (zh) | 事务处理方法、装置、计算机设备及存储介质 | |
US8078582B2 (en) | Data change ordering in multi-log based replication | |
US9996427B2 (en) | Parallel backup for distributed database system environments | |
CN113535656B (zh) | 数据访问方法、装置、设备及存储介质 | |
WO2018157602A1 (zh) | 一种同步活动事务表的方法及装置 | |
WO2013155752A1 (zh) | 面向数据库与Hadoop混合平台的OLAP查询处理方法 | |
US10275400B1 (en) | Systems and methods for forming a fault-tolerant federated distributed database | |
WO2019103950A1 (en) | Multi-region, multi-master replication of database tables | |
JPWO2011108695A1 (ja) | 並列データ処理システム、並列データ処理方法及びプログラム | |
WO2019109854A1 (zh) | 分布式数据库数据处理方法、装置、存储介质及电子装置 | |
KR20090059859A (ko) | 분산파일 시스템에서의 비동기식 데이터 복제 방법 및 그에따른 분산파일 시스템 | |
JP2023541298A (ja) | トランザクション処理方法、システム、装置、機器、及びプログラム | |
CN115114374B (zh) | 事务执行方法、装置、计算设备及存储介质 | |
CN115114294A (zh) | 数据库存储模式的自适应方法、装置、计算机设备 | |
CN114741453A (zh) | 数据同步的方法、***及计算机可读存储介质 | |
US10387384B1 (en) | Method and system for semantic metadata compression in a two-tier storage system using copy-on-write | |
CN111459913B (zh) | 分布式数据库的容量扩展方法、装置及电子设备 | |
JP6626976B2 (ja) | 高スループット、高信頼性のデータ処理システム |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 22888998 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |