WO2021004472A1 - 一种航空运价数据的缓存方法及*** - Google Patents

一种航空运价数据的缓存方法及*** Download PDF

Info

Publication number
WO2021004472A1
WO2021004472A1 PCT/CN2020/100802 CN2020100802W WO2021004472A1 WO 2021004472 A1 WO2021004472 A1 WO 2021004472A1 CN 2020100802 W CN2020100802 W CN 2020100802W WO 2021004472 A1 WO2021004472 A1 WO 2021004472A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
air freight
level cache
node
air
Prior art date
Application number
PCT/CN2020/100802
Other languages
English (en)
French (fr)
Inventor
杜晋芳
孟令斌
文雯
鞠春生
刘冰
费勇博
Original Assignee
中国民航信息网络股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中国民航信息网络股份有限公司 filed Critical 中国民航信息网络股份有限公司
Priority to EP20836693.0A priority Critical patent/EP3998535A4/en
Priority to US17/625,702 priority patent/US11983109B2/en
Publication of WO2021004472A1 publication Critical patent/WO2021004472A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G06F16/2282—Tablespace storage structures; Management thereof
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02—Addressing or allocation; Relocation
    • G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
    • G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/06—Buying, selling or leasing transactions
    • G06Q30/0601—Electronic shopping [e-shopping]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G06F16/24552—Database cache management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G06F16/24553—Query execution of query operations
    • G06F16/24554—Unary operations; Data partitioning operations
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems
    • G06F16/258—Data format conversion from or to a database
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G06F16/278—Data partitioning, e.g. horizontal or vertical partitioning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/02—Reservations, e.g. for tickets, services or events
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/02—Marketing; Price estimation or determination; Fundraising
    • G06Q30/0283—Price estimation or determination
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/06—Buying, selling or leasing transactions
    • G06Q30/0601—Electronic shopping [e-shopping]
    • G06Q30/0611—Request for offers or quotes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
    • G06Q50/40—Business processes related to the transportation industry
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
    • G06F2212/60—Details of cache memory

Definitions

  • the cache system Since the data copies cached in the prior art are all full sets, the cache system has no horizontal scalability. As the total amount of cached data increases, memory resources will become a bottleneck, resulting in poor horizontal scalability of the cache system.
  • the present invention discloses a method and system for caching air freight data to improve the horizontal expansion capability of the caching system.
  • a method for caching air freight data including:
  • the air freight data stored in the first-level cache is fragmented based on the flight origin city and the flight destination city to generate fragmented data, wherein each data node in the fragmented data is Containing a copy of the full set of air freight data with unavailable sharding algorithms and a data subset of air freight data with available sharding algorithms, each of the data subsets composing a full set of data on air freight data with available sharding algorithms;
  • the first level cache is a Redis cluster.
  • the second-level cache is Linux kernel-level shared memory.
  • the segmentation of the air freight data stored in the first-level cache based on the flight departure city and the flight destination city to generate segmented data specifically includes:
  • the node configuration information includes: node name and node address, and the data structure in the first preset list is ⁇ node name, node address, number of data items ,Datasheets ⁇ ;
  • each data structure in the first preset list is used as metadata, and each metadata is merged with a copy of the complete set of air freight data for which the fragmentation algorithm is not available, to generate the fragment Each data node of the data.
  • Optional also includes:
  • a preset fusion algorithm is used for the incremental data to obtain a fusion result of the first level cache and the second level cache.
  • the preset fusion algorithm is: discarding data whose operation time is earlier than a first time in the incremental data, where the first time is: storing the accessed air freight data in all The time when the data in the secondary cache is placed in the second preset list; if the operation of the incremental data is delete, then the corresponding data in the second preset list is deleted; if the incremental data If the operation of is modifying, the corresponding data in the second preset list is modified; if the operation of the incremental data is inserting data, then the corresponding data is inserted in the second preset list.
  • a caching system for air freight data includes:
  • the conversion unit is used to convert air freight data into a data format of the first-level cache and store it in the first-level cache;
  • the generating unit is configured to perform data fragmentation on the air freight data stored in the first-level cache based on the flight departure city and the flight destination city to generate fragmented data, wherein each of the fragmented data
  • the data nodes all contain a copy of the full set of air freight data for which the sharding algorithm is not available and a data subset of the air freight data for which the sharding algorithm is available. Each of the data subsets constitutes the air freight rate for the available sharding algorithm.
  • the storage unit is configured to store the fragmented data in the secondary cache after it takes effect.
  • the generating unit specifically includes:
  • the classification subunit is used to classify the air freight data of the available fragmentation algorithm stored in the first level cache into m different data subsets according to the flight origin city and the flight destination city, where m is a positive integer;
  • the sorting subunit is used to sort each of the data subsets in descending order according to the number of data items contained;
  • the add subunit is used to add the obtained node configuration information to the first preset list, where the node configuration information includes: node name and node address, and the data structure in the first preset list is ⁇ node name, Node address, number of data, data list ⁇ ;
  • the data placement subunit is used to place the air freight data in each data subset into the data list with the least number of data items in the first preset list according to the storage order, and to collect the data subsets
  • the air freight data that has been put in the data list is deleted until the air freight data included in the data subset is cleared;
  • a generating subunit configured to use the data in each data structure in the first preset list as metadata, and merge each metadata with a copy of the full set of air freight data for which fragmentation algorithms are not available To generate each data node of the fragmented data.
  • Optional also includes:
  • a judging unit configured to judge whether the accessed air freight data has incremental data in the first level cache
  • a reading unit configured to read the incremental data from the first-level cache when the judgment unit judges yes;
  • the fusion unit is configured to apply a preset fusion algorithm to the incremental data to obtain a fusion result of the first level cache and the second level cache.
  • the present invention discloses a method and system for caching air freight data.
  • the air freight data is converted into the data format of the first level cache, and stored in the first level cache, and then the first level cache
  • the air freight data stored in the data is fragmented based on the flight origin city and the flight destination city to generate fragmented data.
  • each data node of the fragmented data contains the air freight rate of the unavailable fragmentation algorithm
  • Each data subset composes a full set of air fare data that can be sharded.
  • After the fragmented data is valid, it will be stored in the secondary cache. .
  • each data node of the sharded data cached to the second-level cache contains a copy of the full set of air freight data for which the sharding algorithm is not available, and a data subset of the air freight data for which the sharding algorithm is available, That is to say, each data node contains only part of the air freight data of the available fragmentation algorithm. Therefore, compared to the full set of data cached in the prior art, the present invention improves the horizontal expansion capability of the cache system.
  • Fig. 1 is a diagram of a distributed secondary cache architecture adopted by a method for caching air freight data disclosed in an embodiment of the present invention
  • FIG. 2 is a flowchart of a method for caching air freight data disclosed in an embodiment of the present invention
  • FIG. 3 is a schematic diagram of data fragmentation disclosed in an embodiment of the present invention.
  • FIG. 4 is a flow chart of a method for fragmenting air freight data stored in the first-level cache based on the flight origin city and the flight destination city to generate fragmented data according to an embodiment of the present invention
  • FIG. 5 is a flow chart of a method for fusing the first level cache and the second level cache disclosed in an embodiment of the present invention
  • Fig. 6 is a schematic structural diagram of a caching system for air freight data disclosed in an embodiment of the present invention.
  • the embodiment of the present invention discloses a method and system for caching air freight data.
  • the air freight data is converted into the data format of the first-level cache, and stored in the first-level cache, and then the air freight stored in the first-level cache is checked.
  • Price data based on the flight departure city and flight destination city to generate data fragments, where each data node of the fragmented data contains a full set of data copies of the air freight data for which the fragmentation algorithm is not available
  • the data subset of the air freight data of the available sharding algorithm each data subset constitutes the complete data of the air freight data of the available sharding algorithm, and the sharded data is stored in the second-level cache after it takes effect.
  • each data node of the sharded data cached to the second-level cache contains a copy of the full set of air freight data for which the sharding algorithm is not available, and a data subset of the air freight data for which the sharding algorithm is available, That is to say, each data node contains only part of the air freight data of the available fragmentation algorithm. Therefore, compared to the full set of data cached in the prior art, the present invention improves the horizontal expansion capability of the cache system.
  • the distributed second-level cache architecture used in a method for caching air freight data disclosed in the present invention.
  • the dotted box in Fig. 1 identifies the components of the distributed second-level cache architecture.
  • the architecture mainly includes: a data synchronization service module, a first-level cache, a second-level cache update module, a second-level cache, and a data access interface module.
  • the distributed second-level cache architecture includes two first-level caches and multiple second-level caches.
  • the data synchronization service module is connected to the data entry system and the database.
  • the main function is to process airline data submission requests. After the submitted data is analyzed and processed, it is stored in the global data area and incremental data area of the first level cache.
  • the first level cache is a key-value type memory storage system, and the present invention adopts the Redis cluster as the first level cache.
  • Redis is an open source log-type, key-value (Key-Value) storage system written in ANSI C language, supporting the network, memory-based or persistent.
  • the secondary cache update module is mainly used to distribute air ticket price data to each secondary cache cluster node.
  • the data access interface module the main manifestation is a C++ runtime library, which provides an interface for accessing freight data for the freight calculation system and the freight search system.
  • the second level cache is a linear memory table.
  • the present invention adopts the shared memory Shm under Linux.
  • Shm is a shared memory at the Linux kernel level and is a communication method between processes under Linux.
  • a flow chart of a method for caching air freight data disclosed in an embodiment of the present invention includes the steps:
  • Step S101 Convert the air freight data into a data format of the first level cache, and store it in the first level cache;
  • air freight data is issued by airlines and consists of the ticket price and a series of rules that need to be met to use the ticket price.
  • the rule table required to use the air ticket price has the unique identification of the rule ID.
  • the rule detail data is associated with the rule table through the rule ID.
  • the main business of the freight calculation system and the freight search system is to query the air ticket prices and services from the flight origin city (airport) to the flight destination city (airport). Therefore, the air ticket prices and rules are mutually dependent. , Inquiring air ticket prices must be checked by rules, and air ticket prices and rules are integrated. In order to realize the data fragmentation algorithm, the ticket price, rule table and rule detail data must be integrated, otherwise the data will be fragmented during the calculation and the ticket price query.
  • the present invention adopts the method of merging related data, which specifically converts the air ticket price, rule table and rule detail data into a binary data and stores it in the first-level cache. After this processing, when inquiring air ticket prices, you can also query the rule table and rule details, which not only provides data conditions for the data fragmentation algorithm, but also greatly reduces the frequency of data access, thereby improving system performance .
  • Step S102 Perform data fragmentation on the air freight data stored in the first-level cache based on the flight origin city and the flight destination city to generate fragmented data;
  • each data node of the fragmented data contains a copy of the full set of air freight data for which the fragmentation algorithm is not available and a data subset of the air freight data for which the fragmentation algorithm is available.
  • the set constitutes a complete data set of air freight data with available sharding algorithms.
  • the flight origin city and flight destination city can be referred to as OD for short, where O refers to the flight origin city (Origin City, referred to as O), and D refers to the flight destination city (Destination City, referred to as D).
  • Data fragmentation refers to dividing a data set into independent and orthogonal data subsets according to certain rules, and then distributing the data subsets to different data nodes.
  • the freight calculation service is a service that requests high-frequency memory queries (up to 10,000 times). According to the above three methods, the response of the user request may not meet the standard, which is the reason why the existing technical solution does not perform data fragmentation.
  • the inventor of this application has discovered through research that the essence of the freight calculation system and the freight search system is to calculate and retrieve the air ticket price according to the OD. During business processing, it is split into multiple ODs for query and Therefore, this application adopts the OD-based data fragmentation method, which is a relatively optimal solution.
  • FIG. 3 a schematic diagram of data slicing disclosed in an embodiment of the present invention.
  • data set A is a complete set of air freight data for which a sharding algorithm is available
  • data set B is a data set for which a sharding algorithm is not available.
  • A1, A2,..., An represent mutually independent and orthogonal data subsets of data set A.
  • After slicing, A1, A2,..., An will be distributed to data nodes 1 node 2,... node n, data set B has data copies in all data nodes.
  • Step S103 Store the fragmented data in the second-level cache after being effective.
  • the data caching method disclosed in the present invention first converts the air freight data into the data format of the first level cache and stores it in the first level cache, and then compares the air freight data stored in the first level cache based on the flight schedule.
  • Data fragmentation is performed on the departure city and the flight destination city to generate fragmented data.
  • Each data node of the fragmented data contains a copy of the full set of air freight data for which the fragmentation algorithm is not available, and the available fragmentation algorithm.
  • Data subsets of air freight data Each data subset constitutes a complete data set of air freight data that can be sharded, and the sharded data is stored in the secondary cache after it takes effect.
  • each data node of the sharded data cached to the second-level cache contains a copy of the full set of air freight data for which the sharding algorithm is not available, and a data subset of the air freight data for which the sharding algorithm is available, That is to say, each data node contains only part of the air freight data of the available fragmentation algorithm. Therefore, compared to the full set of data cached in the prior art, the present invention improves the horizontal expansion capability of the cache system.
  • the air freight data when the air freight data is converted into the data format of the first-level cache, that is, when the air freight data is preprocessed, there are two situations.
  • the first is full data processing, and the second is the system. Run-time data processing, the two cases are described below.
  • the full data preprocessing process is: the data synchronization service module (see Figure 1) starts, if the value of the configuration item bLoadAll in the data synchronization service module is 1, the data synchronization service module reads the configuration file and sends it to the database in the order of the configuration file If the air freight data does not need to be pre-processed, the Key-Value generation engine is directly called to generate the air freight data into the data format of the first level cache, and then the data storage interface is called , Stored in the first-level cache; if the air freight data needs to be pre-processed, the air freight data is generated into a data object, and the generated data object is serialized, and the key-value generation engine is called to serialize the The data object generates the data format of the first level cache, and then calls the data storage interface to store it in the first level cache.
  • the data processing process is as follows: the data synchronization service module (see Figure 1) receives a request for data entry into the system to save data; if the air freight data does not require preprocessing, it directly calls the Key-Value generation engine to convert the air freight The data is generated into the data format of the first level cache, and then the data storage interface is called and stored in the first level cache; if the air freight data needs to be preprocessed, the air freight data is generated into a data object, and the generated data object Serialization, by calling the Key-Value generation engine, the serialized data object is generated into the data format of the first level cache, and then the data storage interface is called and stored in the first level cache.
  • the generated and cached data formats are specifically stored in the global data area and incremental data area of the first level cache.
  • an embodiment of the present invention discloses a method of segmenting the air freight data stored in the first-level cache based on the flight origin city and the flight destination city to generate fragmented data
  • a flowchart of the method includes the steps:
  • Step S201 Classify the air freight data of the available fragmentation algorithm stored in the first-level cache into m different data subsets according to the flight origin city and the flight destination city;
  • the flight origin and destination cities of the air freight data included in each data subset are the same, and m is a positive integer.
  • Step S202 Sort each of the data subsets according to the number of data items contained in descending order
  • the air freight data of the available fragmentation algorithm is stored in the form of a table in the first-level cache. After classifying the flight origin city and the flight destination city into m different data subsets, each data subset The air freight data included in the set is also stored in the form of a table based on the flight origin city and the flight destination city. Therefore, the number of data items included in each data subset is the same flight departure city and flight destination city included in the data subset. The number of air freight data in the destination city of the flight.
  • Step S203 Add the acquired node configuration information to the first preset list
  • the node configuration information includes: node name and node address, and the data structure in the first preset list is ⁇ node name, node address, number of data items, data list ⁇ .
  • Step S204 Put the air freight data in each of the data subsets into the data list with the least number of data in the first preset list in the order of storage, and put the data in the data subsets in all the data subsets.
  • the air freight data in the data list is deleted until the avionics freight data included in the data subset is cleared;
  • the number of data items corresponding to the data list is also increased by one.
  • Step S205 Use the data in each data structure in the first preset list as metadata, and merge each metadata with a copy of the full set of air freight data for which the sharding algorithm is not available, to generate a split Each data node of the slice data.
  • each data node of the fragmented data cached in the second-level cache of the present invention contains a full set of copies of the air freight data for which the fragmentation algorithm is not available, and the air tariff for the available fragmentation algorithm
  • the data subset of the data that is, each data node contains only part of the air freight data of the available sharding algorithm. Therefore, compared to the full set of data cached in the prior art, the present invention improves the cache The horizontal scalability of the system.
  • the present invention combines the data of the first-level cache and the second-level cache on the basis of the above embodiments.
  • the integration allows the storage of effective ticket price data based on the second-level cache, and the first-level cache saves all the ticket price data and incremental ticket price data. After a certain algorithm fusion, the stock data and incremental data form the full ticket price Data, thereby solving the real-time problem of air freight data after the release.
  • the fusion technology of the first-level cache and the second-level cache of the present invention solves the real-time problem after the air freight data is released.
  • the specific implementation method is to save the incremental operation in the first-level cache, and save the incremental operation in the first-level cache
  • Table 1 The data structure is shown in Table 1:
  • the integration of the first-level cache and the second-level cache is implemented in the data access interface module.
  • an embodiment of the present invention discloses a flow chart of a method for merging a first-level cache and a second-level cache. The method includes the steps:
  • Step S301 Place the data of the accessed air freight data in the secondary cache in a second preset list
  • Step S302 Determine whether the accessed air freight data has incremental data in the first-level cache, and if so, perform step S303;
  • Step S303 Read the incremental data from the first level cache.
  • Step S304 Use a preset fusion algorithm for the incremental data to obtain a fusion result of the first level cache and the second level cache.
  • the preset fusion algorithm is: discard the data in the incremental data whose operation time is earlier than the first time, where the first time is: place the data of the accessed air freight data in the secondary cache in the second preset The time in the list; if the operation of the incremental data is delete, delete the corresponding data in the second preset list; if the operation of the incremental data is modify, modify the corresponding data in the second preset list If the operation of the incremental data is to insert data, then insert the corresponding data in the second preset list.
  • the present invention also discloses a caching system for air freight data.
  • FIG. 6 a schematic structural diagram of a caching system for air freight data disclosed in an embodiment of the present invention, the system includes:
  • the conversion unit 401 is configured to convert air freight data into a data format of the first-level cache and store it in the first-level cache;
  • air freight data is issued by airlines and consists of the ticket price and a series of rules that need to be met to use the ticket price.
  • the rule table required to use the air ticket price has the unique identification of the rule ID.
  • the rule detail data is associated with the rule table through the rule ID.
  • the main business of the freight calculation system and the freight search system is to query the air ticket prices and services from the flight origin city (airport) to the flight destination city (airport). Therefore, the air ticket prices and rules are mutually dependent. , Inquiring air ticket prices must be checked by rules, and air ticket prices and rules are integrated. In order to realize the data fragmentation algorithm, the ticket price, rule table and rule detail data must be integrated, otherwise the data will be fragmented during the calculation and the ticket price query.
  • the present invention adopts the method of merging related data, which specifically converts the air ticket price, rule table and rule detail data into a binary data and stores it in the first-level cache. After this processing, when inquiring air ticket prices, you can also query the rule table and rule details, which not only provides data conditions for the data fragmentation algorithm, but also greatly reduces the frequency of data access, thereby improving system performance .
  • the generating unit 402 is configured to fragment the air freight data stored in the first-level cache based on the flight origin city and the flight destination city to generate fragmented data;
  • each data node of the fragmented data contains a copy of the full set of air freight data for which the fragmentation algorithm is not available and a data subset of the air freight data for which the fragmentation algorithm is available.
  • the set constitutes a complete data set of air freight data with available sharding algorithms.
  • the storage unit 403 is configured to store the fragmented data in the second-level cache after being effective.
  • the data caching system disclosed in the present invention first converts the air freight data into the data format of the first level cache, and stores it in the first level cache, and then compares the air freight data stored in the first level cache based on the flight schedule.
  • Data fragmentation is performed on the departure city and the flight destination city to generate fragmented data.
  • Each data node of the fragmented data contains a copy of the full set of air freight data for which the fragmentation algorithm is not available, and the available fragmentation algorithm.
  • Data subsets of air freight data Each data subset constitutes a complete data set of air freight data that can be sharded, and the sharded data is stored in the secondary cache after it takes effect.
  • each data node of the sharded data cached to the second-level cache contains a copy of the full set of air freight data for which the sharding algorithm is not available, and a data subset of the air freight data for which the sharding algorithm is available, That is to say, each data node contains only part of the air freight data of the available fragmentation algorithm. Therefore, compared to the full set of data cached in the prior art, the present invention improves the horizontal expansion capability of the cache system.
  • the air freight data when the air freight data is converted into the data format of the first-level cache, that is, when the air freight data is preprocessed, there are two situations.
  • the first is full data processing, and the second is the system. Run-time data processing, the two cases are described below.
  • the full data preprocessing process is: the data synchronization service module (see Figure 1) starts, if the value of the configuration item bLoadAll in the data synchronization service module is 1, the data synchronization service module reads the configuration file and sends it to the database in the order of the configuration file If the air freight data does not need to be pre-processed, the Key-Value generation engine is directly called to generate the air freight data into the data format of the first level cache, and then the data storage interface is called , Stored in the first-level cache; if the air freight data needs to be pre-processed, the air freight data is generated into a data object, and the generated data object is serialized, and the key-value generation engine is called to serialize the data object. The data object generates the data format of the first level cache, and then calls the data storage interface to store it in the first level cache.
  • the data processing process is as follows: the data synchronization service module (see Figure 1) receives a request for data entry into the system to save data; if the air freight data does not require preprocessing, it directly calls the Key-Value generation engine to convert the air freight The data is generated into the data format of the first level cache, and then the data storage interface is called and stored in the first level cache; if the air freight data needs to be preprocessed, the air freight data is generated into a data object, and the generated data object Serialization, by calling the Key-Value generation engine, the serialized data object is generated into the data format of the first level cache, and then the data storage interface is called and stored in the first level cache.
  • the generated and cached data formats are specifically stored in the global data area and incremental data area of the first level cache.
  • the generating unit 402 in the foregoing embodiment specifically includes:
  • the classification subunit is used to classify the air freight data of the available fragmentation algorithm stored in the first level cache into m different data subsets according to the flight origin city and the flight destination city, where m is a positive integer;
  • the sorting subunit is used to sort each of the data subsets in descending order according to the number of data items contained;
  • the add subunit is used to add the obtained node configuration information to the first preset list, where the node configuration information includes: node name and node address, and the data structure in the first preset list is ⁇ node name, Node address, number of data, data list ⁇ ;
  • the data placement subunit is used to place the avionics freight rate data in each data subset into the data list with the least number of data in the first preset list in the order of storage, and place the data
  • the avionics freight data that has been put in the data list in the collection is deleted until the avionics freight data included in the data subset is cleared;
  • the number of data items corresponding to the data list is also increased by one.
  • a generating subunit configured to use the data in each data structure in the first preset list as metadata, and merge each metadata with a copy of the full set of air freight data for which fragmentation algorithms are not available To generate each data node of the fragmented data.
  • each data node of the fragmented data cached in the second-level cache of the present invention contains a full set of copies of the air freight data for which the fragmentation algorithm is not available, and the air tariff for the available fragmentation algorithm
  • the data subset of the data that is, each data node contains only part of the air freight data of the available sharding algorithm. Therefore, compared to the full set of data cached in the prior art, the present invention improves the cache The horizontal scalability of the system.
  • the present invention combines the data of the first-level cache and the second-level cache on the basis of the above embodiments.
  • the integration allows the storage of effective ticket price data based on the second-level cache, and the first-level cache saves all the ticket price data and incremental ticket price data. After a certain algorithm fusion, the stock data and incremental data form the full ticket price Data, thereby solving the real-time problem of air freight data after the release.
  • the data caching system may further include:
  • a judging unit configured to judge whether the accessed air freight data has incremental data in the first level cache
  • a reading unit configured to read the incremental data from the first-level cache when the judgment unit judges yes;
  • the fusion unit is configured to apply a preset fusion algorithm to the incremental data to obtain a fusion result of the first level cache and the second level cache.
  • the preset fusion algorithm is: discard the data in the incremental data whose operation time is earlier than the first time, where the first time is: place the data of the accessed air freight data in the secondary cache in the second preset The time in the list; if the operation of the incremental data is delete, delete the corresponding data in the second preset list; if the operation of the incremental data is modify, then modify the corresponding data in the second preset list If the operation of the incremental data is to insert data, insert the corresponding data in the second preset list.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Strategic Management (AREA)
  • Economics (AREA)
  • General Engineering & Computer Science (AREA)
  • Marketing (AREA)
  • General Business, Economics & Management (AREA)
  • Development Economics (AREA)
  • Accounting & Taxation (AREA)
  • Data Mining & Analysis (AREA)
  • Finance (AREA)
  • Tourism & Hospitality (AREA)
  • Human Resources & Organizations (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Computational Linguistics (AREA)
  • Quality & Reliability (AREA)
  • Operations Research (AREA)
  • Game Theory and Decision Science (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Primary Health Care (AREA)
  • Software Systems (AREA)
  • Educational Administration (AREA)
  • Computing Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

一种航空运价数据的缓存方法及***，所述方法包括：将航空运价数据转换成一级缓存的数据格式，并存入一级缓存中（S101）；对一级缓存中存储的航空运价数据，基于航班始发城市和航班目的城市进行数据分片，生成分片数据（S102）；将分片数据生效后存储至二级缓存中（S103）。由于缓存至二级缓存的分片数据的每个数据节点中，仅包含部分可用分片算法的航空运价数据，因此，相对于缓存的数据副本都是全集而言，提升了缓存***的水平扩展能力。

Description

一种航空运价数据的缓存方法及*** 技术领域
本申请要求于2019年07月10日提交中国专利局、申请号为201910620227.9、发明名称为“一种航空运价数据的缓存方法及***”的国内申请的优先权，其全部内容通过引用结合在本申请中。
背景技术
随着互联网技术的兴起和在线旅游市场的迅猛发展，网上预订和购买机票成为很多旅客出行方式的首选。然而，海量的机票搜索和预订请求，以及机票价格即时发布并生效的需求，对运价计算***和运价搜索***带来了新的挑战。
现有技术在进行数据缓存时，先将所有的生效数据从数据库导出，生成数据文件并压缩，然后利用数据分发网络，将压缩的数据文件分发到各个缓存服务的数据节点主机；当各个数据代理(一种后台程序)检测到数据文件达到本数据节点后，将压缩的数据文件进行解压，对解压后的数据进行完整性校验，并根据数据配置生成索引文件；数据代理将数据和索引文件装载到内存中使缓存生效。当用户请求机票价格数据时，机票计算网络服务会把用户请求转到数据代理，由数据代理从服务器的内存shm中搜索并计算出机票价格数据返回给用户。
由于现有技术缓存的数据副本都是全集，因此，缓存***没有水平扩展能力，随着缓存数据总量的增大，内存资源会成为瓶颈，导致缓存***的水平扩展能力较差。
发明内容
有鉴于此，本发明公开一种航空运价数据的缓存方法及***，以提升缓存***的水平扩展能力。
一种航空运价数据的缓存方法，包括：
将航空运价数据转换成一级缓存的数据格式，并存入所述一级缓存中；
对所述一级缓存中存储的所述航空运价数据，基于航班始发城市和航班目的城市进行数据分片，生成分片数据，其中，所述分片数据的每个数 据节点中，均包含不可用分片算法的航空运价数据的数据全集副本以及可用分片算法的航空运价数据的数据子集，各个所述数据子集组成可用分片算法的航空运价数据的数据全集；
将所述分片数据生效后存储至二级缓存中。
可选的，所述一级缓存为Redis集群。
可选的，所述二级缓存为Linux内核级共享内存。
可选的，所述对所述一级缓存中存储的所述航空运价数据，基于航班始发城市和航班目的城市进行数据分片，生成分片数据，具体包括：
将所述一级缓存中存储的可用分片算法的航空运价数据，按照航班始发城市和航班目的城市分类成m个不同的数据子集，m为正整数；
对各个所述数据子集按照包含的数据条数从大到小排序；
将获取的节点配置信息添加至第一预设列表，其中，所述节点配置信息包括：节点名和节点地址，所述第一预设列表中的数据结构为{节点名，节点地址，数据条数，数据列表}；
将每个所述数据子集中的航空运价数据按照存储顺序依次放入所述第一预设列表中数据条数最少的数据列表中，并将所述数据子集中已放入所述数据列表中的航空运价数据删除，直至所述数据子集中包含的航空运价数据清零；
将所述第一预设列表中每个数据结构中的数据作为元数据，将每个所述元数据与不可用分片算法的航空运价数据的数据全集副本进行合并，生成所述分片数据的各个数据节点。
可选的，还包括：
将被访问航空运价数据在所述二级缓存中的数据放置第二预设列表中；
判断所述被访问航空运价数据在所述一级缓存中是否有增量数据；
如果是，则从所述一级缓存中读取所述增量数据；
对所述增量数据采用预设融合算法，得到所述一级缓存和所述二级缓存的融合结果。
可选的，所述预设融合算法为：将所述增量数据中操作时间早于第一时间的数据丢弃，其中，所述第一时间为：将所述被访问航空运价数据在 所述二级缓存中的数据放置所述第二预设列表中的时间；如果所述增量数据的操作为删除，则删除所述第二预设列表中对应的数据；如果所述增量数据的操作为修改，则修改所述第二预设列表中对应的数据；如果所述增量数据的操作为***数据，则在所述第二预设列表中***对应的数据。
一种航空运价数据的缓存***，包括：
转换单元，用于将航空运价数据转换成一级缓存的数据格式，并存入所述一级缓存中；
生成单元，用于对所述一级缓存中存储的所述航空运价数据，基于航班始发城市和航班目的城市进行数据分片，生成分片数据，其中，所述分片数据的每个数据节点中，均包含不可用分片算法的航空运价数据的数据全集副本以及可用分片算法的航空运价数据的数据子集，各个所述数据子集组成可用分片算法的航空运价数据的数据全集；
存储单元，用于将所述分片数据生效后存储至二级缓存中。
可选的，所述生成单元，具体包括：
分类子单元，用于将所述一级缓存中存储的可用分片算法的航空运价数据，按照航班始发城市和航班目的城市分类成m个不同的数据子集，m为正整数；
排序子单元，用于对各个所述数据子集按照包含的数据条数从大到小排序；
添加子单元，用于将获取的节点配置信息添加至第一预设列表，其中，所述节点配置信息包括：节点名和节点地址，所述第一预设列表中的数据结构为{节点名，节点地址，数据条数，数据列表}；
数据放置子单元，用于将每个所述数据子集中的航空运价数据按照存储顺序依次放入所述第一预设列表中数据条数最少的数据列表中，并将所述数据子集中已放入所述数据列表中的航空运价数据删除，直至所述数据子集中包含的航空运价数据清零；
生成子单元，用于将所述第一预设列表中每个数据结构中的数据作为元数据，将每个所述元数据与不可用分片算法的航空运价数据的数据全集副本进行合并，生成所述分片数据的各个数据节点。
可选的，还包括：
放置单元，用于将被访问航空运价数据在所述二级缓存中的数据放置第二预设列表中；
判断单元，用于判断所述被访问航空运价数据在所述一级缓存中是否有增量数据；
读取单元，用于在所述判断单元判断为是的情况下，则从所述一级缓存中读取所述增量数据；
融合单元，用于对所述增量数据采用预设融合算法，得到所述一级缓存和所述二级缓存的融合结果。
从上述的技术方案可知，本发明公开了一种航空运价数据的缓存方法及***，首先将航空运价数据转换成一级缓存的数据格式，并存入一级缓存中，然后对一级缓存中存储的航空运价数据，基于航班始发城市和航班目的城市进行数据分片，生成分片数据，其中，分片数据的每个数据节点中，均包含不可用分片算法的航空运价数据的数据全集副本以及可用分片算法的航空运价数据的数据子集，各个数据子集组成可用分片算法的航空运价数据的数据全集，将分片数据生效后存储至二级缓存中。由于缓存至二级缓存的分片数据的每个数据节点中，包含的均是不可用分片算法的航空运价数据的数据全集副本以及可用分片算法的航空运价数据的数据子集，也就是说，每个数据节点中仅包含部分可用分片算法的航空运价数据，因此，相对于现有技术缓存的数据副本都是全集而言，本发明提升了缓存***的水平扩展能力。
附图说明
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据公开的附图获得其他的附图。
图1为本发明实施例公开的一种航空运价数据的缓存方法采用的分布 式二级缓存架构图；
图2为本发明实施例公开的一种航空运价数据的缓存方法流程图；
图3为本发明实施例公开的一种数据分片示意图；
图4为本发明实施例公开的一种对一级缓存中存储的航空运价数据，基于航班始发城市和航班目的城市进行数据分片，生成分片数据的方法流程图；
图5为本发明实施例公开的一种一级缓存和二级缓存的融合方法流程图；
图6为本发明实施例公开的一种航空运价数据的缓存***的结构示意图。
具体实施方式
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。
本发明实施例公开了一种航空运价数据的缓存方法及***，首先将航空运价数据转换成一级缓存的数据格式，并存入一级缓存中，然后对一级缓存中存储的航空运价数据，基于航班始发城市和航班目的城市进行数据分片，生成分片数据，其中，分片数据的每个数据节点中，均包含不可用分片算法的航空运价数据的数据全集副本以及可用分片算法的航空运价数据的数据子集，各个数据子集组成可用分片算法的航空运价数据的数据全集，将分片数据生效后存储至二级缓存中。由于缓存至二级缓存的分片数据的每个数据节点中，包含的均是不可用分片算法的航空运价数据的数据全集副本以及可用分片算法的航空运价数据的数据子集，也就是说，每个数据节点中仅包含部分可用分片算法的航空运价数据，因此，相对于现有技术缓存的数据副本都是全集而言，本发明提升了缓存***的水平扩展能力。
参见图1，本发明公开的一种航空运价数据的缓存方法采用的分布式二级缓存架构图，图1中的虚线框标识了分布式二级缓存架构的组成部分，分布式二级缓存架构主要包括：数据同步服务模块、一级缓存、二级缓存更新模块、二级缓存和数据访问接口模块，其中，分布式二级缓存架构中包括两个一级缓存和多个二级缓存。
数据同步服务模块，分别与数据录入***和数据库连接，主要作用是处理航空公司数据提交请求，对提交数据进行分析处理后，存入一级缓存的全局数据区和增量数据区。
一级缓存，是一种key-value型内存存储***，本发明采用Redis集群作为一级缓存。Redis是一个开源的使用ANSI C语言编写、支持网络、可基于内存亦可持久化的日志型、键值对(Key-Value)的高性能存储***。
二级缓存更新模块，主要作用是将机票价格数据分发到各个二级缓存集群节点上。
数据访问接口模块，主要表现形式是一个C++运行时库，为运价计算***和运价搜索***提供访问运价数据的接口。
二级缓存，是一块线性内存表，本发明采用linux下的共享内存Shm，Shm是Linux内核级共享内存，是linux下的进程之间的一种通信方法。
参见图2，本发明一实施例公开的一种航空运价数据的缓存方法流程图，该方法包括步骤：
步骤S101、将航空运价数据转换成一级缓存的数据格式，并存入所述一级缓存中；
从业务数据标准来看，航空运价数据是由航空公司发布，并由机票价格及使用该机票价格需要满足的一系列规则组成。
从数据格式来看，机票价格发布时会指定航班始发城市(机场)和航班目的城市(机场)。使用机票价格所需满足的规则表具有规则ID唯一标识。规则细节数据则通过规则ID与规则表关联。
从本质上看，运价计算***和运价搜索***的主要业务时查询从航班始发城市(机场)到航班目的城市(机场)的机票价格及服务，因此，机票价格和规则时相互依赖的，查询机票价格必然进行规则校验，机票价格 和规则是一体的。为实现数据分片算法，必须将机票价格、规则表和规则细节数据综合在一起，否则在计算和机票价格查询时会造成数据上的割裂。
基于上述论述可知，为实现数据分片算法，需要对机票价格、规则表和规则细节数据进行预处理。本发明采用关联数据合并方法，具体是将机票价格、规则表和规则细节数据转换成一个二进制数据存储到一级缓存中。经过这种处理，在查询机票价格时，也可以查询到规则表和规则细节数据，不仅为数据分片算法提供了数据条件，而且也极大的减少了数据的访问频率，从而提升了***性能。
步骤S102、对所述一级缓存中存储的所述航空运价数据，基于航班始发城市和航班目的城市进行数据分片，生成分片数据；
其中，所述分片数据的每个数据节点中，均包含不可用分片算法的航空运价数据的数据全集副本以及可用分片算法的航空运价数据的数据子集，各个所述数据子集组成可用分片算法的航空运价数据的数据全集。
在实际应用中，可以将航班始发城市和航班目的城市简称为OD，其中O指的是航班始发城市(Origin City，简称为O)，D指的是航班目的城市(Destination City，简称为D)。
数据分片，指的是按照一定的规则，将数据集划分成相互独立和正交的数据子集，然后将数据子集分布到不同的数据节点上。
业界对于数据分片的方式主要有以下三种：①、hash方式；②、一致性hash(consistent hash)和③、按照数据范围(range based)。从运价计算特性来看，这三种方式都不是最优的数据分片方式，运价计算业务是单请求高频查询内存的业务(最高达万次)。按上述三种方式，用户请求的响应可能不达标，这也是现有技术方案不做数据分片的原因。本申请的发明人经过研究发现，运价计算***和运价搜索***的本质是按照OD计算和检索机票价格，在业务处理时，低于复查行程也是将其拆分成多个OD来查询和计算，因此，本申请采用基于OD的数据分片方法，是相对最优的解决方案。
具体的，参见图3，本发明一实施例公开的数据分片示意图，图3中，数据集A为可用分片算法的航空运价数据的数据全集，数据集B为不可用分片算法的航空运价数据的数据全集，A1、A2、…..、An表示数据集A的相互独立、正交的数据子集，分片后，A1、A2、…..、An会分布到数据节点 1节点2、......节点n中，数据集B在所有的数据节点中都有数据副本。
步骤S103、将所述分片数据生效后存储至二级缓存中。
综上可知，本发明公开的数据缓存方法，首先将航空运价数据转换成一级缓存的数据格式，并存入一级缓存中，然后对一级缓存中存储的航空运价数据，基于航班始发城市和航班目的城市进行数据分片，生成分片数据，其中，分片数据的每个数据节点中，均包含不可用分片算法的航空运价数据的数据全集副本以及可用分片算法的航空运价数据的数据子集，各个数据子集组成可用分片算法的航空运价数据的数据全集，将分片数据生效后存储至二级缓存中。由于缓存至二级缓存的分片数据的每个数据节点中，包含的均是不可用分片算法的航空运价数据的数据全集副本以及可用分片算法的航空运价数据的数据子集，也就是说，每个数据节点中仅包含部分可用分片算法的航空运价数据，因此，相对于现有技术缓存的数据副本都是全集而言，本发明提升了缓存***的水平扩展能力。
上述实施例中，在将航空运价数据转换成一级缓存的数据格式，也即在对航空运价数据进行预处理时，分两种情况，第一种是全量数据处理，第二中是***运行时数据处理，下面分别针对这两种情况进行说明。
全量数据处理：在***开始部署时，由于一级缓存的数据结构与数据库是不同的，因此为让***正常运行，需要将数据库中存储的有效的航空运价数据全部按照一级缓存的数据结构存入。
全量数据预处理流程为：数据同步服务模块(见图1)启动，如果数据同步服务模块中的配置项bLoadAll的值为1时，数据同步服务模块读取配置文件，按照配置文件的顺序到数据库中读取相应的航空运价数据，如果所述航空运价数据不需要预处理，则直接调用Key-Value生成引擎，将航空运价数据生成一级缓存的数据格式，然后调用数据存入接口，存入一级缓存中；如果所述航空运价数据需要预处理，则将航空运价数据生成数据对象，并对生成的数据对象序列化，通过调用Key-Value生成引擎，将序列化的数据对象生成一级缓存的数据格式，然后调用数据存入接口，存入一级缓存中。
***运行时数据处理：为保证数据的完整性，将录入***的每一条航空运价数据，都保存到一级缓存中。
***运行时数据处理流程为：数据同步服务模块(见图1)接收到数据录入***保存数据的请求；如果航空运价数据不需要预处理，则直接调用Key-Value生成引擎，将航空运价数据生成一级缓存的数据格式，然后调用数据存入接口，存入一级缓存中；如果所述航空运价数据需要预处理，则将航空运价数据生成数据对象，并对生成的数据对象序列化，通过调用Key-Value生成引擎，将序列化的数据对象生成一级缓存的数据格式，然后调用数据存入接口，存入一级缓存中。
在实际应用中，具体将生成的以及缓存的数据格式存入一级缓存的全局数据区和增量数据区。
为进一步优化上述实施例，参见图4，本发明一实施例公开的一种对一级缓存中存储的航空运价数据，基于航班始发城市和航班目的城市进行数据分片，生成分片数据的方法流程图，该方法包括步骤：
步骤S201、将所述一级缓存中存储的可用分片算法的航空运价数据，按照航班始发城市和航班目的城市分类成m个不同的数据子集；
其中，每个数据子集中包含的航空运价数据的航班始发城市和航班目的城市相同，m为正整数。
步骤S202、对各个所述数据子集按照包含的数据条数从大到小排序；
需要说明的是，可用分片算法的航空运价数据在一级缓存中以表格的形式存储，在按照航班始发城市和航班目的城市分类成m个不同的数据子集后，每一个数据子集中包含的航空运价数据也是基于航班始发城市和航班目的城市以表格的形式存储，因此，每个数据子集中包含的数据条数也即该数据子集中包含的相同的航班始发城市和航班目的城市的航空运价数据的数量。
步骤S203、将获取的节点配置信息添加至第一预设列表；
其中，所述节点配置信息包括：节点名和节点地址，所述第一预设列表中的数据结构为{节点名，节点地址，数据条数，数据列表}。
步骤S204、将每个所述数据子集中的航空运价数据按照存储顺序依次 放入所述第一预设列表中数据条数最少的数据列表中，并将所述数据子集中已放入所述数据列表中的航空运价数据删除，直至所述数据子集中包含的航电运价数据清零；
其中，每次将航空运价数据放置数据条数最少的数据列表中后，该数据列表对应的数据条数也相应加一。
步骤S205、将所述第一预设列表中每个数据结构中的数据作为元数据，将每个所述元数据与不可用分片算法的航空运价数据的数据全集副本进行合并，生成分片数据的各个数据节点。
综上可知，由于本发明缓存至二级缓存的分片数据的每个数据节点中，包含的均是不可用分片算法的航空运价数据的数据全集副本以及可用分片算法的航空运价数据的数据子集，也就是说，每个数据节点中仅包含部分可用分片算法的航空运价数据，因此，相对于现有技术缓存的数据副本都是全集而言，本发明提升了缓存***的水平扩展能力。
本领域技术人员知晓，传统的机票价格缓存方法无法实现航空运价数据的实时生效，为解决这一技术问题，本发明在上述实施例的基础上，将一级缓存和二级缓存的数据进行了融合，使得基于二级缓存保存的是存量生效机票价格数据，一级缓存保存的是所有机票价格数据及增量机票价格数据，存量数据和增量数据经过一定算法融合之后形成全量的机票价格数据，从而解决了航空运价数据发布后的实时性问题。
本发明的一级缓存和二级缓存融合技术解决了航空运价数据发布后的实时性问题，具体实现方法是将增量操作保存至一级缓存中，保存至一级缓存中的增量操作的数据结构如下表1：
表1
Figure PCTCN2020100802-appb-000001
Figure PCTCN2020100802-appb-000002
其中，一级缓存和二级缓存的融合工作是在数据访问接口模块中实现的。
参见图5，本发明一实施例公开的一种一级缓存和二级缓存的融合方法流程图，该方法包括步骤：
步骤S301、将被访问航空运价数据在二级缓存中的数据放置第二预设列表中；
步骤S302、判断所述被访问航空运价数据在一级缓存中是否有增量数据，如果是，则执行步骤S303；
步骤S303、从所述一级缓存中读取所述增量数据；
步骤S304、对所述增量数据采用预设融合算法，得到所述一级缓存和所述二级缓存的融合结果。
其中，预设融合算法为：将增量数据中操作时间早于第一时间的数据 丢弃，其中，第一时间为：将被访问航空运价数据在二级缓存中的数据放置第二预设列表中的时间；如果增量数据的操作为删除，则删除所述第二预设列表中对应的数据；如果所述增量数据的操作为修改，则修改所述第二预设列表中对应的数据；如果所述增量数据的操作为***数据，则在所述第二预设列表中***对应的数据。
需要说明的是，针对增量数据的删除，在数据同步服务模块中有一个计时器，通过该计时器定时去删除所有过时的增量数据。
与上述方法实施例相对应，本发明还公开了一种航空运价数据的缓存***。
参见图6，本发明一实施例公开的一种航空运价数据的缓存***的结构示意图，该***包括：
转换单元401，用于将航空运价数据转换成一级缓存的数据格式，并存入所述一级缓存中；
从业务数据标准来看，航空运价数据是由航空公司发布，并由机票价格及使用该机票价格需要满足的一系列规则组成。
从数据格式来看，机票价格发布时会指定航班始发城市(机场)和航班目的城市(机场)。使用机票价格所需满足的规则表具有规则ID唯一标识。规则细节数据则通过规则ID与规则表关联。
从本质上看，运价计算***和运价搜索***的主要业务时查询从航班始发城市(机场)到航班目的城市(机场)的机票价格及服务，因此，机票价格和规则时相互依赖的，查询机票价格必然进行规则校验，机票价格和规则是一体的。为实现数据分片算法，必须将机票价格、规则表和规则细节数据综合在一起，否则在计算和机票价格查询时会造成数据上的割裂。
基于上述论述可知，为实现数据分片算法，需要对机票价格、规则表和规则细节数据进行预处理。本发明采用关联数据合并方法，具体是将机票价格、规则表和规则细节数据转换成一个二进制数据存储到一级缓存中。经过这种处理，在查询机票价格时，也可以查询到规则表和规则细节数据，不仅为数据分片算法提供了数据条件，而且也极大的减少了数据的访问频率，从而提升了***性能。
生成单元402，用于对所述一级缓存中存储的所述航空运价数据，基于航班始发城市和航班目的城市进行数据分片，生成分片数据；
其中，所述分片数据的每个数据节点中，均包含不可用分片算法的航空运价数据的数据全集副本以及可用分片算法的航空运价数据的数据子集，各个所述数据子集组成可用分片算法的航空运价数据的数据全集。
存储单元403，用于将所述分片数据生效后存储至二级缓存中。
综上可知，本发明公开的数据缓存***，首先将航空运价数据转换成一级缓存的数据格式，并存入一级缓存中，然后对一级缓存中存储的航空运价数据，基于航班始发城市和航班目的城市进行数据分片，生成分片数据，其中，分片数据的每个数据节点中，均包含不可用分片算法的航空运价数据的数据全集副本以及可用分片算法的航空运价数据的数据子集，各个数据子集组成可用分片算法的航空运价数据的数据全集，将分片数据生效后存储至二级缓存中。由于缓存至二级缓存的分片数据的每个数据节点中，包含的均是不可用分片算法的航空运价数据的数据全集副本以及可用分片算法的航空运价数据的数据子集，也就是说，每个数据节点中仅包含部分可用分片算法的航空运价数据，因此，相对于现有技术缓存的数据副本都是全集而言，本发明提升了缓存***的水平扩展能力。
上述实施例中，在将航空运价数据转换成一级缓存的数据格式，也即在对航空运价数据进行预处理时，分两种情况，第一种是全量数据处理，第二中是***运行时数据处理，下面分别针对这两种情况进行说明。
全量数据处理：在***开始部署时，由于一级缓存的数据结构与数据库是不同的，因此为让***正常运行，需要将数据库中存储的有效的航空运价数据全部按照一级缓存的数据结构存入。
全量数据预处理流程为：数据同步服务模块(见图1)启动，如果数据同步服务模块中的配置项bLoadAll的值为1时，数据同步服务模块读取配置文件，按照配置文件的顺序到数据库中读取相应的航空运价数据，如果所述航空运价数据不需要预处理，则直接调用Key-Value生成引擎，将航空运价数据生成一级缓存的数据格式，然后调用数据存入接口，存入一级缓存中；如果所述航空运价数据需要预处理，则将航空运价数据生成数据对象， 并对生成的数据对象序列化，通过调用Key-Value生成引擎，将序列化的数据对象生成一级缓存的数据格式，然后调用数据存入接口，存入一级缓存中。
***运行时数据处理：为保证数据的完整性，将录入***的每一条航空运价数据，都保存到一级缓存中。
***运行时数据处理流程为：数据同步服务模块(见图1)接收到数据录入***保存数据的请求；如果航空运价数据不需要预处理，则直接调用Key-Value生成引擎，将航空运价数据生成一级缓存的数据格式，然后调用数据存入接口，存入一级缓存中；如果所述航空运价数据需要预处理，则将航空运价数据生成数据对象，并对生成的数据对象序列化，通过调用Key-Value生成引擎，将序列化的数据对象生成一级缓存的数据格式，然后调用数据存入接口，存入一级缓存中。
在实际应用中，具体将生成的以及缓存的数据格式存入一级缓存的全局数据区和增量数据区。
为进一步优化上述实施例，上述实施例中的生成单元402具体包括：
分类子单元，用于将所述一级缓存中存储的可用分片算法的航空运价数据，按照航班始发城市和航班目的城市分类成m个不同的数据子集，m为正整数；
排序子单元，用于对各个所述数据子集按照包含的数据条数从大到小排序；
添加子单元，用于将获取的节点配置信息添加至第一预设列表，其中，所述节点配置信息包括：节点名和节点地址，所述第一预设列表中的数据结构为{节点名，节点地址，数据条数，数据列表}；
数据放置子单元，用于将每个所述数据子集中的航电运价数据按照存储顺序依次放入所述第一预设列表中数据条数最少的数据列表中，并将所述数据子集中已放入所述数据列表中的航电运价数据删除，直至所述数据子集中包含的航电运价数据清零；
其中，每次将航电运价数据放置数据条数最少的数据列表中后，该数 据列表对应的数据条数也相应加一。
生成子单元，用于将所述第一预设列表中每个数据结构中的数据作为元数据，将每个所述元数据与不可用分片算法的航空运价数据的数据全集副本进行合并，生成所述分片数据的各个数据节点。
综上可知，由于本发明缓存至二级缓存的分片数据的每个数据节点中，包含的均是不可用分片算法的航空运价数据的数据全集副本以及可用分片算法的航空运价数据的数据子集，也就是说，每个数据节点中仅包含部分可用分片算法的航空运价数据，因此，相对于现有技术缓存的数据副本都是全集而言，本发明提升了缓存***的水平扩展能力。
本领域技术人员知晓，传统的机票价格缓存方法无法实现航空运价数据的实时生效，为解决这一技术问题，本发明在上述实施例的基础上，将一级缓存和二级缓存的数据进行了融合，使得基于二级缓存保存的是存量生效机票价格数据，一级缓存保存的是所有机票价格数据及增量机票价格数据，存量数据和增量数据经过一定算法融合之后形成全量的机票价格数据，从而解决了航空运价数据发布后的实时性问题。
因此，为进一步优化上述实施例，数据缓存***还可以包括：
放置单元，用于将被访问航空运价数据在所述二级缓存中的数据放置第二预设列表中；
判断单元，用于判断所述被访问航空运价数据在所述一级缓存中是否有增量数据；
读取单元，用于在所述判断单元判断为是的情况下，则从所述一级缓存中读取所述增量数据；
融合单元，用于对所述增量数据采用预设融合算法，得到所述一级缓存和所述二级缓存的融合结果。
其中，预设融合算法为：将增量数据中操作时间早于第一时间的数据丢弃，其中，第一时间为：将被访问航空运价数据在二级缓存中的数据放置第二预设列表中的时间；如果增量数据的操作为删除，则删除所述第二预设列表中对应的数据；如果所述增量数据的操作为修改，则修改所述第二预设列表中对应的数据；如果所述增量数据的操作为***数据，则在所 述第二预设列表中***对应的数据。
针对增量数据的删除，在数据同步服务模块中有一个计时器，通过该计时器定时去删除所有过时的增量数据。
需要说明的是，***实施例中各组成部分的具体工作原理，请参见方法实施例对应部分，此处不再赘述。
最后，还需要说明的是，在本文中，诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。
本说明书中各个实施例采用递进的方式描述，每个实施例重点说明的都是与其他实施例的不同之处，各个实施例之间相同相似部分互相参见即可。
对所公开的实施例的上述说明，使本领域专业技术人员能够实现或使用本发明。对这些实施例的多种修改对本领域的专业技术人员来说将是显而易见的，本文中所定义的一般原理可以在不脱离本发明的精神或范围的情况下，在其它实施例中实现。因此，本发明将不会被限制于本文所示的这些实施例，而是要符合与本文所公开的原理和新颖特点相一致的最宽的范围。

Claims (9)

  1. 一种航空运价数据的缓存方法，其特征在于，包括：
    将航空运价数据转换成一级缓存的数据格式，并存入所述一级缓存中；
    对所述一级缓存中存储的所述航空运价数据，基于航班始发城市和航班目的城市进行数据分片，生成分片数据，其中，所述分片数据的每个数据节点中，均包含不可用分片算法的航空运价数据的数据全集副本以及可用分片算法的航空运价数据的数据子集，各个所述数据子集组成可用分片算法的航空运价数据的数据全集；
    将所述分片数据生效后存储至二级缓存中。
  2. 根据权利要求1所述的缓存方法，其特征在于，所述一级缓存为Redis集群。
  3. 根据权利要求1所述的缓存方法，其特征在于，所述二级缓存为Linux内核级共享内存。
  4. 根据权利要求1所述的缓存方法，其特征在于，所述对所述一级缓存中存储的所述航空运价数据，基于航班始发城市和航班目的城市进行数据分片，生成分片数据，具体包括：
    将所述一级缓存中存储的可用分片算法的航空运价数据，按照航班始发城市和航班目的城市分类成m个不同的数据子集，m为正整数；
    对各个所述数据子集按照包含的数据条数从大到小排序；
    将获取的节点配置信息添加至第一预设列表，其中，所述节点配置信息包括：节点名和节点地址，所述第一预设列表中的数据结构为{节点名，节点地址，数据条数，数据列表}；
    将每个所述数据子集中的航空运价数据按照存储顺序依次放入所述第一预设列表中数据条数最少的数据列表中，并将所述数据子集中已放入所述数据列表中的航空运价数据删除，直至所述数据子集中包含的航空运价数据清零；
    将所述第一预设列表中每个数据结构中的数据作为元数据，将每个所述元数据与不可用分片算法的航空运价数据的数据全集副本进行合并，生成所述分片数据的各个数据节点。
  5. 根据权利要求1所述的缓存方法，其特征在于，还包括：
    将被访问航空运价数据在所述二级缓存中的数据放置第二预设列表中；
    判断所述被访问航空运价数据在所述一级缓存中是否有增量数据；
    如果是，则从所述一级缓存中读取所述增量数据；
    对所述增量数据采用预设融合算法，得到所述一级缓存和所述二级缓存的融合结果。
  6. 根据权利要求5所述的缓存方法，其特征在于，所述预设融合算法为：将所述增量数据中操作时间早于第一时间的数据丢弃，其中，所述第一时间为：将所述被访问航空运价数据在所述二级缓存中的数据放置所述第二预设列表中的时间；如果所述增量数据的操作为删除，则删除所述第二预设列表中对应的数据；如果所述增量数据的操作为修改，则修改所述第二预设列表中对应的数据；如果所述增量数据的操作为***数据，则在所述第二预设列表中***对应的数据。
  7. 一种航空运价数据的缓存***，其特征在于，包括：
    转换单元，用于将航空运价数据转换成一级缓存的数据格式，并存入所述一级缓存中；
    生成单元，用于对所述一级缓存中存储的所述航空运价数据，基于航班始发城市和航班目的城市进行数据分片，生成分片数据，其中，所述分片数据的每个数据节点中，均包含不可用分片算法的航空运价数据的数据全集副本以及可用分片算法的航空运价数据的数据子集，各个所述数据子集组成可用分片算法的航空运价数据的数据全集；
    存储单元，用于将所述分片数据生效后存储至二级缓存中。
  8. 根据权利要求7所述的缓存***，其特征在于，所述生成单元，具体包括：
    分类子单元，用于将所述一级缓存中存储的可用分片算法的航空运价数据，按照航班始发城市和航班目的城市分类成m个不同的数据子集，m为正整数；
    排序子单元，用于对各个所述数据子集按照包含的数据条数从大到小排序；
    添加子单元，用于将获取的节点配置信息添加至第一预设列表，其中，所述节点配置信息包括：节点名和节点地址，所述第一预设列表中的数据结构为{节点名，节点地址，数据条数，数据列表}；
    数据放置子单元，用于将每个所述数据子集中的航空运价数据按照存储顺序依次放入所述第一预设列表中数据条数最少的数据列表中，并将所述数据子集中已放入所述数据列表中的航空运价数据删除，直至所述数据子集中包含的航电运价数据清零；
    生成子单元，用于将所述第一预设列表中每个数据结构中的数据作为元数据，将每个所述元数据与不可用分片算法的航空运价数据的数据全集副本进行合并，生成所述分片数据的各个数据节点。
  9. 根据权利要求7所述的缓存***，其特征在于，还包括：
    放置单元，用于将被访问航空运价数据在所述二级缓存中的数据放置第二预设列表中；
    判断单元，用于判断所述被访问航空运价数据在所述一级缓存中是否有增量数据；
    读取单元，用于在所述判断单元判断为是的情况下，则从所述一级缓存中读取所述增量数据；
    融合单元，用于对所述增量数据采用预设融合算法，得到所述一级缓存和所述二级缓存的融合结果。
PCT/CN2020/100802 2019-07-10 2020-07-08 一种航空运价数据的缓存方法及*** WO2021004472A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP20836693.0A EP3998535A4 (en) 2019-07-10 2020-07-08 METHOD AND SYSTEM FOR CACHEING AIR FREIGHT RATE DATA
US17/625,702 US11983109B2 (en) 2019-07-10 2020-07-08 Air freight rate data caching method and system

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910620227.9A CN110309184B (zh) 2019-07-10 2019-07-10 一种航空运价数据的缓存方法及***
CN201910620227.9 2019-07-10

Publications (1)

Publication Number Publication Date
WO2021004472A1 true WO2021004472A1 (zh) 2021-01-14

Family

ID=68079786

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/100802 WO2021004472A1 (zh) 2019-07-10 2020-07-08 一种航空运价数据的缓存方法及***

Country Status (4)

Country Link
US (1) US11983109B2 (zh)
EP (1) EP3998535A4 (zh)
CN (1) CN110309184B (zh)
WO (1) WO2021004472A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110309184B (zh) 2019-07-10 2021-05-25 中国民航信息网络股份有限公司 一种航空运价数据的缓存方法及***
CN110737665B (zh) * 2019-10-21 2023-07-18 中国民航信息网络股份有限公司 一种数据处理方法及装置

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103207919A (zh) * 2013-04-26 2013-07-17 北京亿赞普网络技术有限公司 一种MongoDB集群快速查询计算的方法及装置
CN106030626A (zh) * 2013-12-11 2016-10-12 天巡有限公司 用于提供诸如飞机票价可用性的票价可用性的方法和***
CN108804465A (zh) * 2017-05-04 2018-11-13 中兴通讯股份有限公司 一种分布式缓存数据库数据迁移的方法及***
CN108920099A (zh) * 2018-06-22 2018-11-30 中国人民解放军战略支援部队信息工程大学 基于多种分片方式的数据动态存储***及方法
US20190197174A1 (en) * 2017-12-22 2019-06-27 Warevalley Co., Ltd. Method and system for replicating data to heterogeneous database and detecting synchronization error of heterogeneous database through sql packet analysis
CN110309184A (zh) * 2019-07-10 2019-10-08 中国民航信息网络股份有限公司 一种航空运价数据的缓存方法及***

Family Cites Families (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7509393B2 (en) * 2001-12-19 2009-03-24 International Business Machines Corporation Method and system for caching role-specific fragments
CN100505762C (zh) * 2006-04-19 2009-06-24 华中科技大学 适用于对象网络存储的分布式多级缓存***
US7975109B2 (en) * 2007-05-30 2011-07-05 Schooner Information Technology, Inc. System including a fine-grained memory and a less-fine-grained memory
CN100558089C (zh) * 2007-06-08 2009-11-04 北京工业大学 一种基于网络过滤器的内容过滤网关实现方法
GB2487992A (en) * 2011-01-31 2012-08-15 Sita Information Networking Computing Uk Ltd Inventory control system
CN102117248A (zh) * 2011-03-09 2011-07-06 浪潮(北京)电子信息产业有限公司 一种缓存***和在缓存***中缓存数据的方法
US9037801B2 (en) * 2012-09-27 2015-05-19 Amadeus S.A.S. Method and system of storing and retrieving data
CN203520386U (zh) * 2013-07-10 2014-04-02 北京旋极信息技术股份有限公司 一种缓存管理装置及航空飞行器
US20150088554A1 (en) * 2013-09-24 2015-03-26 Argus Health Systems, Inc. Methods, Systems, and Servers for Processing Health Insurance Claims
US20150154511A1 (en) * 2013-11-29 2015-06-04 Amadeus S.A.S. Availability watching
US10049052B2 (en) * 2014-10-27 2018-08-14 Nxp Usa, Inc. Device having a cache memory
US9990400B2 (en) * 2015-10-26 2018-06-05 Salesforce.Com, Inc. Builder program code for in-memory cache
CN105630984B (zh) * 2015-12-25 2020-08-21 中国民航信息网络股份有限公司 一种运价搜索***
CN105654338A (zh) * 2015-12-25 2016-06-08 中国民航信息网络股份有限公司 规则运价计算方法及装置、***
CN105630982A (zh) * 2015-12-25 2016-06-01 中国民航信息网络股份有限公司 航班数据缓存方法及***
CN106055640A (zh) * 2016-05-31 2016-10-26 乐视控股（北京）有限公司 缓存的管理方法及***
CN105975638A (zh) * 2016-06-23 2016-09-28 中国民航大学 一种基于NoSQL的海量航空物流小文件存储结构及其存取方法
CN108022156A (zh) * 2017-12-08 2018-05-11 上海理工大学 一种互动式标准零件自动报价***
CN109947668B (zh) * 2017-12-21 2021-06-29 北京京东尚科信息技术有限公司 存储数据的方法和装置

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103207919A (zh) * 2013-04-26 2013-07-17 北京亿赞普网络技术有限公司 一种MongoDB集群快速查询计算的方法及装置
CN106030626A (zh) * 2013-12-11 2016-10-12 天巡有限公司 用于提供诸如飞机票价可用性的票价可用性的方法和***
CN108804465A (zh) * 2017-05-04 2018-11-13 中兴通讯股份有限公司 一种分布式缓存数据库数据迁移的方法及***
US20190197174A1 (en) * 2017-12-22 2019-06-27 Warevalley Co., Ltd. Method and system for replicating data to heterogeneous database and detecting synchronization error of heterogeneous database through sql packet analysis
CN108920099A (zh) * 2018-06-22 2018-11-30 中国人民解放军战略支援部队信息工程大学 基于多种分片方式的数据动态存储***及方法
CN110309184A (zh) * 2019-07-10 2019-10-08 中国民航信息网络股份有限公司 一种航空运价数据的缓存方法及***

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3998535A4

Also Published As

Publication number Publication date
EP3998535A4 (en) 2023-03-01
EP3998535A1 (en) 2022-05-18
US11983109B2 (en) 2024-05-14
CN110309184B (zh) 2021-05-25
US20220374355A1 (en) 2022-11-24
CN110309184A (zh) 2019-10-08

Similar Documents

Publication Publication Date Title
US10958752B2 (en) Providing access to managed content
US6292880B1 (en) Alias-free content-indexed object cache
US6289358B1 (en) Delivering alternate versions of objects from an object cache
US6493721B1 (en) Techniques for performing incremental data updates
US6484161B1 (en) Method and system for performing online data queries in a distributed computer system
US6209003B1 (en) Garbage collection in an object cache
US6374241B1 (en) Data merging techniques
US6754799B2 (en) System and method for indexing and retrieving cached objects
WO2019133928A1 (en) Hierarchical, parallel models for extracting in real-time high-value information from data streams and system and method for creation of same
US20040205044A1 (en) Method for storing inverted index, method for on-line updating the same and inverted index mechanism
US20060041606A1 (en) Indexing system for a computer file store
CN104679898A (zh) 一种大数据访问方法
CN104778270A (zh) 一种用于多文件的存储方法
WO2021004472A1 (zh) 一种航空运价数据的缓存方法及***
CN103678694A (zh) 视频资源的倒排索引文件建立方法及其***
CN102819586A (zh) 一种基于高速缓存的url分类方法和设备
Cai et al. An optimization strategy of massive small files storage based on HDFS
US7752181B2 (en) System and method for performing a data uniqueness check in a sorted data set
US20200218705A1 (en) System and method of managing indexing for search index partitions
Ji Affinity-based management of main memory database clusters
Zhao et al. ICRA: index based cache replacement algorithm for cloud storage
US7823110B2 (en) Method and system for processing geometrical layout design data
Frieder et al. On scalable information retrieval systems
CN114168075B (zh) 基于数据关联性提升负载访问性能的方法、设备及***
US11709824B2 (en) Consolidating transaction log requests and transaction logs in a database transaction log service

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20836693

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2020836693

Country of ref document: EP

Effective date: 20220210