CN117827837A - Forward compatibility method of cache database and related equipment - Google Patents
Forward compatibility method of cache database and related equipment Download PDFInfo
- Publication number
- CN117827837A CN117827837A CN202311866154.4A CN202311866154A CN117827837A CN 117827837 A CN117827837 A CN 117827837A CN 202311866154 A CN202311866154 A CN 202311866154A CN 117827837 A CN117827837 A CN 117827837A
- Authority
- CN
- China
- Prior art keywords
- cache
- database
- serialization
- information class
- record
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000004364 calculation method Methods 0.000 claims abstract description 50
- 230000006870 function Effects 0.000 claims description 39
- 238000012795 verification Methods 0.000 claims description 9
- 230000003068 static effect Effects 0.000 claims description 6
- 238000010276 construction Methods 0.000 claims description 4
- 238000013507 mapping Methods 0.000 claims description 4
- 238000012545 processing Methods 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a forward compatible method of a cache database and related equipment, which utilizes a code generation script to generate a table cache record structure, a table cache information class and a table structure information class corresponding to each table, compiles and obtains a serialization program, a cache management program and a calculation engine, utilizes the serialization program to serialize the table record into serialized data, utilizes the cache management program to load the serialized data to construct the cache database, utilizes the calculation engine to access the cache database to acquire memory first addresses of all the table records, and enables the calculation engine to read the data based on the memory first addresses, the table cache record structure size and the offset. The invention records the size of the table cache record structure and the offset of the indication part in the table cache record structure in the table cache information class, so that when the table structure is newly added or a field is added, the serialization program can be forward compatible with the Oracle database, the cache management program can be forward compatible with the serialization data, and the calculation engine can be forward compatible with the cache database.
Description
Technical Field
The invention relates to the technical field of data storage, in particular to a method for forward compatibility of a cache database and related equipment.
Background
At present, in order to reduce the load of an Oracle database and improve the performance of a computing engine in a civil aviation freight rate system, a cache database is introduced in the civil aviation freight rate system. The data in the cache database is from an Oracle database, the serialization program exports the Oracle database data into different serialization data according to each table, the serialization data is loaded into the cache database by the cache management program, and the computing engine accesses the cache database to inquire the data so as to replace directly accessing the Oracle database. The table structure versions of the Oracle database and the serialization program must be consistent to export the serialization data, the table structure versions of the serialization data and the cache management program must be consistent to load the serialization data into the cache database, and the table structure versions of the calculation engine and the cache database must be consistent to correctly query the data.
Along with the increasing diversification of the demands of airlines, the table structure in the Oracle database is changed frequently, and most of the table structures are new tables, new fields and the like based on old versions. When the cache database is online, the Oracle database, the serialization program, the cache management program and the computing engine need to be updated in sequence, the cache database has large data volume, the computing nodes of the production cluster are numerous, version updating takes longer time, and therefore the situation that the computing nodes of the updated cache management program and the computing engine use new version data and the computing nodes which are not updated still use outdated old version data easily occurs, so that the data used by different computing nodes in the production cluster are inconsistent, and different results can be obtained by a user sending the same request.
Disclosure of Invention
In view of this, the present invention discloses a method and related device for forward compatibility of a cache database, so as to implement forward compatibility of a serialization program with an Oracle database, forward compatibility of a cache management program with serialized data, and forward compatibility of a computing engine with the cache database, without modifying codes or configuration when a table structure is added with a new table or an additional field.
A method of forward compatibility of a cache database, comprising:
generating a table cache record structure, a table cache information class and a table structure information class corresponding to each table in an Oracle database by using a code generation script, wherein the table cache information class comprises: the size of the table cache record structure and the offset of the indication part in the table cache record structure;
compiling a serialization program and a cache management program based on the table cache record structure, the table cache information class, the table structure information class and a cache database version number, and compiling a calculation engine based on the table cache record structure and the cache database version number;
serializing the table records of the Oracle database into serialized data using the serialization program, wherein the serialized data comprises: a serialization file and a version file, wherein the size of the table cache record structure and the offset are stored in the serialization file;
Loading the serialized data by using the cache management program to construct a cache database;
and accessing the cache database by using the calculation engine to obtain the memory head addresses of all table records in the cache database, so that the calculation engine reads data based on the memory head addresses, the table cache record structure size and the offset.
A cache database forward compatible apparatus comprising:
the system comprises a table information generating unit, a table information generating unit and a code generating script, wherein the table information generating unit is used for generating a table cache record structure, a table cache information class and a table structure information class corresponding to each table in an Oracle database, and the table cache information class comprises: the size of the table cache record structure and the offset of the indication part in the table cache record structure;
a compiling unit, configured to compile a serialization program and a cache management program based on the table cache record structure, the table cache information class, the table structure information class, and a cache database version number, and compile a calculation engine based on the table cache record structure and the cache database version number;
a serialization unit, configured to serialize a table record of the Oracle database into serialized data by using the serialization program, where the serialized data includes: a serialization file and a version file, wherein the size of the table cache record structure and the offset are stored in the serialization file;
The construction unit is used for loading the serialized data by using the cache management program and constructing a cache database;
and the access unit is used for accessing the cache database by using the calculation engine and obtaining the memory head addresses of all table records in the cache database so that the calculation engine can read data based on the memory head addresses, the table cache record structure size and the offset.
An electronic device, the electronic device comprising: a memory and a processor;
the memory is used for storing at least one instruction;
the processor is configured to execute the at least one instruction to implement the method of forward compatibility of the cache database described above.
A computer readable storage medium storing at least one instruction that when executed by a processor implements the method of cache database forward compatibility described above.
According to the technical scheme, the invention discloses a forward compatible method of a cache database and related equipment, a code generation script is utilized to generate a table cache record structure, a table cache information class and a table structure information class corresponding to each table in the Oracle database, a serialization program and a cache management program are compiled based on the table cache record structure, the table cache information class, the table structure information class and a version number of the cache database, a calculation engine is compiled based on the table cache record structure and the version number of the cache database, the serialization program is utilized to serialize the table record of the Oracle database into serialized data, the cache management program is utilized to load the serialized data to construct the cache database, the calculation engine is utilized to access the cache database, and memory head addresses of all table records in the cache database are obtained, so that the calculation engine can read the data based on the memory head addresses, the size of the table cache record structure and the offset. The invention generates the table cache record structure, the table cache information class and the table structure information class corresponding to each table, compiles the table cache record structure, the table cache information class and the table structure information class into the serialization program, the cache management program and the calculation engine, records the size of the table cache record structure and the offset of an indication part in the table cache record structure in the table cache information class, so that when the table structure is added with a new table or an additional field, the serialization program can be forward compatible with an Oracle database, the cache management program can be forward compatible with the serialization data, and the calculation engine can be forward compatible with the cache database.
Drawings
The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by reference to the following detailed description when taken in conjunction with the accompanying drawings. The same or similar reference numbers will be used throughout the drawings to refer to the same or like elements. It should be understood that the figures are schematic and that elements and components are not necessarily drawn to scale.
FIG. 1 is a flowchart of a method for forward compatibility of a cache database according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a storage structure of a serialized file according to an embodiment of the present invention;
FIG. 3 is a flowchart of a method for constructing a cache database by loading serialized data using a cache management program according to an embodiment of the present invention;
fig. 4 is a schematic diagram of a device structure for forward compatibility of a cache database according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure have been shown in the accompanying drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustration purposes only and are not intended to limit the scope of the present disclosure.
The term "including" and variations thereof as used herein are intended to be open-ended, i.e., including, but not limited to. The term "based on" is based at least in part on. The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments. Related definitions of other terms will be given in the description below.
It should be noted that the terms "first," "second," and the like in this disclosure are merely used to distinguish between different devices, modules, or units and are not used to define an order or interdependence of functions performed by the devices, modules, or units. .
The embodiment of the invention discloses a forward compatible method and related equipment for a cache database, which are characterized in that a code generation script is utilized to generate a table cache record structure, a table cache information class and a table structure information class corresponding to each table in the Oracle database, a serialization program and a cache management program are compiled based on the table cache record structure, the table cache information class and a version number of the cache database, a calculation engine is compiled based on the table cache record structure and the version number of the cache database, the table record of the Oracle database is serialized into serialized data by utilizing the serialization program, the serialized data is loaded by utilizing the cache management program to construct the cache database, the calculation engine is utilized to access the cache database, and memory head addresses of all table records in the cache database are obtained, so that the calculation engine can read the data based on the memory head addresses, the size of the table cache record structure and the offset. The invention generates the table cache record structure, the table cache information class and the table structure information class corresponding to each table, compiles the table cache record structure, the table cache information class and the table structure information class into the serialization program, the cache management program and the calculation engine, records the size of the table cache record structure and the offset of an indication part in the table cache record structure in the table cache information class, so that when the table structure is added with a new table or an additional field, the serialization program can be forward compatible with an Oracle database, the cache management program can be forward compatible with the serialization data, and the calculation engine can be forward compatible with the cache database.
Referring to fig. 1, a flowchart of a method for forward compatibility of a cache database is disclosed in an embodiment of the present invention, where the method includes:
and step S101, generating a table cache record structure, a table cache information class and a table structure information class corresponding to each table in the Oracle database by using a code generation script.
The table cache record structure refers to a structure of a table recorded in a serialized file and a cache database.
The table cache information class refers to various constant information and static functions of the table contained in the member.
The table structure information class refers to a table structure information class used for querying a cache database, and can be used for taking out all table records according to table names or searching the cache database according to the table names and indexes.
In this embodiment, the table cache information class includes: the size of the table cache record structure and the offset of the indication part in the table cache record structure.
Step S102, compiling a serialization program and a cache management program based on the table cache record structure, the table cache information class, the table structure information class and the cache database version number, and compiling a calculation engine based on the table cache record structure and the cache database version number.
The version number of the cache database is in the form of constant character strings in the code. The program checks it strictly. The developer can manually modify to limit the compatible functionality of the program.
Step S103, the table record of the Oracle database is serialized into serialized data by using a serialization program.
Wherein the serialization data comprises: serializing the file and version file.
The size of the table cache record structure and the offset of the indication part in the table cache record structure are stored in the serialization file.
The serialization procedure in this embodiment uses Proc programming.
When a new version of the table structure adds a new table or a new field, the serialization program is forward compatible with the Oracle database because the serialization program uses Proc programming and the fields in SQL (Structured Query Language ) used by the program are all present in the Oracle database and are of the same type.
Step S104, loading the serialized data by using a cache management program to construct a cache database.
The cache database comprises data blocks with non-fixed sizes and control blocks with fixed sizes. The data sequence of each table in the data block is divided into a data area and an index area. The control block stores various sizes, offsets, address information.
Step 105, accessing the cache database by using the computing engine to obtain the memory head addresses of all table records in the cache database, so that the computing engine reads data based on the memory head addresses, the table cache record structure size and the offset.
In practical application, the space structure of the control block in the cache database is fixed, and the hash bucket and hash chain table node of each table in the data block are also of fixed size, so that when the table structure of the cache database is newly added with fields, the calculation engine can query and obtain the memory head address of the table record.
Wherein the process of the computing engine accessing the cache database comprises: (1) obtaining all table records of a cache database; (2) searching the cache database according to the index value.
(1) The process of obtaining all table records of the cache database comprises the following steps: the computing engine is connected with the cache database, calls a corresponding query cache function, obtains the memory head address of the data area of the table, the number of table records, the size of the table cache record structure and the offset of the indication part in the table cache record structure from the control block through the name of the table which is input into the function, then traverses from the memory head address of the index area of the table, takes the size of the table cache record structure of the control block as a step length, obtains the memory head addresses of all table records, wherein the memory head addresses are the memory head addresses of the record part, and the memory head addresses of the indication part after the offset is added, and the two addresses are respectively converted into the record part and the indication part type of the cache record structure of the computing engine and then are transmitted out for the computing engine to read normally.
(2) The process of searching the cache database according to the index value comprises the following steps: the difference between searching the cache database according to the index value and obtaining all table records of the cache database is that in the called query cache function, the memory head address of the index area of the table is obtained through the name of the table, then the hash chain table node is obtained through the index value, and the head address of the table record with the same index value is obtained.
As can be seen from the foregoing, the present invention discloses a method for forward compatibility of a cache database, which uses a code generation script to generate a table cache record structure, a table cache information class and a table structure information class corresponding to each table in an Oracle database, compiles a serialization program and a cache management program based on the table cache record structure, the table cache information class and a version number of the cache database, compiles a calculation engine based on the table cache record structure and the version number of the cache database, utilizes the serialization program to serialize the table record of the Oracle database into serialized data, utilizes the cache management program to load the serialized data to construct the cache database, uses the calculation engine to access the cache database, and obtains the memory head addresses of all the table records in the cache database, so that the calculation engine can read the data based on the memory head addresses, the size of the table cache record structure and the offset. The invention generates the table cache record structure, the table cache information class and the table structure information class corresponding to each table, compiles the table cache record structure, the table cache information class and the table structure information class into the serialization program, the cache management program and the calculation engine, records the size of the table cache record structure and the offset of an indication part in the table cache record structure in the table cache information class, so that when the table structure is added with a new table or an additional field, the serialization program can be forward compatible with an Oracle database, the cache management program can be forward compatible with the serialization data, and the calculation engine can be forward compatible with the cache database.
In addition, the present invention can control whether the forward compatibility function is in effect by modifying the cached database version number in the code or configuration.
It is noted that the flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
To further optimize the above embodiment, step S101 may specifically include:
writing table configuration corresponding to each table in the Oracle database;
and analyzing and processing each table configuration by utilizing a code generation script, wherein each generated table corresponds to a table cache record structure, a table cache information class and a table structure information class.
In this embodiment, a table configuration corresponding to each table in the Oracle database is written, where the table configuration includes: the table name, field type and length of the table in the Oracle database, the screening condition during serialization, and the index used for querying the cache database.
Aiming at the table cache record structure, the table cache information class and the table structure information class, the following description is given:
(1) The table cache record structure is of a type of a table record in a serialized file and a cache database and comprises a record part and an indication part;
the field of the record part stores data from the Oracle database, and the field type corresponds to the table configuration;
the field of the indication part identifies whether a field is available, the field type is the same and the length is fixed.
The field order of the record section and the instruction section is the order of the fields in the table configuration.
Taking the buffer record structure before and after the addition of the global_ind field in the TPM table shown in table 1 as an example, the record portion and the instruction portion included in the buffer record structure will be described.
TABLE 1
In Table 1, ALIGN is the last fixed field used for byte alignment and is also used as an identification, not from the Oracle database.
(2) The table cache information class includes: constant information and static functions;
the constant information includes: the size of the table cache record structure, the offset of the indication part in the table cache record structure and the maximum length of the index value;
the static function includes: the method comprises a serialization function and a table taking and recording index value function, wherein the serialization function is used for generating a serialization file, and the table taking and recording index value function is used for taking out field values from a table record and splicing index values to establish an index area.
The offset of the indication part in the cache record structure in the table cache record structure is realized by sizeof and offsetof keywords and is used for traversing and reading the new version table record by the old version program.
The maximum length of the index value is used to calculate the index region size in the data block.
(3) The table structure information class stores table names and field information of the table structure of each table in a Map (a data structure) form, the table names are used as key words in key value pairs, the field names, the field sizes of the fields and the offset of the fields in a record part are spliced into character strings according to the sequence of the fields in a table cache record structure, and the character strings are used as value values in the key value pairs.
The field size and the offset need to instantiate the table cache record structure of each table, and then the table cache record structure is realized through sizeof and offsetf keywords.
To further optimize the above embodiment, step S103 may specifically include:
serializing the character string of the version number of the cache database and the Map in the table structure information class into a character string by using a serialization program to obtain a version file;
calling a serialization function of a table cache information class corresponding to each table by using a serialization program to generate a serialization file of each table;
wherein the serialization data comprises: version files and serialized files.
The serialization program is programmed using Proc, and the fields in SQL (Structured Query Language ) used by the serialization program are all present and of the same type in the Oracle database. The field order of the SELECT in which the SQL statement resides is the order of the fields in the table configuration.
In practical application, the serialization program uses a method such as boost serialization to serialize the character string of the version number of the cache database and the Map in the table structure information class into the character string, so as to obtain the version file.
It should be noted that, referring to the schematic storage structure of the serialized file shown in fig. 2, the tail of the serialized file is 1 fixed-length integer for storing the number of header information, and then the fixed-length header information of the corresponding number. Different header information can correspond to different SQL sentences used by the serialization function, even can correspond to different table cache record structures, namely, the table record in the file needs to be listed, and the type is the table cache record structure. The range (see table record 1, table record 2, … table record m, table record m+1, table record m+2, … … table record n in fig. 2 for details), number and service fields of the stored table records can be used for distinguishing data in the file on the service for personalized loading of the cache management program. The table cache record structure size and the offset of the indication part in the table cache record structure are also stored in the header information (see header information 1 and header information 2 in fig. 2 for details).
In summary, when a new version of the table structure adds a new table or adds a new field, the serialization program is forward compatible with the Oracle database because the serialization program uses Proc programming and the fields in SQL used by the program are all present in the Oracle database and are identical in type.
In order to further optimize the foregoing embodiments, referring to fig. 3, a flowchart of a method for constructing a cache database by loading serialized data by using a cache management program according to an embodiment of the present invention, that is, step S104 may specifically include:
step S201, the correctness of the version file in the serialized data is checked by using a cache management program.
The cache management program reads out the character strings from the version files, and uses a method such as boost serialization to inversely sequence the character strings into the cache database version numbers and Map maps, and compares the cache database version numbers and Map maps of the table structure information classes of the cache management program respectively.
And the cache management program strictly checks the version number of the cache database, and if the version numbers of the cache database are the same, the verification of the version number of the cache database is successful.
The verification process for the Map includes: the cache management program uses each table name of the program as a key, so that if the serialized data is more than the cache management program, the table cannot be checked, and the checking result is not influenced; and (3) respectively taking out a value character string from the map of the self table structure information class and the map from the version file by using the key, and if the latter can be obtained by adding the character string at the end of the former, indicating that the table structure in the serialized data adds a field after the table structure of the cache management program, wherein the value verification passes. And when all the values fetched pass the verification, the verification is successful.
Step S202, if the version number and Map of the cache database in the version file are the same as the version number and Map of the table structure information class of the cache database of the cache management program, determining that the version file passes verification.
Step S203, the size of the table cache record structure, the offset of the indication part in the table cache record structure, the version number of the cache database and the memory head address of each table index area in the cache database are written into the control block.
Step S204, a data block corresponding to the control block is created in the memory by using the cache management program, and the serialized file is loaded into the data block.
Wherein, the process of creating the database is specifically as follows:
for each table in the knowledge of the cache management program, the program firstly reads the number of header information from the tail part of the serialization file of the table, then traverses the header information, acquires the number of required table records and the size of the table cache record structure of the table, calculates the constant and the function in the program, then can acquire the size of the data area and the index area of each table, and after the size is rounded and added to MB, can calculate the size of the data block and the memory head address of the data area and the index area of each table in the cache database. And the cache management program applies for the shared memory space for the data block according to the calculated size, and writes the size, the offset, the memory head address information and the version number of the cache database into the control block.
If the variables are: the size of a table cache record structure of a certain table is recorded as Vs1, and the number of table records is recorded as Vn1; constant is given by: the maximum length of the index value is marked as Cs1, the size of the hash bucket is marked as Cs2, the size of a node of the hash chain table is marked as Cs3, the calculation function of the number of the hash bucket is marked as Fh (x), the number of the index is marked as Cn1, and the calculation method of the size of the table is as follows:
data region size=vs 1×vn1;
index region total size= (Cs 2 Fh (Vn 1) +cs 3 Vn 1) ×cn1+cs1×vn1.
The process of loading the data block includes:
the cache management program obtains the size and offset information of each table from the control block, then selects the needed header information from the corresponding serialization file, and calls the memcpy library function to copy the memory space where the table record corresponding to the header information is located to the data area of the table in the data block.
For each table known by the cache management program, the program acquires the number of table records, the size of the table cache record structure, the offset of an indication part in the table cache record structure, the memory head address of a data area of the table in a data block and the number of index areas from the memory head address of the data area, traverses the whole data area by taking the size of the table cache record structure in the control block as a step size, transmits the head address of each table record into a table-taking record index value function of a table cache information class, takes the value of a required field of the record part in the structure in the function and splices the values into an index value character string, copies the index value character string into the index area for establishing a hash index, and two pointers of the hash chain point to the table records of the data area and the index value character string of the index area respectively. The program will traverse the index value function with the same number of index areas, each time using the table corresponding to the index area to build all index areas.
Because the previous version verification can ensure that the length and the position of each field of a record part of a table cache record structure of the program are the same as those in the serialized file, when the table structure of the serialized file is added with the fields, the program can normally traverse the data area table record and can also correctly take out the index value by using a table-taking and recording index value function; when a new table is added to the table structure of the serialized file, the serialized file corresponding to the new table is not loaded.
Thus, the cache manager achieves forward compatibility of serialized data when a new version of the table structure adds a table or field.
Step S205, traversing the table records according to the size and the offset of the table cache record structure and suggesting indexes to obtain a cache database.
In summary, according to the invention, by generating the table cache record structure, the table cache information class and the table structure information class corresponding to each table, compiling the table cache record structure, the table cache information class and the table structure information class into the serialization program, the cache management program and the calculation engine, and simultaneously recording the size of the table cache record structure and the offset of the indication part in the table cache record structure in the table cache information class, when the table structure is added with a new table or an additional field, the serialization program can be forward compatible with the Oracle database, the cache management program can be forward compatible with the serialization data, and the calculation engine can be forward compatible with the cache database.
In practical applications, computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, including, but not limited to, an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
Corresponding to the embodiment of the method, the invention also discloses a device for forward compatibility of the cache database.
Referring to fig. 4, a schematic structural diagram of a device for forward compatibility of a cache database according to an embodiment of the present invention may include:
The table information generating unit 301 is configured to generate a table cache record structure, a table cache information class, and a table structure information class corresponding to each table in the Oracle database by using a code generation script, where the table cache information class includes: the size of the table cache record structure and the offset of the indication part in the table cache record structure;
the table cache record structure refers to a structure of a table recorded in a serialized file and a cache database.
The table cache information class refers to various constant information and static functions of the table contained in the member.
The table structure information class refers to a table structure information class used for querying a cache database, and can be used for taking out all table records according to table names or searching the cache database according to the table names and indexes.
A compiling unit 302, configured to compile a serialization program and a cache management program based on the table cache record structure, the table cache information class, the table structure information class, and a cache database version number, and compile a calculation engine based on the table cache record structure and the cache database version number;
the version number of the cache database is in the form of constant character strings in the code. The program checks it strictly. The developer can manually modify to limit the compatible functionality of the program.
A serialization unit 303, configured to serialize the table record of the Oracle database into serialized data by using the serialization program, where the serialized data includes: a serialization file and a version file, wherein the size of the table cache record structure and the offset are stored in the serialization file;
wherein the serialization data comprises: serializing the file and version file.
The size of the table cache record structure and the offset of the indication part in the table cache record structure are stored in the serialization file.
The serialization procedure in this embodiment uses Proc programming.
When a new version of the table structure adds a new table or a new field, the serialization program is forward compatible with the Oracle database because the serialization program uses Proc programming and the fields in SQL (Structured Query Language ) used by the program are all present in the Oracle database and are of the same type.
A construction unit 304, configured to load the serialized data by using the cache management program, and construct a cache database;
the cache database comprises data blocks with non-fixed sizes and control blocks with fixed sizes. The data sequence of each table in the data block is divided into a data area and an index area. The control block stores various sizes, offsets, address information.
And an accessing unit 305, configured to access the cache database by using the computing engine, and obtain the memory head addresses of all table records in the cache database, so that the computing engine reads data based on the memory head addresses, the table cache record structure size and the offset.
In practical application, the space structure of the control block in the cache database is fixed, and the hash bucket and hash chain table node of each table in the data block are also of fixed size, so that when the table structure of the cache database is newly added with fields, the calculation engine can query and obtain the memory head address of the table record.
Wherein the process of the computing engine accessing the cache database comprises: (1) obtaining all table records of a cache database; (2) searching the cache database according to the index value.
(1) The process of obtaining all table records of the cache database comprises the following steps: the computing engine is connected with the cache database, calls a corresponding query cache function, obtains the memory head address of the data area of the table, the number of table records, the size of the table cache record structure and the offset of the indication part in the table cache record structure from the control block through the name of the table which is input into the function, then traverses from the memory head address of the index area of the table, takes the size of the table cache record structure of the control block as a step length, obtains the memory head addresses of all table records, wherein the memory head addresses are the memory head addresses of the record part, and the memory head addresses of the indication part after the offset is added, and the two addresses are respectively converted into the record part and the indication part type of the cache record structure of the computing engine and then are transmitted out for the computing engine to read normally.
(2) The process of searching the cache database according to the index value comprises the following steps: the difference between searching the cache database according to the index value and obtaining all table records of the cache database is that in the called query cache function, the memory head address of the index area of the table is obtained through the name of the table, then the hash chain table node is obtained through the index value, and the head address of the table record with the same index value is obtained.
As can be seen from the foregoing, the present invention discloses a device for forward compatibility of a cache database, which generates a table cache record structure, a table cache information class, and a table structure information class corresponding to each table in an Oracle database by using a code generation script, compiles a serialization program and a cache management program based on the table cache record structure, the table cache information class, and a table cache version number, compiles a calculation engine based on the table cache record structure and the cache database version number, utilizes the serialization program to serialize the table record of the Oracle database into serialized data, utilizes the cache management program to load the serialized data to construct the cache database, utilizes the calculation engine to access the cache database, and obtains the memory head addresses of all the table records in the cache database, so that the calculation engine can read the data based on the memory head addresses, the table cache record structure size, and the offset. The invention generates the table cache record structure, the table cache information class and the table structure information class corresponding to each table, compiles the table cache record structure, the table cache information class and the table structure information class into the serialization program, the cache management program and the calculation engine, records the size of the table cache record structure and the offset of an indication part in the table cache record structure in the table cache information class, so that when the table structure is added with a new table or an additional field, the serialization program can be forward compatible with an Oracle database, the cache management program can be forward compatible with the serialization data, and the calculation engine can be forward compatible with the cache database.
In addition, the present invention can control whether the forward compatibility function is in effect by modifying the cached database version number in the code or configuration.
The units described in the embodiments of the present disclosure may be implemented by software or hardware. The name of the unit does not in any way constitute a limitation of the unit itself, for example the first acquisition unit may also be described as "unit acquiring at least two internet protocol addresses".
The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a system on a chip (SOC), a Complex Programmable Logic Device (CPLD), and the like.
To further optimize the above embodiment, the table information generating unit 301 may specifically be configured to:
writing table configuration corresponding to each table in the Oracle database;
and analyzing and processing each table configuration by utilizing a code generation script, wherein each generated table corresponds to a table cache record structure, a table cache information class and a table structure information class.
In this embodiment, a table configuration corresponding to each table in the Oracle database is written, where the table configuration includes: the table name, field type and length of the table in the Oracle database, the screening condition during serialization, and the index used for querying the cache database.
Details of the table cache record structure, the table cache information class and the table structure information class are described in the corresponding parts of the method embodiment, and are not repeated here.
To further optimize the above embodiment, the serialization unit 303 may specifically be configured to:
serializing the character string of the version number of the cache database and the Map in the table structure information class into a character string by using a serialization program to obtain a version file;
calling a serialization function of a table cache information class corresponding to each table by using a serialization program to generate a serialization file of each table;
wherein the serialization data comprises: version files and serialized files.
The serialization program is programmed using Proc, and the fields in SQL (Structured Query Language ) used by the serialization program are all present and of the same type in the Oracle database. The field order of the SELECT in which the SQL statement resides is the order of the fields in the table configuration.
In practical application, the serialization program uses a method such as boost serialization to serialize the character string of the version number of the cache database and the Map in the table structure information class into the character string, so as to obtain the version file.
To further optimize the above embodiment, the construction unit 304 is specifically configured to:
the cache management program is utilized to carry out correctness checking on version files in the serialized data;
if the version number and the mapping Map of the cache database in the version file are the same as the version number and the mapping Map of the table structure information class of the cache database of the cache management program, determining that the version file passes the verification;
writing the size of the table cache record structure, the offset, the version number of the cache database and the memory head address of each table index area in the cache database into a control block;
creating a data block corresponding to the control block in a memory by using the cache management program, and loading the serialized file into the data block;
and traversing the table record according to the size of the table cache record structure and the offset and suggesting an index to obtain the cache database.
The specific operation principle of each component in the embodiment of the device should be specifically described, please refer to the corresponding portion of the method embodiment, and the detailed description is omitted herein.
Corresponding to the above embodiment, as shown in fig. 5, the present invention further provides an electronic device, where the electronic device may include: a processor 1 and a memory 2;
wherein the processor 1 and the memory 2 complete communication with each other through the communication bus 3;
a processor 1 for executing at least one instruction;
a memory 2 for storing at least one instruction;
the processor 1 may be a central processing unit CPU or a specific integrated circuit ASIC (Application Specific Integrated Circuit) or one or more integrated circuits configured to implement embodiments of the present invention.
The memory 2 may comprise a high-speed RAM memory or may further comprise a non-volatile memory (non-volatile memory), such as at least one disk memory.
Wherein the processor executes at least one instruction to implement the steps shown in the method embodiment for forward compatibility of the cache database.
Corresponding to the above embodiment, the invention also discloses a computer readable storage medium storing at least one instruction, which when executed by a processor, implements the steps shown in the method embodiment of forward compatibility of the cache database.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
As can be seen from the foregoing, the present invention discloses an electronic device and a computer readable storage medium, which generates a table cache record structure, a table cache information class, and a table structure information class corresponding to each table in an Oracle database by using a code generation script, compiles a serialization program and a cache management program based on the table cache record structure, the table cache information class, the table structure information class, and a table cache version number, compiles a calculation engine based on the table cache record structure and the cache database version number, serializes a table record of the Oracle database into serialized data by using the serialization program, loads the serialized data by using the cache management program to construct a cache database, accesses the cache database by using the calculation engine, and obtains a memory first address of all table records in the cache database, so that the calculation engine reads data based on the memory first address, the table cache record structure size, and offset. The invention generates the table cache record structure, the table cache information class and the table structure information class corresponding to each table, compiles the table cache record structure, the table cache information class and the table structure information class into the serialization program, the cache management program and the calculation engine, records the size of the table cache record structure and the offset of an indication part in the table cache record structure in the table cache information class, so that when the table structure is added with a new table or an additional field, the serialization program can be forward compatible with an Oracle database, the cache management program can be forward compatible with the serialization data, and the calculation engine can be forward compatible with the cache database.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are example forms of implementing the claims.
While several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.
The foregoing description is only of the preferred embodiments of the present disclosure and description of the principles of the technology being employed. It will be appreciated by persons skilled in the art that the scope of the disclosure referred to in this disclosure is not limited to the specific combinations of features described above, but also covers other embodiments which may be formed by any combination of features described above or equivalents thereof without departing from the spirit of the disclosure. Such as those described above, are mutually substituted with the technical features having similar functions disclosed in the present disclosure (but not limited thereto).
Claims (10)
1. A method for forward compatibility of a cache database, comprising:
generating a table cache record structure, a table cache information class and a table structure information class corresponding to each table in an Oracle database by using a code generation script, wherein the table cache information class comprises: the size of the table cache record structure and the offset of the indication part in the table cache record structure;
compiling a serialization program and a cache management program based on the table cache record structure, the table cache information class, the table structure information class and a cache database version number, and compiling a calculation engine based on the table cache record structure and the cache database version number;
serializing the table records of the Oracle database into serialized data using the serialization program, wherein the serialized data comprises: a serialization file and a version file, wherein the size of the table cache record structure and the offset are stored in the serialization file;
loading the serialized data by using the cache management program to construct a cache database;
and accessing the cache database by using the calculation engine to obtain the memory head addresses of all table records in the cache database, so that the calculation engine reads data based on the memory head addresses, the table cache record structure size and the offset.
2. The method according to claim 1, wherein the generating, by using the code generation script, a table cache record structure, a table cache information class, and a table structure information class corresponding to each table in the Oracle database includes:
writing table configuration corresponding to each table in the Oracle database;
and analyzing and processing each table configuration by using the code generation script, wherein each generated table corresponds to the table cache record structure, the table cache information class and the table structure information class.
3. A method according to claim 1 or 2, wherein the table cache record structure is a serialized file and a type of table record in a cache database, comprising a record portion and an indication portion;
the field of the record part stores data from the Oracle database, and the field type corresponds to table configuration;
the field of the indication part identifies whether fields are available, the field types are the same and the length is fixed.
4. The method according to claim 1 or 2, wherein the table cache information class comprises: constant information and static functions;
the constant information includes: the size of the table cache record structure, the offset of the indication part in the table cache record structure and the maximum length of the index value;
The static function includes: the method comprises a serialization function and a table taking and recording index value function, wherein the serialization function is used for generating a serialization file, and the table taking and recording index value function is used for taking out field values from a table record and splicing index values to establish an index area.
5. The method according to claim 1 or 2, wherein the table structure information class stores table names and field information of a table structure of each table in a Map form, the table names are used as key words in key value pairs, field names, field sizes of the fields and offsets of the fields in a record part are spliced into character strings in the order of the fields in a table cache record structure, and the character strings are used as values in the key value pairs.
6. The method of claim 1, wherein said serializing the table records of the Oracle database into serialized data using the serialization program comprises:
serializing the character string of the version number of the cache database and the Map in the table structure information class into a character string by using the serialization program to obtain a version file;
calling a serialization function of a table cache information class corresponding to each table by using the serialization program to generate a serialization file of each table;
Wherein the serialized data comprises: the version file and the serialized file;
the serialization program is programmed using Proc, and the fields in the structured query language SQL used by the serialization program are all present and of the same type in the Oracle database.
7. The method of claim 1, wherein loading the serialized data with the cache manager to construct a cache database comprises:
the cache management program is utilized to carry out correctness checking on version files in the serialized data;
if the version number and the mapping Map of the cache database in the version file are the same as the version number and the mapping Map of the table structure information class of the cache database of the cache management program, determining that the version file passes the verification;
writing the size of the table cache record structure, the offset, the version number of the cache database and the memory head address of each table index area in the cache database into a control block;
creating a data block corresponding to the control block in a memory by using the cache management program, and loading the serialized file into the data block;
and traversing the table record according to the size of the table cache record structure and the offset and suggesting an index to obtain the cache database.
8. A forward-compatible device for caching databases, comprising:
the system comprises a table information generating unit, a table information generating unit and a code generating script, wherein the table information generating unit is used for generating a table cache record structure, a table cache information class and a table structure information class corresponding to each table in an Oracle database, and the table cache information class comprises: the size of the table cache record structure and the offset of the indication part in the table cache record structure;
a compiling unit, configured to compile a serialization program and a cache management program based on the table cache record structure, the table cache information class, the table structure information class, and a cache database version number, and compile a calculation engine based on the table cache record structure and the cache database version number;
a serialization unit, configured to serialize a table record of the Oracle database into serialized data by using the serialization program, where the serialized data includes: a serialization file and a version file, wherein the size of the table cache record structure and the offset are stored in the serialization file;
the construction unit is used for loading the serialized data by using the cache management program and constructing a cache database;
And the access unit is used for accessing the cache database by using the calculation engine and obtaining the memory head addresses of all table records in the cache database so that the calculation engine can read data based on the memory head addresses, the table cache record structure size and the offset.
9. An electronic device, the electronic device comprising: a memory and a processor;
the memory is used for storing at least one instruction;
the processor is configured to execute the at least one instruction to implement the method of cache database forward compatibility of any one of claims 1 to 7.
10. A computer readable storage medium storing at least one instruction which when executed by a processor implements a method of cache database forward compatibility as claimed in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311866154.4A CN117827837A (en) | 2023-12-30 | 2023-12-30 | Forward compatibility method of cache database and related equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311866154.4A CN117827837A (en) | 2023-12-30 | 2023-12-30 | Forward compatibility method of cache database and related equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117827837A true CN117827837A (en) | 2024-04-05 |
Family
ID=90522345
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311866154.4A Pending CN117827837A (en) | 2023-12-30 | 2023-12-30 | Forward compatibility method of cache database and related equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117827837A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118034940A (en) * | 2024-04-12 | 2024-05-14 | 北京麟卓信息科技有限公司 | Harvard architecture data cache size measuring and calculating method based on cache organization form detection |
-
2023
- 2023-12-30 CN CN202311866154.4A patent/CN117827837A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118034940A (en) * | 2024-04-12 | 2024-05-14 | 北京麟卓信息科技有限公司 | Harvard architecture data cache size measuring and calculating method based on cache organization form detection |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20230306135A1 (en) | Masking sensitive information in records of filtered accesses to unstructured data | |
| CN109952564B (en) | Formation and manipulation of test data in a database system | |
| US11392567B2 (en) | Just-in-time multi-indexed tables in a shared log | |
| US11334474B2 (en) | Fast change impact analysis tool for large-scale software systems | |
| US10650032B1 (en) | Filtering pipeline optimizations for unstructured data | |
| US10678784B2 (en) | Dynamic column synopsis for analytical databases | |
| US10621071B2 (en) | Formation and manipulation of test data in a database system | |
| US10664508B1 (en) | Server-side filtering of unstructured data items at object storage services | |
| CN117827837A (en) | Forward compatibility method of cache database and related equipment | |
| US10649981B2 (en) | Direct access to object state in a shared log | |
| US10733163B2 (en) | Record insertion by generating unique index bases | |
| CN113312338A (en) | Data consistency checking method, device, equipment, medium and program product | |
| US11074260B2 (en) | Space-efficient methodology for representing label information in large graph data for fast distributed graph query | |
| CN114281803A (en) | Data migration method, device, equipment, medium and program product | |
| US11080332B1 (en) | Flexible indexing for graph databases | |
| US10635541B2 (en) | Fine-grained conflict resolution in a shared log | |
| US9697239B1 (en) | Token-based database system and method of interfacing with the token-based database system | |
| US10908924B2 (en) | System and methods for loading objects from hash chains | |
| US11256602B2 (en) | Source code file retrieval | |
| US10949336B2 (en) | Path-sensitive static analysis | |
| CN112860628A (en) | File system scale management method and system | |
| US20200125361A1 (en) | Dynamic modeling for opaque code during static analysis | |
| US11907217B2 (en) | Database object validation for reusing captured query plans | |
| US11409729B2 (en) | Managing database object schema virtual changes | |
| CN116069755A (en) | mapper file processing method and related equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |