CN103106144B - A kind of internal memory index compression method and apparatus - Google Patents
A kind of internal memory index compression method and apparatus Download PDFInfo
- Publication number
- CN103106144B CN103106144B CN201110362273.7A CN201110362273A CN103106144B CN 103106144 B CN103106144 B CN 103106144B CN 201110362273 A CN201110362273 A CN 201110362273A CN 103106144 B CN103106144 B CN 103106144B
- Authority
- CN
- China
- Prior art keywords
- index
- internal memory
- key assignments
- hash
- table space
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Memory System (AREA)
Abstract
The present invention discloses a kind of internal memory index compression method and apparatus.Described method comprises: in internal memory, create index ltsh chain table space; When there being internal memory index key assignments to need to be saved in internal memory, according to the first hash function, Hash calculation being carried out to this internal memory index key assignments, obtaining Hash key assignments; Checking in described index ltsh chain table space whether there is the value identical with described Hash key assignments, is do not operate, otherwise is saved in described index ltsh chain table space by described Hash key assignments.Technical scheme of the present invention can effectively be compressed internal memory index, reduces internal memory index taking internal memory.
Description
Technical field
The present invention relates to field of computer technology, particularly a kind of internal memory index compression method and apparatus.
Background technology
In prior art, comprise character types in internal memory index key assignments, because the length of character types is uncertain, the amount of ram taken is also uncertain.The use of internal memory index to internal memory is higher is in most cases all cause because the occupancy of character types is higher.
Summary of the invention
In view of this, the invention provides a kind of internal memory index compression method and apparatus, technical scheme of the present invention can effectively be compressed internal memory index, reduces internal memory index taking internal memory.
For achieving the above object, technical scheme of the present invention is achieved in that
The invention discloses a kind of internal memory index compression method, the method comprises:
Index ltsh chain table space is created in internal memory;
When there being internal memory index key assignments to need to be saved in internal memory, according to the first hash function, Hash calculation being carried out to this internal memory index key assignments, obtaining Hash key assignments;
Checking in described index ltsh chain table space whether there is the value identical with described Hash key assignments, is do not operate, otherwise is saved in described index ltsh chain table space by described Hash key assignments.
In said method, described index ltsh chain table space comprises n index area, and corresponding sequence number is followed successively by 0 ~ n-1;
Check that whether having there is the value identical with described Hash key assignments in described index ltsh chain table space comprises: according to the second hash function, Hash calculation is carried out to this internal memory index key assignments, the value that mould n computing obtains in 0 ~ n-1 is carried out to this Hash calculation result, determine the index area that sequence number is corresponding with this value, check in this index area determined whether there is the value identical with described Hash key assignments;
Describedly Hash key assignments is saved in described index ltsh chain table space and comprises: described Hash key assignments is saved in this index area determined.
In the above-mentioned methods, described Hash key assignments is saved in this index area determined and comprises: build a directory entry, this directory entry comprises described Hash key assignments and a chain list index; The directory entry of structure is added in this index area determined; Wherein, the next directory entry of described chained list pointed.
In the above-mentioned methods, described first hash function is DJB hash function; Described second hash function is CRC16 hash function.
The invention also discloses a kind of internal memory index compression device, this device comprises: memory modules, creation module and internal memory index compression module, wherein:
Creation module, for creating index ltsh chain table space in memory modules, and notifies internal memory index compression module;
Internal memory index compression module, for when receiving internal memory index key assignments, according to the first hash function, Hash calculation is carried out to this internal memory index key assignments, obtain Hash key assignments, then check in the index ltsh chain table space in memory modules whether there is the value identical with described Hash key assignments, be do not operate, otherwise described Hash key assignments is saved in described index ltsh chain table space.
In said apparatus, described creation module, for creating the index ltsh chain table space comprising n index area in memory modules; The sequence number of this n index area is followed successively by 0 ~ n-1;
Described internal memory index compression module, for carrying out Hash calculation according to the second hash function to this internal memory index key assignments, the value that mould n computing obtains in 0 ~ n-1 is carried out to this Hash calculation result, determine the index area that sequence number is corresponding with this value, check in this index area determined whether there is the value identical with described Hash key assignments, if there is no then described Hash key assignments is saved in this index area determined.
In said apparatus, described internal memory index compression module, for building a directory entry, adds in this index area determined by the directory entry of structure; Wherein, described directory entry comprises described Hash key assignments and a chain list index; The next directory entry of described chained list pointed.
In said apparatus, described internal memory index compression module, for carrying out Hash calculation according to DJB hash function to this internal memory index key assignments, obtains Hash key assignments; And for carrying out Hash calculation according to CRC16 hash function to this internal memory index key assignments, the value that mould n computing obtains in 0 ~ n-1 is carried out to this Hash calculation result.
The beneficial effect of the embodiment of the present invention is: in technical scheme of the present invention, after Hash operation is carried out to internal memory index key value, the Hash key assignments of integer type is saved in internal memory, the data of integer type take data the taking internal memory that will be significantly smaller than character types to internal memory, the scheme of therefore directly preserving the internal memory index key assignments of character types in hinge structure in internal memory is compared, effectively can compress internal memory index, reduce internal memory index taking internal memory.
Accompanying drawing explanation
Fig. 1 is the process flow diagram of a kind of internal memory index compression method in the embodiment of the present invention;
Fig. 2 is the structural representation of a kind of internal memory index compression device in the embodiment of the present invention.
Embodiment
For making the object, technical solutions and advantages of the present invention clearly, below in conjunction with accompanying drawing, embodiment of the present invention is described further in detail.
Fig. 1 is the process flow diagram of a kind of internal memory index compression method in the embodiment of the present invention.As shown in Figure 1, the method comprises:
101, in internal memory, create index ltsh chain table space;
102, when there being internal memory index key assignments to need to be saved in internal memory, according to the first hash function, Hash calculation being carried out to this internal memory index key assignments, obtaining Hash key assignments;
103, checking in described index ltsh chain table space whether there is the value identical with described Hash key assignments, is do not operate, otherwise is saved in described index ltsh chain table space by described Hash key assignments.
The index ltsh chain table space created in step 101 comprises n index area, and corresponding sequence number is followed successively by 0 ~ n-1.
Whether checking in step 103 has existed the value identical with described Hash key assignments in described index ltsh chain table space comprises: carry out Hash calculation according to the second hash function to this internal memory index key assignments, the value that mould n computing obtains in 0 ~ n-1 is carried out to this Hash calculation result, determine the index area that sequence number is corresponding with this value, check in this index area determined whether there is the value identical with described Hash key assignments;
Described in step 103, Hash key assignments is saved in described index ltsh chain table space to comprise: described Hash key assignments is saved in this index area determined.
Be saved in this index area determined by described Hash key assignments and be specifically as follows: build a directory entry, this directory entry comprises described Hash key assignments and a chain list index; The directory entry of structure is added in this index area determined; Wherein, the next directory entry of described chained list pointed.
The first described hash function and the second hash function can get arbitrary hash function.In one embodiment of the invention, described first hash function is DJB hash function; Described second hash function is CRC16 hash function.
Provide a specific embodiment to be below described such scheme.
First in internal memory, the index ltsh chain table space that one comprises n index area is created, specifically can be as shown in table 1:
| Index area 0 |
| Index area 1 |
| ...... |
| Index area n-1 |
Table 1
N gets prime number 16785407 in the present embodiment.
The data structure of index of definition entry comprises: Hash key assignments and chain list index, wherein:
Hash key assignments: integer type, the integer that the internal memory index key assignments for character types obtains after Hash operation;
Chain list index: integer type, points to next directory entry structure, easy-to-look-up next directory entry.
When needing the internal memory index key assignments Key1 of character types to be saved in internal memory, calculate according to hash function CRC16, namely CRC16 (Key1) obtains integer h1, to h1 carry out modulo operation to x, i.e. x=h1%n, the computational complexity of h1%n is O (1), and quickly, x is the sequence number that Key1 needs the index area preserved to speed.The directory entry index1 of a structure Key1, the Hash key assignments Hkey1 in index1 carries out Hash operation according to hash function DJB to Key1 to obtain, i.e. Hash key assignments Hkey1=DJB (Key1).All directory entries in traversal index area x, check whether Hash key assignments Hkey1 exists, and is, does not do any operation, otherwise are added in index area x by directory entry index1.The next directory entry of the chained list pointed index1 wherein in directory entry index1.
What deposit in internal memory due to directory entry is the Hash key assignments of integer type, instead of the internal memory index key assignments of concrete character types, therefore index internal memory can be restricted to fixed size (4 bytes, integer), namely character type major key be mapped to the fixing integer structure of EMS memory occupation.
Based on above-described embodiment, provide the structure of a kind of internal memory index compression device in the present invention.
Fig. 2 is the structural representation of a kind of internal memory index compression device in the embodiment of the present invention.As shown in Figure 2, this device comprises: memory modules 201, creation module 202 and internal memory index compression module 203, wherein:
Creation module 202, for creating index ltsh chain table space in memory modules 201, and notifies internal memory index compression module 203;
Internal memory index compression module 203, for when receiving internal memory index key assignments, according to the first hash function, Hash calculation is carried out to this internal memory index key assignments, obtain Hash key assignments, then check in the index ltsh chain table space in memory modules 201 whether there is the value identical with described Hash key assignments, be do not operate, otherwise described Hash key assignments is saved in described index ltsh chain table space.
In fig. 2, described creation module 202, for creating the index ltsh chain table space comprising n index area in memory modules 201; The sequence number of this n index area is followed successively by 0 ~ n-1;
Described internal memory index compression module 203, for carrying out Hash calculation according to the second hash function to this internal memory index key assignments, the value that mould n computing obtains in 0 ~ n-1 is carried out to this Hash calculation result, determine the index area that sequence number is corresponding with this value, check in this index area determined whether there is the value identical with described Hash key assignments, if there is no then described Hash key assignments is saved in this index area determined.
In fig. 2, described internal memory index compression module 203, for building a directory entry, adds in this index area determined by the directory entry of structure; Wherein, described directory entry comprises described Hash key assignments and a chain list index; The next directory entry of described chained list pointed.
In fig. 2, described internal memory index compression module 203, for carrying out Hash calculation according to DJB hash function to this internal memory index key assignments, obtains Hash key assignments; And for carrying out Hash calculation according to CRC16 hash function to this internal memory index key assignments, the value that mould n computing obtains in 0 ~ n-1 is carried out to this Hash calculation result.
In sum, in technical scheme of the present invention, after Hash operation is carried out to internal memory index key value, the Hash key assignments of integer type is saved in internal memory, the data of integer type take data the taking internal memory that will be significantly smaller than character types to internal memory, the scheme of therefore directly preserving the internal memory index key assignments of character types in hinge structure in internal memory is compared, effectively can compress internal memory index, reduce internal memory index taking internal memory.
The foregoing is only preferred embodiment of the present invention, be not intended to limit protection scope of the present invention.All any amendments done within the spirit and principles in the present invention, equivalent replacement, improvement etc., be all included in protection scope of the present invention.
Claims (8)
1. an internal memory index compression method, is characterized in that, the method comprises:
Index ltsh chain table space is created in internal memory;
When there being internal memory index key assignments to need to be saved in internal memory, according to the first hash function, Hash calculation being carried out to this internal memory index key assignments, obtaining the Hash key assignments of integer type;
Checking in described index ltsh chain table space whether there is the value identical with the Hash key assignments of described integer type, is do not operate, otherwise is saved in described index ltsh chain table space by the Hash key assignments of described integer type.
2. method according to claim 1, is characterized in that, described index ltsh chain table space comprises n index area, and corresponding sequence number is followed successively by 0 ~ n-1;
Check that whether having there is the value identical with described Hash key assignments in described index ltsh chain table space comprises: according to the second hash function, Hash calculation is carried out to this internal memory index key assignments, the value that mould n computing obtains in 0 ~ n-1 is carried out to this Hash calculation result, determine the index area that sequence number is corresponding with this value, check in this index area determined whether there is the value identical with described Hash key assignments;
Describedly Hash key assignments is saved in described index ltsh chain table space and comprises: described Hash key assignments is saved in this index area determined.
3. method according to claim 2, is characterized in that, described Hash key assignments is saved in this index area determined and comprises:
Build a directory entry, this directory entry comprises described Hash key assignments and a chain list index;
The directory entry of structure is added in this index area determined;
Wherein, the next directory entry of described chained list pointed.
4. method according to claim 2, is characterized in that,
Described first hash function is DJB hash function;
Described second hash function is CRC16 hash function.
5. an internal memory index compression device, is characterized in that, this device comprises: memory modules, creation module and internal memory index compression module, wherein:
Creation module, for creating index ltsh chain table space in memory modules, and notifies internal memory index compression module;
Internal memory index compression module, for when receiving internal memory index key assignments, according to the first hash function, Hash calculation is carried out to this internal memory index key assignments, obtain the Hash key assignments of integer type, then check in the index ltsh chain table space in memory modules whether there is the value identical with the Hash key assignments of described integer type, be do not operate, otherwise the Hash key assignments of described integer type is saved in described index ltsh chain table space.
6. device according to claim 5, is characterized in that,
Described creation module, for creating the index ltsh chain table space comprising n index area in memory modules; The sequence number of this n index area is followed successively by 0 ~ n-1;
Described internal memory index compression module, for carrying out Hash calculation according to the second hash function to this internal memory index key assignments, the value that mould n computing obtains in 0 ~ n-1 is carried out to this Hash calculation result, determine the index area that sequence number is corresponding with this value, check in this index area determined whether there is the value identical with described Hash key assignments, if there is no then described Hash key assignments is saved in this index area determined.
7. device according to claim 6, is characterized in that,
Described internal memory index compression module, for building a directory entry, adds in this index area determined by the directory entry of structure;
Wherein, described directory entry comprises described Hash key assignments and a chain list index; The next directory entry of described chained list pointed.
8. device according to claim 6, is characterized in that,
Described internal memory index compression module, for carrying out Hash calculation according to DJB hash function to this internal memory index key assignments, obtains Hash key assignments;
Described internal memory index compression module, for carrying out Hash calculation according to CRC16 hash function to this internal memory index key assignments, carries out to this Hash calculation result the value that mould n computing obtains in 0 ~ n-1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110362273.7A CN103106144B (en) | 2011-11-15 | 2011-11-15 | A kind of internal memory index compression method and apparatus |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110362273.7A CN103106144B (en) | 2011-11-15 | 2011-11-15 | A kind of internal memory index compression method and apparatus |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103106144A CN103106144A (en) | 2013-05-15 |
| CN103106144B true CN103106144B (en) | 2015-10-28 |
Family
ID=48314020
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201110362273.7A Active CN103106144B (en) | 2011-11-15 | 2011-11-15 | A kind of internal memory index compression method and apparatus |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103106144B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108153757B (en) * | 2016-12-02 | 2020-04-03 | 深圳市中兴微电子技术有限公司 | Hash table management method and device |
| CN109508334B (en) * | 2018-11-23 | 2019-10-11 | 中科驭数（北京）科技有限公司 | For the data compression method of block chain database, access method and system |
| CN112988654A (en) * | 2019-12-12 | 2021-06-18 | 蜜蜂计算(香港)股份有限公司 | Circuit system suitable for Hash algorithm |
| CN112395213B (en) * | 2020-11-18 | 2023-05-30 | 之江实验室 | ACEH index structure and method based on memory hot spot data |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101315628A (en) * | 2007-06-01 | 2008-12-03 | 华为技术有限公司 | Internal memory database system and method and device for implementing internal memory data base |
| CN101719141A (en) * | 2009-12-24 | 2010-06-02 | 成都市华为赛门铁克科技有限公司 | File processing method and system based on directory object |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080168209A1 (en) * | 2007-01-09 | 2008-07-10 | Ibm Corporation | Data protection via software configuration of multiple disk drives |
-
2011
- 2011-11-15 CN CN201110362273.7A patent/CN103106144B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101315628A (en) * | 2007-06-01 | 2008-12-03 | 华为技术有限公司 | Internal memory database system and method and device for implementing internal memory data base |
| CN101719141A (en) * | 2009-12-24 | 2010-06-02 | 成都市华为赛门铁克科技有限公司 | File processing method and system based on directory object |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103106144A (en) | 2013-05-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021042515A1 (en) | Graph data storage and query method and apparatus, and computer-readable storage medium | |
| CN100562021C (en) | A kind of control method of distributed multi-source data that can be synchronous and device | |
| CN103106144B (en) | A kind of internal memory index compression method and apparatus | |
| CN102970043B (en) | A kind of compression hardware system based on GZIP and accelerated method thereof | |
| CN101599986B (en) | Method for increasing transmission speed of files and computer system | |
| CN107027036A (en) | A kind of FPGA isomeries accelerate decompression method, the apparatus and system of platform | |
| CN108804768A (en) | One kind making the light-weighted method of BIM models | |
| US20180254888A1 (en) | Combining hashes of data blocks | |
| CN102780685A (en) | Method and system for compressing and encrypting data | |
| CN106648955B (en) | Compression method and related device | |
| CN104104717A (en) | Inputting channel data statistical method and device | |
| US10817178B2 (en) | Compressing and compacting memory on a memory device wherein compressed memory pages are organized by size | |
| CN106302245A (en) | The compression method of packet and device in a kind of LTE system | |
| CN104579948A (en) | Method and device for fragmenting message | |
| CN104579970B (en) | A kind of strategy matching device of IPv6 messages | |
| CN103580991B (en) | The method for uploading and equipment of a kind of Email attachment | |
| CN102930004B (en) | Hash value storage method, device and chip | |
| CN107422980B (en) | Internet of things data file storage system and data file storage method thereof | |
| CN106936545A (en) | Data transmission method and device | |
| CN104503862B (en) | The method and apparatus for obtaining the check value using channel bag | |
| CN105389387A (en) | Compression based deduplication performance and deduplication rate improving method and system | |
| CN108694205B (en) | Method and device for matching target field | |
| CN112486848A (en) | Test data generation method and device, chip and storage medium | |
| CN102316173A (en) | Method and device for aggregating network address | |
| CN103685509A (en) | Method for synchronizing file delta |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CP02 | Change in the address of a patent holder |
Address after: Room 810, 8 / F, 34 Haidian Street, Haidian District, Beijing 100080 Patentee after: BEIJING D-MEDIA COMMUNICATION TECHNOLOGY Co.,Ltd. Address before: 100089 Beijing city Haidian District wanquanzhuang Road No. 28 Wanliu new building A block 5 layer Patentee before: BEIJING D-MEDIA COMMUNICATION TECHNOLOGY Co.,Ltd. |
|
| CP02 | Change in the address of a patent holder |