CN102129472B - Construction method for high-efficiency hybrid storage structure of semantic-orient search engine - Google Patents
Construction method for high-efficiency hybrid storage structure of semantic-orient search engine Download PDFInfo
- Publication number
- CN102129472B CN102129472B CN 201110093092 CN201110093092A CN102129472B CN 102129472 B CN102129472 B CN 102129472B CN 201110093092 CN201110093092 CN 201110093092 CN 201110093092 A CN201110093092 A CN 201110093092A CN 102129472 B CN102129472 B CN 102129472B
- Authority
- CN
- China
- Prior art keywords
- data
- hot spot
- spot data
- control unit
- formation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a construction method for a high-efficiency hybrid storage structure of a semantic-orient search engine. The high-efficiency hybrid storage structure comprises an internal memory, a special controller 1, a solid state disk, a special controller 2 and a hybrid disk, wherein the special controller 2 is located between the solid state disk and the hybrid disk, and the special controller 2 is used for compressing data, decompressing data and updating a hot data region in the solid state disk; the solid state disk is used for storing hot queues which cannot be completely stored in the internal memory; when a user submits an inquiry request, the special controller 1 reads the hot data inquired by the user from the hot queues of the solid state disk to the internal memory, and when the hot queues are needed to be generated, the special controller 1 reads the counting information of the hot data in the internal memory to the solid state disk; the invention provides the construction method for the high-efficiency hybrid storage structure of the semantic-orient search engine, and the inquiry efficiency of the user can be improved by using the storage structure constructed by the method in an internet system.
Description
(1), technical field: the present invention relates to a kind of construction method of storage organization, particularly relate to a kind of construction method of the high efficient mixed storage organization towards the semantic search engine.
(2), background technology: search engine is most important one type of application in the internet; Search engine is handled general media informations such as the magnanimity info web of internet generation every day, dark netting index certificate, audio frequency and video; And organize efficiently and index, so that one-stop, personalized, intelligentized concurrent retrieval service to be provided to mass user.The application characteristic of search engine is mass data storage, mass data processing, the concurrent visit of mass user, the accuracy and the real-time of inquiring about is all had higher requirements.
Along with the rapid expansion of Internet user and webpage quantity, search engine system is increasing at the pressure of aspects such as calculating, I/O and storage.Represent the traditional search engines of info web based on the keyword vector structureization, need mate and mark, return the nearest search result < keyword, document, speech rate>tlv triple.The advantage of this way of search is to be convenient to realize that seek rate is fast, and recall ratio is high.But the simple coupling to keyword causes traditional search engines to have precision ratio on the low side, can't reflect shortcomings such as article domain knowledge.User's single query requests will cause search engine system inside to produce repeatedly accessing operation.Be accompanied by increasing rapidly of search engine index webpage quantity; Existing hardware memory access ability can't satisfy is all inquiring about in the index data in real time, and existing solution is to improve the access efficiency of index data through distributed storage mode and hierarchical cache technology.
On the other hand, cloud data mining technology and cloud search engine technique are also bred in the continuous development of cloud computing environment, satisfying user's under the cloud environment new services demand, thereby make an important application that becomes cloud environment towards the semantic search of cloud.
Search based on document semantic has precision ratio preferably, and existing semantic indexing data are to obtain through the secondary calculating to traditional index data, and the storage of the semantic indexing data of magnanimity document is not also had good solution.
(3), summary of the invention:
The technical matters that the present invention will solve is: overcome the defective of prior art, a kind of construction method of the high efficient mixed storage organization towards the semantic search engine is provided, the storage organization that in internet system, uses this method to make up can improve user's search efficiency.
Technical scheme of the present invention:
A kind of construction method of the high efficient mixed storage organization towards the semantic search engine, this high efficient mixed storage organization contains internal memory, nonshared control unit 1, solid state hard disc, nonshared control unit 2 and hybrid hard disk; General data is placed on the hybrid hard disk; Hybrid hard disk contains a flash memories; Flash memories possesses the simple count function to the data of frequent visit; Be used to write down the data of frequent visit in the flash memories, often the data of visit belong to the hot spot data district, and the intermediate variable that system generates is contained in the hot spot data district; Nonshared control unit 2 is between solid state hard disc and hybrid hard disk; The effect of nonshared control unit 2 is that the hot spot data district in packed data, decompressed data and the solid state hard disc upgrades; General data on 2 pairs of hybrid hard disks of nonshared control unit conducts interviews, stores; And hot spot data found; The visit frequency of nonshared control unit 2 through recording data blocks with in the frequently-used data often the data of visit be placed into the hot spot data district in the solid state hard disc, and this hot spot data district is upgraded according to the update cycle of setting; Solid state hard disc is as the transition between speed, data storage requirement, memory size and the cost; Deposit those focus formations that completely to leave in the focus formation in the internal memory in the solid state hard disc; When the submit queries request, nonshared control unit 1 with the hot spot data of user inquiring by reading in the internal memory in the focus formation in the solid state hard disc, and with hot spot data count, compressed; When needs generate the focus formation; Nonshared control unit 1 reads the count information of the hot spot data in the internal memory in the solid state hard disc, gathers the ordering in the temporal data formation in the focus formation, upgrades the focus formation after being resequenced by nonshared control unit 2.
Confidence level according to hot spot data is safeguarded a focus formation; The focus formation contains hot spot data formation and temporal data formation; The hot spot data that the hot spot data queue for storing is current, the data that the backstage pushes are deposited the temporal data formation on the solid state hard disc from hybrid hard disk by nonshared control unit 2 earlier, again according to the time window of setting; Calculate by the confidence level in 2 pairs of hot spot data formations of nonshared control unit and the temporal data formation at set intervals; And the formation of ordering renewal focus, nonshared control unit 2 also carries out cluster through the off-line cluster to newly-generated index data, obtains the degree of correlation than a higher m association area; Each association area is carried out the storage of branch bucket, reduce the calculated amount of inquiry.
When the submit queries request, this query requests arrives the inquiry proxy module in the search engine system, and the inquiry proxy module is at first inquired about the focus formation; If inquire the focus formation, then the focus formation is accessed from its memory block, if do not inquire the focus formation; The search index module of just query requests being delivered in the search engine system is inquired about; The search index module is at first returned Query Result after inquiring the focus formation, then Query Result is delivered to the name server of storage data directory metadata; Read document and return to the inquiry proxy module by the name service administration module that moves on the name server; Upgrade the index number of times simultaneously, be pushed to the memory block of hot spot data to result of calculation, and upgrade the focus formation; Upgrading the focus formation is to carry out once at a distance from a period of time, and main operation is the confidence level to the ordering of hot spot data and calculating hot spot data.
The confidence level of hot spot data is described through five-tuple, and five-tuple contains freshness, access times, visiting frequency, field authority's scoring and time threshold; The time length that freshness self-explanatory characters' part is retained in system, total number of times that access times record document data is visited, visiting frequency is the access times in the time window; Time window is set according to visit capacity by the system that numerical ability allows; Field authority scoring is centesimal system, document is marked through the manual work mark by the professional, because of document size bigger; Field authority's scoring is only applicable to paroxysmal hot spot data; Time threshold for the hot spot data of overtime threshold value, need upgrade its confidence level for the life cycle of the hot spot data of setting again.
If the initial score of the time keeping freshness that takes place according to incident, the time of in system, retaining along with incident increases, and freshness is exponential damping.
Beneficial effect of the present invention:
1, the present invention adopts multistage storage organization, and has defined the memory access agreement and the control strategy of data, can realize effectively that the classification storage of traditional index data, credible hot spot data are found and the compression memory of semantic indexing data.The present invention adopts the mode that hot spot data broadcasting pushes, the index compression coding combines with the frequently-used data cache queries, effectively less data move number of times, the reduction system is to the access price of storer.Adopt branch field, sectional lists, branch rank to organize to index data; The support hot spot data is found; Data association is found; Through safeguarding the access bandwidth and the memory access mode of hot spot data formation and access queue preference strategy dynamic adjusting data, be combined in the propelling movement mode on the data topology network, improve user's search efficiency.
2, the present invention cooperates corresponding with it data store organisation through introducing multiple memory capacity and I/O speed different storage devices such as hybrid hard disk, solid state hard disc and internal memory.Exchanges data through between nonshared control unit 1 and nonshared control unit 2 pairs of hybrid hard disks, solid state hard disc and internal memories is carried out record; And through introducing self-defined weight system and clustering processing; Generate and safeguard the hot spot data formation that is fit to semantic indexing; Dwindle the query context of data on the one hand, promote the accuracy of data temperature tolerance on the other hand, thereby reach the purpose of efficient access semantic indexing data.
(4), description of drawings:
Fig. 1 is the structural representation towards the high efficient mixed storage organization of semantic search engine;
Fig. 2 is the synoptic diagram of safeguarding of focus formation;
Fig. 3 is the structural representation of ranked data formation;
Fig. 4 is to the memory access process flow diagram towards the high efficient mixed storage organization of semantic search engine.
(5), embodiment:
Referring to Fig. 1~Fig. 4, among the figure, towards the construction method of the high efficient mixed storage organization of semantic search engine be: this high efficient mixed storage organization contains internal memory, nonshared control unit 1, solid state hard disc, nonshared control unit 2 and hybrid hard disk; General data is placed on the hybrid hard disk; Hybrid hard disk contains a flash memories; Flash memories possesses the simple count function to the data of frequent visit; Be used to write down the data of frequent visit in the flash memories, often the data of visit belong to the hot spot data district, and the intermediate variable that system generates is contained in the hot spot data district; Nonshared control unit 2 is between solid state hard disc and hybrid hard disk; The effect of nonshared control unit 2 is that the hot spot data district in packed data, decompressed data and the solid state hard disc upgrades; General data on 2 pairs of hybrid hard disks of nonshared control unit conducts interviews, stores; And hot spot data found; The visit frequency of nonshared control unit 2 through recording data blocks with in the frequently-used data often the data of visit be placed into the hot spot data district in the solid state hard disc, and this hot spot data district is upgraded according to the update cycle of setting; Solid state hard disc is as the transition between speed, data storage requirement, memory size and the cost; Deposit those focus formations that completely to leave in the focus formation in the internal memory in the solid state hard disc; When the submit queries request, nonshared control unit 1 with the hot spot data of user inquiring by reading in the internal memory in the focus formation in the solid state hard disc, and with hot spot data count, compressed; When needs generate the focus formation; Nonshared control unit 1 reads the count information of the hot spot data in the internal memory in the solid state hard disc, gathers the ordering in the temporal data formation in the focus formation, upgrades the focus formation after being resequenced by nonshared control unit 2.
Confidence level according to hot spot data is safeguarded a focus formation; The focus formation contains hot spot data formation and temporal data formation; The hot spot data that the hot spot data queue for storing is current, the data that the backstage pushes are deposited the temporal data formation on the solid state hard disc from hybrid hard disk by nonshared control unit 2 earlier, again according to the time window of setting; Calculate by the confidence level in 2 pairs of hot spot data formations of nonshared control unit and the temporal data formation at set intervals; And the formation of ordering renewal focus, nonshared control unit 2 also carries out cluster through the off-line cluster to newly-generated index data, obtains the degree of correlation than a higher m association area; Each association area is carried out the storage of branch bucket, reduce the calculated amount of inquiry.
When the submit queries request, this query requests arrives the inquiry proxy module in the search engine system, and the inquiry proxy module is at first inquired about the focus formation; If inquire the focus formation, then the focus formation is accessed from its memory block, if do not inquire the focus formation; The search index module of just query requests being delivered in the search engine system is inquired about; The search index module is at first returned Query Result after inquiring the focus formation, then Query Result is delivered to the name server of storage data directory metadata; Read document and return to the inquiry proxy module by the name service administration module that moves on the name server; Upgrade the index number of times simultaneously, be pushed to the memory block of hot spot data to result of calculation, and upgrade the focus formation; Upgrading the focus formation is to carry out once at a distance from a period of time, and main operation is the confidence level to the ordering of hot spot data and calculating hot spot data.
The confidence level of hot spot data is described through five-tuple, and five-tuple contains freshness, access times, visiting frequency, field authority's scoring and time threshold; The time length that freshness self-explanatory characters' part is retained in system, total number of times that access times record document data is visited, visiting frequency is the access times in the time window; Time window is set according to visit capacity by the system that numerical ability allows; Field authority scoring is centesimal system, document is marked through the manual work mark by the professional, because of document size bigger; Field authority's scoring is only applicable to paroxysmal hot spot data; Time threshold for the hot spot data of overtime threshold value, need upgrade its confidence level for the life cycle of the hot spot data of setting again.
If the initial score of the time keeping freshness that takes place according to incident, the time of in system, retaining along with incident increases, and freshness is exponential damping.
The maintaining method of focus formation is: existing memory device can't satisfy the real-time coupling requirement to all index datas; For reducing the cost of data access; Improve data access speed; In conjunction with 80/20 principle of user to the network data demand, we have designed the hierachical data structure among Fig. 2, and general data comprises semantic indexing data, semantic indexing gauge outfit, secondary hot spots data queue and hot spot data formation etc.According to the frequency that possibly visited, above data leave in Flash chip, solid state hard disc and the internal memory of common sector, hybrid hard disk of hybrid hard disk successively.What index datastore district 1 comprised in the index datastore district n is the semantic indexing clauses and subclauses that the branch bucket is deposited.During the request of system responses user inquiring, inquire about, can effectively reduce the average length in data-moving path by the order that hot spot data formation, secondary hot spots data queue, semantic indexing gauge outfit, the temperature of semantic indexing data are successively decreased successively.
What safeguard in the hot spot data formation is the highest semantic indexing clauses and subclauses of visited temperature in one period, sorts according to the focus confidence level of data.The method of semantic indexing data entering hot spot data formation as shown in Figure 3; After the focus confidence level of semantic indexing data reaches certain threshold value; Read in the working area through nonshared control unit; Carry out further focus confidence level accumulation, until reaching threshold value, the hot spot data formation that can participate in is next time upgraded.
Claims (4)
1. construction method towards the high efficient mixed storage organization of semantic search engine, it is characterized in that: this high efficient mixed storage organization contains internal memory, nonshared control unit 1, solid state hard disc, nonshared control unit 2 and hybrid hard disk; General data is placed on the hybrid hard disk; Hybrid hard disk contains a flash memories; Flash memories possesses the simple count function to the data of frequent visit; Be used to write down the data of frequent visit in the flash memories, often the data of visit belong to the hot spot data district, and the intermediate variable that system generates is contained in the hot spot data district; Nonshared control unit 2 is between solid state hard disc and hybrid hard disk; The effect of nonshared control unit 2 is that the hot spot data district in packed data, decompressed data and the solid state hard disc upgrades; General data on 2 pairs of hybrid hard disks of nonshared control unit conducts interviews, stores; And hot spot data found; The visit frequency of nonshared control unit 2 through recording data blocks with in the frequently-used data often the data of visit be placed into the hot spot data district in the solid state hard disc, and this hot spot data district is upgraded according to the update cycle of setting; Deposit those focus formations that completely to leave in the focus formation in the internal memory in the solid state hard disc; When the submit queries request; Nonshared control unit 1 with the hot spot data of user inquiring by reading in the internal memory in the focus formation in the solid state hard disc; And with hot spot data count, compressed, when needs generated the focus formation, nonshared control unit 1 read the count information of the hot spot data in the internal memory in the solid state hard disc; Gather the ordering in the temporal data formation in the focus formation, upgrade the focus formation after resequencing by nonshared control unit 2.
2. the construction method of the high efficient mixed storage organization towards the semantic search engine according to claim 1; It is characterized in that: the confidence level according to said hot spot data is safeguarded a focus formation, and the focus formation contains hot spot data formation and temporal data formation, the hot spot data that the hot spot data queue for storing is current; The data that the backstage pushes are deposited the temporal data formation on the solid state hard disc from hybrid hard disk by nonshared control unit 2 earlier; According to the time window of setting, calculate by the confidence level in 2 pairs of hot spot data formations of nonshared control unit and the temporal data formation at set intervals again, and the focus formation is upgraded in ordering; Nonshared control unit 2 also carries out cluster through the off-line cluster to newly-generated index data; Obtain the degree of correlation than a higher m association area, each association area is carried out the storage of branch bucket, reduce the calculated amount of inquiry.
3. the construction method of the high efficient mixed storage organization towards the semantic search engine according to claim 2 is characterized in that: when the submit queries request, this query requests arrives the inquiry proxy module in search engine system; The inquiry proxy module is at first inquired about the focus formation; If inquire the focus formation, then the focus formation is accessed from its memory block, if do not inquire the focus formation; The search index module of just query requests being delivered in the search engine system is inquired about; The search index module is at first returned Query Result after inquiring the focus formation, then Query Result is delivered to the name server of storage data directory metadata; Read document and return to the inquiry proxy module by the name service administration module that moves on the name server; Upgrade the index number of times simultaneously, be pushed to the memory block of hot spot data to result of calculation, and upgrade the focus formation; Upgrading the focus formation is to carry out once at a distance from a period of time, and main operation is the confidence level to the ordering of hot spot data and calculating hot spot data.
4. the construction method of the high efficient mixed storage organization towards the semantic search engine according to claim 2; It is characterized in that: the confidence level of said hot spot data is described through five-tuple, and five-tuple contains freshness, access times, visiting frequency, field authority's scoring and time threshold; The time length that freshness self-explanatory characters' part is retained in system, total number of times that access times record document data is visited, visiting frequency is the access times in the time window; Time window is set according to visit capacity by the system that numerical ability allows; The authoritative scoring in field is centesimal system, document is marked through the manual work mark by the professional, and time threshold is the life cycle of the hot spot data of setting; For the hot spot data of overtime threshold value, need upgrade its confidence level again.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN 201110093092 CN102129472B (en) | 2011-04-14 | 2011-04-14 | Construction method for high-efficiency hybrid storage structure of semantic-orient search engine |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN 201110093092 CN102129472B (en) | 2011-04-14 | 2011-04-14 | Construction method for high-efficiency hybrid storage structure of semantic-orient search engine |
Publications (2)
Publication Number | Publication Date |
---|---|
CN102129472A CN102129472A (en) | 2011-07-20 |
CN102129472B true CN102129472B (en) | 2012-12-19 |
Family
ID=44267554
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN 201110093092 Active CN102129472B (en) | 2011-04-14 | 2011-04-14 | Construction method for high-efficiency hybrid storage structure of semantic-orient search engine |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN102129472B (en) |
Families Citing this family (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102388374A (en) * | 2011-09-28 | 2012-03-21 | 华为技术有限公司 | Method and device for data storage |
CN103092867B (en) * | 2011-11-03 | 2017-02-01 | ***通信集团甘肃有限公司 | Method and system for managing data, and data analyzing device |
CN102609360B (en) * | 2012-01-12 | 2015-03-25 | 华为技术有限公司 | Data processing method, data processing device and data processing system |
CN104571954B (en) * | 2014-12-26 | 2018-05-18 | 杭州华为数字技术有限公司 | A kind of date storage method and device |
CN105808150B (en) * | 2014-12-29 | 2018-09-07 | 先智云端数据股份有限公司 | Solid state disk fast taking system for hybrid storage facilities |
CN105551045B (en) * | 2015-12-16 | 2019-07-26 | 联想(北京)有限公司 | A kind of temperature calculation method and electronic equipment |
CN106095832B (en) * | 2016-06-01 | 2020-02-18 | 东软集团股份有限公司 | Distributed parallel data processing method and device |
CN110019985B (en) * | 2017-12-29 | 2021-09-24 | 阿里巴巴(中国)有限公司 | Index file establishing and inquiring methods and devices |
CN108875058A (en) * | 2018-06-28 | 2018-11-23 | 中国建设银行股份有限公司 | file data management method and device |
CN109656978A (en) * | 2018-12-24 | 2019-04-19 | 泰华智慧产业集团股份有限公司 | The optimization method of near real-time search service |
CN111610936B (en) * | 2020-05-25 | 2023-04-14 | 广州市百果园信息技术有限公司 | Object storage platform, object aggregation method and device and server |
CN112076464B (en) * | 2020-09-04 | 2022-06-21 | 腾讯科技(深圳)有限公司 | Data request processing method and device, computer equipment and storage medium |
US11494119B2 (en) * | 2020-09-10 | 2022-11-08 | Micron Technology, Inc. | Memory searching component |
CN113284573A (en) * | 2021-06-02 | 2021-08-20 | 山东健康医疗大数据有限公司 | Method and device for searching document database |
CN117331514B (en) * | 2023-12-01 | 2024-02-23 | 超越科技股份有限公司 | Solid-state disk data compression system and method based on region division |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN1799050A (en) * | 2003-05-01 | 2006-07-05 | 微软公司 | Concept network |
CN1851705A (en) * | 2006-05-30 | 2006-10-25 | 南京大学 | Body-bused subject type network reptile system configuration method |
CN101441662A (en) * | 2008-11-28 | 2009-05-27 | 北京交通大学 | Topic information acquisition method based on network topology |
-
2011
- 2011-04-14 CN CN 201110093092 patent/CN102129472B/en active Active
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN1799050A (en) * | 2003-05-01 | 2006-07-05 | 微软公司 | Concept network |
CN1851705A (en) * | 2006-05-30 | 2006-10-25 | 南京大学 | Body-bused subject type network reptile system configuration method |
CN101441662A (en) * | 2008-11-28 | 2009-05-27 | 北京交通大学 | Topic information acquisition method based on network topology |
Also Published As
Publication number | Publication date |
---|---|
CN102129472A (en) | 2011-07-20 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN102129472B (en) | Construction method for high-efficiency hybrid storage structure of semantic-orient search engine | |
CN102521405B (en) | Massive structured data storage and query methods and systems supporting high-speed loading | |
CN102364474B (en) | Metadata storage system for cluster file system and metadata management method | |
CN103019958B (en) | Usage data attribute manages the method for the data in solid-state memory | |
CN104850358B (en) | A kind of magneto-optic electricity mixing storage system and its data acquisition and storage method | |
CN105677826A (en) | Resource management method for massive unstructured data | |
CN102521406A (en) | Distributed query method and system for complex task of querying massive structured data | |
CN102375853A (en) | Distributed database system, method for building index therein and query method | |
CN103336849A (en) | Method and device for increasing retrieval speed in database retrieval system | |
CN103176754A (en) | Reading and storing method for massive amounts of small files | |
CN106528451B (en) | The cloud storage frame and construction method prefetched for the L2 cache of small documents | |
EP2765522B1 (en) | Method and device for data pre-heating | |
CA2871313A1 (en) | Method and system for managing power grid data | |
CN104111898A (en) | Hybrid storage system based on multidimensional data similarity and data management method | |
CN107368608A (en) | The HDFS small documents buffer memory management methods of algorithm are replaced based on ARC | |
CN102467572A (en) | Data block inquiring method for supporting data de-duplication program | |
CN109982104A (en) | The video of mobile awareness prefetches and caching Replacement Decision method in a kind of mobile edge calculations | |
CN104657461A (en) | File system metadata search caching method based on internal memory and SSD (Solid State Disk) collaboration | |
CN102314397A (en) | Method for processing cache data block | |
CN111159176A (en) | Method and system for storing and reading mass stream data | |
CN109388341A (en) | A kind of system storage optimization method based on Device Mapper | |
CN102737068A (en) | Method and equipment for performing cache management on retrieval data | |
CN101788887A (en) | System and method of I/O cache stream based on database in disk array | |
CN104375782A (en) | Read-write solution method for ten-million-level small file data | |
CN112685337B (en) | Method for hierarchically caching read and write data in storage cluster |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
C06 | Publication | ||
PB01 | Publication | ||
C10 | Entry into substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
C14 | Grant of patent or utility model | ||
GR01 | Patent grant |