US10891340B2 - Method of and system for updating search index database - Google Patents
Method of and system for updating search index database Download PDFInfo
- Publication number
- US10891340B2 US10891340B2 US16/505,110 US201916505110A US10891340B2 US 10891340 B2 US10891340 B2 US 10891340B2 US 201916505110 A US201916505110 A US 201916505110A US 10891340 B2 US10891340 B2 US 10891340B2
- Authority
- US
- United States
- Prior art keywords
- documents
- subset
- chunks
- size
- update
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
- 238000000034 method Methods 0.000 title claims abstract description 83
- 238000012217 deletion Methods 0.000 claims abstract description 19
- 230000037430 deletion Effects 0.000 claims abstract description 19
- 230000004044 response Effects 0.000 claims description 23
- 230000003993 interaction Effects 0.000 claims description 21
- 230000008859 change Effects 0.000 claims description 5
- 238000005516 engineering process Methods 0.000 description 69
- 238000004891 communication Methods 0.000 description 29
- 230000008569 process Effects 0.000 description 16
- 230000009193 crawling Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 10
- 230000004048 modification Effects 0.000 description 8
- 238000012986 modification Methods 0.000 description 8
- 238000012545 processing Methods 0.000 description 6
- 238000004590 computer program Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 201000001997 microphthalmia with limb anomalies Diseases 0.000 description 3
- 239000013598 vector Substances 0.000 description 3
- 230000006870 function Effects 0.000 description 2
- 241000239290 Araneae Species 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- 230000000153 supplemental effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/162—Delete operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2272—Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2379—Updates performed during online database operations; commit processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/93—Document management systems
Definitions
- the present technology relates to information retrieval in general and, more specifically, to a method and a system for updating a search index database of a search engine server.
- Various global or local communication networks offer a user a vast amount of information.
- the information includes a multitude of contextual topics, such as but not limited to, news and current affairs, maps, company information, financial information and resources, traffic information, games and entertainment related information.
- Users use a variety of client devices (desktop, laptop, notebook, smartphone, tablets and the like) to have access to rich content (like images, audio, video, animation, and other multimedia content from such networks).
- a given user can access a resource on the communication network by two principle means.
- the given user can access a particular resource directly, either by typing an address of the resource (typically an URL or Universal Resource Locators or by clicking a link in an electronic message or in another web resource.
- the given user may conduct a search using a search engine to locate a resource of interest.
- the latter is particularly suitable in those circumstances, where the given user knows a topic of interest, but does not know the exact address of the resource she is interested in.
- a search engine generally operates in three phases: web crawling, indexing, and searching.
- the search engine executes a plurality of web crawlers known also as “spiders”, which are computer programs that “crawl” or browse the World Wide Web (Web) and download copies of digital documents such as web pages, which are stored in a storage medium.
- the stored digital documents are processed to extract information, and the extracted information is used for building a search index.
- the search index stores the processed information in a suitable format for allowing fast retrieval.
- the digital documents are indexed according to some or all of the terms contained in the digital document, which terms can potentially match one or more future search query terms.
- a so-called inverted index of the data collection is maintained and updated by the system, to be then used in execution of a given search query.
- the inverted index comprises a plurality of “posting lists”, where every posting list corresponds to a term and contains references to data items comprising that search term.
- a user typically enters and submits a search query in a field on the search engine
- the search engine retrieves digital documents relevant to the search terms included in the search query from the search index
- a machine learning algorithm typically ranks the digital documents based on their computed relevancy to the terms of the search query.
- the ranked digital documents are then provided to the user in a search engine results pages (SERPs).
- U.S. Patent Application Publication No. 2009/0193406 A1 published on Jul. 30, 2009 to James Charles Williams and titled “Bulk Search Index Updates” teaches performing bulk updates of a search index for an information repository.
- a batched set of update requests is run and a set of documents to be updated based on the set of requests is identified.
- a bulk update method to use is selected based on an estimate of the cost of performing the bulk update.
- a bulk update method based on updating only the indexes of the documents to be updated may be used instead of a bulk update method that involves re-indexing the full set of documents in the repository.
- U.S. Pat. No. 9,430,543 B2 granted on Aug. 30, 2016 to Walmart and titled “Incrementally updating a large key-value store” teaches methods, systems, and computer program products for incrementally updating a large key-value store that can include a computer-implemented method of updating a master database comprising refreshing a refresh set of oldest entries in the master database, merging an emergency set of entries in an emergency database into the master database, and deleting the emergency set of entries from the emergency database.
- the computer-implemented method can further comprise merging a changed set of entries in a changed database into the master database and deleting the changed set of entries from the changed database.
- Other embodiments of related methods and systems are also provided.
- the number of pages on the Web is growing exponentially, and it is estimated that there are more than tens of trillions of documents available on the Web at the moment.
- Generally, only a segment of the available documents on the Web are indexed by search engines in a search index, as a majority of the documents are in the “Deep Web” and/or are not indexed for various reasons. Nonetheless, web crawlers need to crawl and save billions of documents, where only a portion of the saved documents are processed and indexed, as processing and indexing the documents for building a search index is a computationally expensive task due to, among other factors, a trade-off between the constant growth of the number of documents on the Web, and the computational capacity that is required to process and store the documents by the search engine datacenters.
- search index must be updated frequently to add new documents that may be relevant to users of the search engine, to delete documents that may be outdated, not presented frequently or at all in response to queries submitted by users, and/or that do not receive any user interactions because of their low relevancy to queries.
- prior art approaches for updating a search index may be strenuous and resource intensive, as one or more working copies of the search index must be stored and used to provide users with search results in response to queries in real-time, while another “updated” copy of the search index must also be stored while it is updated, thus requiring maintaining at least two copies of the search index, while only one is used actively, which demands storage and computational resources.
- embodiments of the present technology are directed to a method and a system for updating a search index database.
- the system as envisaged by the developers of the present technology may allow managing the limited amount of processing power and storage that can be allocated for maintaining and updating a search index, by separating the search index into portions or “chunks”, and by selectively updating documents in the portions based on different criteria, such as a size of “active” documents in the chunks, a size of documents to add and to delete, available storage space, among other factors.
- a computer-implemented method for dynamically updating a search index of a server the search index being divided into a plurality of chunks, each chunk of the plurality of chunks including a respective portion of a plurality of documents
- the method executable by the server, the method comprising: receiving, by the server, a request to update a set of documents in the search index, the request to update the set of documents including deletion of a first subset of documents, computing, by the server, for each given chunk of a set of chunks from the plurality of chunks, each given chunk including at least one document of the first subset of documents: a respective updated active size of a respective set of active documents of the respective portion of the plurality of documents in the given chunk, the respective set of active documents not including the at least one document of the first subset of documents to delete, a respective updated total size of the respective portion of the plurality of documents in the given chunk, the respective portion of the plurality of documents not including the at least
- the request to update the set of documents in the search index further comprises: addition of a second subset of documents to the plurality of documents in the plurality of chunks.
- the updating at least the portion of the set of documents in the subset of chunks comprises: removing, by the server, from the subset of candidate chunks, the first subset of documents for deletion, and distributing, by the server, the second subset of documents in the subset of candidate chunks.
- the method further comprises, prior to the distributing the second subset of documents in the subset of candidate chunks: adding, by the server, remaining documents in the subset of candidate chunks to the second subset of documents.
- the subset of candidate chunks comprises a predetermined number of chunks having a respective lowest ratio.
- each respective document of the plurality of documents of the search index has: a respective size, a location in a respective chunk, and an identifier of the respective chunk.
- the distributing the second subset of documents in the subset of candidate chunks comprises, for a given document in the second subset of documents: updating, by the server, the identifier of the respective chunk of the given document, and updating, by the server, the location of the given document in the respective chunk.
- the set of documents to update is associated with an update size, and the update size is below a predetermined update size threshold.
- each given chunk of the plurality of chunks includes: a respective set of active documents having a respective active size, a respective set of inactive documents having a respective inactive size, and the respective portion of the plurality of documents having a respective total size.
- the method further comprises, prior to the computing: determining, by the server, if the update size is above the predetermined update size threshold, and in response to the update size being above the predetermined update size threshold: dividing, by the server, the set of documents to update into at least two subsets of documents, such that each of the at least two subsets is below the predetermined change size threshold, and performing, by the server, the computing, the duplicating, and the updating for each of the at least two subsets.
- the computing, the duplicating, and the updating is executed offline.
- the server is a search engine server
- each respective set of inactive documents in the plurality of chunks of the search index includes at least one of: documents having received no user interactions in response to any query during a predetermined period of time, and documents not having been presented in response to any query during the predetermined period of time.
- the server is connected to a storage medium, the storage medium including a first portion, and a second portion, the search index is stored in the first portion, the subset of candidate chunks is stored in the second portion, and a size limit of the first portion is below a size limit of the second portion.
- a system for dynamically updating a search index the search index being divided into a plurality of chunks, each chunk of the plurality of chunks including a respective portion of a plurality of documents
- the system comprising: a processor, a non-transitory computer-readable medium comprising instructions, the processor, upon executing the instructions, being configured to: receive a request to update a set of documents in the search index, the request to update the set of documents including deletion of a first subset of documents, compute for each given chunk of a set of chunks from the plurality of chunks, each given chunk including at least one document of the first subset of documents: a respective updated active size of a respective set of active documents of the respective portion of the plurality of documents in the given chunk, the respective set of active documents not including the at least one document of the first subset of documents to delete, a respective updated total size of the respective portion of the plurality of documents in the given chunk, the respective portion of the plurality of documents
- the request to update the set of documents in the search index further comprises: addition of a second subset of documents to the plurality of documents in the plurality of chunks.
- the processor is configured to: remove from the subset of candidate chunks, the first subset of documents for deletion, and distribute the second subset of documents in the subset of candidate chunks.
- the processor is further configured to, prior to the distributing the second subset of documents in the subset of candidate chunks: add remaining documents in the subset of candidate chunks to the second subset of documents.
- the subset of candidate chunks comprises a predetermined number of chunks having a respective lowest ratio.
- each respective document of the plurality of documents of the search index has: a respective size, a location in a respective chunk, and an identifier of the respective chunk.
- the processor is configured to, for a given document in the second subset of documents: update the identifier of the respective chunk of the given document, and update the location of the given document in the respective chunk.
- the set of documents to update is associated with an update size, and the update size is below a predetermined update size threshold.
- each given chunk of the plurality of chunks includes: a respective set of active documents having a respective active size, a respective set of inactive documents having a respective inactive size, and the respective portion of the plurality of documents having a respective total size.
- system is further configured to, prior to the computing: determine if the update size is above the predetermined update size threshold, and in response to the update size being above the predetermined update size threshold: divide the set of documents to update into at least two subsets of documents, such that each of the at least two subsets is below the predetermined change size threshold, and perform the computing, the duplicating, and the updating for each of the at least two subsets.
- the computing, the duplicating, and the updating is executed offline.
- the server is a search engine server
- each respective set of inactive documents in the plurality of chunks of the search index includes at least one of: documents having received no user interactions in response to any query during a predetermined period of time, and documents not having been presented in response to any query during the predetermined period of time
- the system is connected to a storage medium, the storage medium including a first portion, and a second portion, the search index is stored in the first portion, the subset of candidate chunks is stored in the second portion, and a size limit of the first portion is below a size limit of the second portion.
- a “server” is a computer program that is running on appropriate hardware and is capable of receiving requests (e.g. from electronic devices) over a network, and carrying out those requests, or causing those requests to be carried out.
- the hardware may be one physical computer or one physical computer system, but neither is required to be the case with respect to the present technology.
- the use of the expression a “server” is not intended to mean that every task (e.g. received instructions or requests) or any particular task will have been received, carried out, or caused to be carried out, by the same server (i.e.
- electronic device is any computer hardware that is capable of running software appropriate to the relevant task at hand.
- electronic devices include personal computers (desktops, laptops, netbooks, etc.), smartphones, and tablets, as well as network equipment such as routers, switches, and gateways.
- a device acting as an electronic device in the present context is not precluded from acting as a server to other electronic devices.
- the use of the expression “a electronic device” does not preclude multiple electronic devices being used in receiving/sending, carrying out or causing to be carried out any task or request, or the consequences of any task or request, or steps of any method described herein.
- a “database” is any structured collection of data, irrespective of its particular structure, the database management software, or the computer hardware on which the data is stored, implemented or otherwise rendered available for use.
- a database may reside on the same hardware as the process that stores or makes use of the information stored in the database or it may reside on separate hardware, such as a dedicated server or plurality of servers.
- information includes information of any nature or kind whatsoever capable of being stored in a database.
- information includes, but is not limited to audiovisual works (images, movies, sound records, presentations etc.), data (location data, numerical data, etc.), text (opinions, comments, questions, messages, etc.), documents, spreadsheets, etc.
- computer usable information storage medium is intended to include media of any nature and kind whatsoever, including RAM, ROM, disks (CD-ROMs, DVDs, floppy disks, hard drivers, etc.), USB keys, solid state-drives, tape drives, etc.
- an “indication” of an information element may be the information element itself or a pointer, reference, link, or other indirect mechanism enabling the recipient of the indication to locate a network, memory, database, or other computer-readable medium location from which the information element may be retrieved.
- an indication of a document could include the document itself (i.e. its contents), or it could be a unique document descriptor identifying a file with respect to a particular file system, or some other means of directing the recipient of the indication to a network location, memory address, database table, or other location where the file may be accessed.
- the degree of precision required in such an indication depends on the extent of any prior understanding about the interpretation to be given to information being exchanged as between the sender and the recipient of the indication. For example, if it is understood prior to a communication between a sender and a recipient that an indication of an information element will take the form of a database key for an entry in a particular table of a predetermined database containing the information element, then the sending of the database key is all that is required to effectively convey the information element to the recipient, even though the information element itself was not transmitted as between the sender and the recipient of the indication.
- first”, “second”, “third”, etc. have been used as adjectives only for the purpose of allowing for distinction between the nouns that they modify from one another, and not for the purpose of describing any particular relationship between those nouns.
- first server and “third server” is not intended to imply any particular order, type, chronology, hierarchy or ranking (for example) of/between the server, nor is their use (by itself) intended imply that any “second server” must necessarily exist in any given situation.
- reference to a “first” element and a “second” element does not preclude the two elements from being the same actual real-world element.
- a “first” server and a “second” server may be the same software and/or hardware, in other cases they may be different software and/or hardware.
- plural of search results is intended to include a set of more than one search result, where general (e.g., Web) search results and/or vertical search results are integrated together within the set of search results or on a search results page.
- general search results may be integrated into general (e.g., Web) search results within a search results page, or vice-versa, i.e., general search results may be integrated into vertical search results within a search results page.
- general search results and vertical search results may be aggregated and ranked relative to each other.
- a plurality of search results may include only general search results or only vertical search results, e.g., search results from a particular vertical domain of interest.
- Implementations of the present technology each have at least one of the above-mentioned object and/or aspects, but do not necessarily have all of them. It should be understood that some aspects of the present technology that have resulted from attempting to attain the above-mentioned object may not satisfy this object and/or may satisfy other objects not specifically recited herein.
- FIG. 1 depicts a schematic diagram of a system implemented in accordance with non-limiting embodiments of the present technology.
- FIG. 2 depicts a schematic diagram of a structure of a search index implemented by system of FIG. 1 in accordance with non-limiting embodiments of the present technology.
- FIG. 3 depicts a schematic diagram an update process of the search index of FIG. 2 , the update process being executed by the system of FIG. 1 in accordance with non-limiting embodiments of the present technology.
- FIG. 4 depicts a schematic diagram of another stage of the update process of FIG. 3 , the update process being executed by the system of FIG. 1 in accordance with non-limiting embodiments of the present technology.
- FIG. 5 depicts a schematic diagram of another stage of the update process of FIG. 3 , the update process being executed by the system of FIG. 1 in accordance with non-limiting embodiments of the present technology.
- FIG. 6 depicts a flowchart of a method for updating the search index of FIG. 2 , the method being executed by the system of FIG. 1 in accordance with non-limiting embodiments of the present technology.
- any functional block labeled as a “processor” or a “graphics processing unit” may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software.
- the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared.
- the processor may be a general purpose processor, such as a central processing unit (CPU) or a processor dedicated to a specific purpose, such as a graphics processing unit (GPU).
- CPU central processing unit
- GPU graphics processing unit
- processor or “controller” should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, network processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), read-only memory (ROM) for storing software, random access memory (RAM), and non-volatile storage.
- DSP digital signal processor
- ASIC application specific integrated circuit
- FPGA field programmable gate array
- ROM read-only memory
- RAM random access memory
- non-volatile storage Other hardware, conventional and/or custom, may also be included.
- the system 100 comprises a first client device 102 , a second client device 104 , a third client device 106 , and a fourth client device 108 coupled to a communications network 110 via a respective communication link 115 (only one numbered in FIG. 1 ).
- the system 100 comprises an indexing server 120 , a search engine server 140 , and a tracking server 160 coupled to the communications network 110 via their respective communication link 115 (only one numbered in FIG. 1 ).
- the indexing server 120 , the search engine server 140 , and the tracking server 160 may be implemented as a single server.
- the first client device 102 may be implemented as a smartphone
- the second client device 104 may be implemented as a laptop
- the third client device 106 may be implemented as a smartphone
- the fourth client device 108 may be implemented as a tablet.
- the communications network 110 can be implemented as the Internet. In other embodiments of the present technology, the communications network 110 can be implemented differently, such as any wide-area communications network, local-area communications network, a private communications network and the like.
- the given communication link 115 is implemented is not particularly limited and will depend on how the associated one of the first client device 102 , the second client device 104 , the third client device 106 and the fourth client device 108 are implemented.
- the associated one of the communication link 115 can be implemented as a wireless communication link (such as but not limited to, a 3G communications network link, a 4G communications network link, a Wireless Fidelity, or WiFi® for short, Bluetooth® and the like).
- the associated communication link 115 can be either wireless (such as the Wireless Fidelity, or WiFi® for short, Bluetooth® or the like) or wired (such as an Ethernet based connection).
- first client device 102 the second client device 104 , the third client device 106 , the fourth client device 108 , the communication link 115 and the communications network 110 are provided for illustration purposes only. As such, those skilled in the art will easily appreciate other specific implementational details for the first client device 102 , the second client device 104 , the third client device 106 , the fourth client device 108 and the communication link 115 and the communications network 110 . As such, by no means, examples provided herein above are meant to limit the scope of the present technology.
- first client device 102 the second client device 104 , the third client device 106 and the fourth client device 108 are illustrated, it is contemplated that any number of client devices could be connected to the system 100 . It is further contemplated that in some implementations, the number of client devices included in the system 100 could number in the tens or hundreds of thousands.
- the indexing server 120 can be implemented as a conventional computer server.
- the indexing server 120 can be implemented as a DellTM PowerEdgeTM Server running the MicrosoftTM Windows ServerTM operating system. Needless to say, the indexing server 120 can be implemented in any other suitable hardware and/or software and/or firmware or a combination thereof. In the depicted non-limiting embodiment of present technology, the indexing server 120 is a single server. In alternative non-limiting embodiments of the present technology, the functionality of the indexing server 120 may be distributed and may be implemented via multiple servers. In other embodiments, the functionality of the indexing server 120 may be implemented completely or partially by the search engine server 140 . In some embodiments of the present technology, indexing server 120 is under control and/or management of a search engine operator. Alternatively, the indexing server 120 can be under control and/or management of another service provider.
- the purpose of the indexing server 120 is to (i) crawl the Web and retrieve documents, such as a plurality of documents 117 ; (ii) process the plurality of documents 117 ; and (iii) index at least a portion of the processed plurality of documents 117 in a search index database 146 of the search engine server 140 ; and (iv) update the search index database 146 of the search engine server 140 .
- the indexing server 120 executes a plurality of crawlers 122 , a parser 130 , a first MLA 132 , and is connected to a crawling database 136 , and a preindexing database 138 .
- the plurality of documents 117 may be hosted by various computer systems accessible over the communication network 110 , for example.
- the nature of the plurality of documents 117 is not particularly limited.
- the plurality of documents 117 may also be referred to as “a plurality of web pages”, “web pages”, “web documents” or simply “documents”.
- a given one of the plurality of documents 117 may be any form of structured digital information that can be retrieved or accessed via a corresponding Universal Resource Locator (URL), without departing from the scope of the present technology.
- URL Universal Resource Locator
- a given one of the plurality of documents 117 may contain one or more sentences.
- a given one of the plurality of documents 117 can be, for example, a web page containing text and/or images (such as, for example, a news article recently published and related to some breaking news).
- Another given one of the plurality of documents 117 can be, as another example, a digital version of a book (such as, for example, a digital version of “Pride and Prejudice” by Jane Austin).
- Another given one of the plurality of documents 117 can be, as another example, an article on WikipediaTM, which can be updated from time to time.
- the plurality of documents 117 may have been recently created (or updated) or otherwise may have been recently made accessible over the communication network 110 . Indeed, a very large number of web pages are created or otherwise made accessible over the communication network 110 everyday and, as such, there may be a need to index at least some of these web pages so as to provide their content to users of a given search engine.
- the plurality of documents 117 may be “fresh” web pages, such as web pages having fresh content that is likely to be updated relatively frequently (e.g., weather), while at least some others of the plurality of documents 117 may be “stagnant” web pages, such as web pages having stagnant content that is less likely to change or be changed with less frequent intervals (e.g., Wikipedia article on the Canadian constitution).
- usefulness of fresh content to users of a given search engine usually (i) peaks close to the moment of its creation and (ii) drops after some period of time.
- usefulness of stagnant content to users of the given search engine usually (i) is lower near the moment of its creation than usefulness of fresh content near the moment of its creation but (ii) is somewhat constant throughout time.
- the plurality of documents 117 may have been previously indexed, while at least some others of the plurality of documents 117 may not have been previously indexed.
- the plurality of documents 117 may comprise “new” web pages which have not been previously indexed.
- the plurality of documents 117 may comprise “old” web pages which have been previously indexed.
- the plurality of documents 117 may comprise “updated” web pages which are, in a sense, “updated” versions of old web pages, where the content of the updated version of the web page is different from the content of the old version of the web page which has been previously indexed.
- the plurality of crawlers 122 includes main crawlers 124 and express crawlers 126 .
- the main crawlers 124 are configured to crawl the plurality of documents 117 on the Web starting from a list of URLs to visit (not depicted), where hyperlinks in the documents are also identified and added to the list to visit.
- the express crawlers 126 are configured to perform express indexing of “fresh” documents (which may be included the plurality of documents 117 ) that appeared on the Web just before the crawling procedure, such that the fresh documents are also available for indexing.
- the main crawlers 124 and the express crawlers 126 also include respective schedulers (not depicted) which create schedules and orders of documents to crawl on the Web.
- the plurality of crawlers 122 including the main crawlers 124 and the express crawlers 126 generally download the plurality of documents 117 , and store the plurality of documents 117 in the crawling database 136 for further processing thereof, as will be described in more detail herein below.
- the parser 130 is generally configured to parse the plurality of documents 117 stored in the crawling database 136 and extract information therefrom, such as but not limited to index terms, and hyperlinks.
- the parser 130 generally clears markup language from the crawled documents, such that text information can be extracted and later analyzed or used directly for indexing.
- the parser 130 may further extract a plurality of features from the documents, such as features used for ranking (ranking features), URL features, click features, document features, and the like.
- the parser 130 may save the extracted information as feature vectors in the preindexing database 138 .
- the indexing server 120 may execute a feature extractor (not depicted) for extracting the features of the plurality of documents 117 from the crawling database 136 .
- the first MLA 132 is configured to select at least a portion of the plurality of documents 117 for inclusion in the search index database 146 of the search engine server 140 based on feature vectors of the plurality of documents 117 , as well as other information associated with plurality of documents 117 , the feature vectors being stored in the preindexing database 138 .
- the indexing server 120 is configured to update the search index database 146 upon receiving an indication for an update, or at predetermined intervals of time. How the indexing server 120 updates the search index database 146 will be explained in more detail herein below.
- the search engine server 140 can be implemented as a conventional computer server.
- the search engine server 140 can be implemented as a DellTM PowerEdgeTM Server running the MicrosoftTM Windows ServerTM operating system.
- the search engine server 140 can be implemented in any other suitable hardware and/or software and/or firmware or a combination thereof.
- the search engine server 140 is a single server.
- the functionality of the search engine server 140 may be distributed and may be implemented via multiple servers.
- the search engine server 140 is under control and/or management of a search engine operator.
- the search engine server 140 can be under control and/or management of a service provider.
- the purpose of the search engine server 140 is to (i) execute searches; (ii) execute analysis of search results and perform ranking of search results in response to a search request; (iii) group results and compile the search engine result page (SERP) to be outputted to an electronic device (such as one of the first client device 102 , the second client device 104 , the third client device 106 and the fourth client device 108 ), the electronic device having been used to submit the search request that resulted in the SERP.
- an electronic device such as one of the first client device 102 , the second client device 104 , the third client device 106 and the fourth client device 108
- the search engine server 140 may maintains the search index database 146 .
- the search engine server 140 is configured to perform general domain searches, as is known to those skilled in the art.
- a general domain search is not confined to search a specific category of results but is able to provide all results that best match the search query.
- Such a general (category-independent) search by the search engine server 140 may return search results that include non-category specific digital content as well as category specific digital content, such as images, videos, news, shopping, blogs, books, places, discussions, recipes, tickets, biographical information, patents, stocks, timelines, etc., and other digital content that is closely related and directed toward a certain type of digital content.
- a general domain search may be a WWW search.
- a search performed in a general domain generates a “general search result” or “general search result item.”
- Such general search results are also referred to herein as “web results.”
- a web result includes a website link and a snippet that summarizes content of the website.
- a user may select a website link of a web result to navigate to the webpage related to the user search query.
- the terms “general search result” and “general search result item” are used interchangeably herein, as are the terms “search result” and “search result item.”
- the search engine server 140 is also configured to perform vertical domain searches.
- a vertical domain may be an information domain containing specialized content, such as content of a single type (e.g., media type, genre of content, topicality, etc.).
- a vertical domain thus includes a specific subset of a larger set of data, for example, a specific subset of web data.
- a vertical domain may include specific information such as images, videos, news, shopping, blogs, books, places, discussions, recipes, tickets, biographical information, patents, stocks, timelines, etc.
- a search performed in a vertical domain generates a “vertical search result” or a “vertical search result item”. Such vertical search results are also referred to herein as “verticals” and “vertical results.”
- the terms “vertical search result” and “vertical search result item” are used interchangeably herein.
- the search engine server 140 is configured to generate a ranked search results list, including the results from the general domain search and the vertical domain search.
- the search engine server may execute one or more MLAs 142 for ranking documents to generate the ranked search results list.
- the one or more MLAs 142 may rank the documents by relevancy to the user-submitted search query based on some or all of: (i) how popular a given search query is in searches; (ii) how many results have been returned; (iii) whether the search query contains any determinative terms (such as “images”, “movies”, “weather” or the like), (iv) how often a particular search query is typically used with determinative terms by other users; and (v) how often other uses performing a similar search have selected a particular resource or a particular vertical search results when results were presented using the SERP.
- determinative terms such as “images”, “movies”, “weather” or the like
- the search engine server 140 can thus calculate and assign a relevance score (based on, but not limited to, the different criteria listed above) to each search result obtained in response to a user-submitted search query and generate a SERP, where search results are ranked according to their respective relevance scores.
- the search engine server 140 typically maintains the above-mentioned search index database 146 .
- search index database 146 index documents (or indication of documents), such as, but not limited to, web pages, images, videos, Portable Document Format (PDF) documents, Microsoft WordTM documents, Microsoft PowerPointTM documents, that have been, as an non-limiting example, selected by the first MLA 132 .
- search index database 146 is maintained in a form of posting lists.
- the search engine server 140 analyzes the search index database 146 and retrieves documents that contain terms of the query, and ranks them according to a ranking algorithm.
- the tracking server 160 can be implemented as a conventional computer server.
- the tracking server 160 can be implemented as a DellTM PowerEdgeTM Server running the MicrosoftTM Windows ServerTM operating system.
- the tracking server 160 can be implemented in any other suitable hardware and/or software and/or firmware or a combination thereof.
- the tracking server 160 is a single server.
- the functionality of the tracking server 160 may be distributed and may be implemented via multiple servers.
- the functionality of the tracking server 160 may be implemented completely or partially by the search engine server 140 .
- the tracking server 160 is under control and/or management of a search engine operator.
- the tracking server 160 can be under control and/or management of another service provider.
- the purpose of the tracking server 160 is to track user interactions with the search engine server 140 , such as search queries and terms entered by users, and documents subsequently accessed by the users (such as users of the first client device 102 , the second client device 104 , the third client device 106 , and the fourth client device 108 ).
- the tracking server 160 may track user interactions (such as, for example, click-through data) when users perform general domain searches and vertical domain searches on the search engine server 140 .
- Non-limiting examples of user interactions tracked or computed by the tracking server 160 include:
- the tracking server 160 may save the tracked user interactions in a tracking database 162 .
- the tracking database 162 includes a search query log 164 , and a user interaction log 166 .
- the purpose of the search query log 164 is to maintain a log of searches submitted on the search engine server 140 . More specifically, the search query log 164 may maintain a list of queries, each respective query of the list having respective search terms, the associated documents that were listed by the search engine server 140 in response to the respective search query, a number of submissions over time of the respective query (referred to as a frequency of past use herein below), and may also contain a list of users (or group of users) identified by anonymous IDs (or without an ID altogether) and the respective documents they have clicked on after submitting the respective search query. In some embodiments, the search query log 164 may be updated every time a new search is performed on the search engine server 140 . In other embodiments, the search query log 164 may be updated at predetermined times. In some embodiments, there may be a plurality of copies of the search query log 164 , each corresponding to the search query log 164 at different points in time.
- the manner in which the user interaction log 166 is structured is not limited.
- the user interaction log 166 can be linked to the search query log 164 , and can list user-interaction parameters as tracked by the tracking server 160 after a user has submitted a search query and clicked on one or more documents in a SERP on the search engine server 140 .
- the user interaction log 166 may maintain: (i) a reference to a respective document or an indication thereof, which may be identified, as a non-limiting example, by an ID number or an URL; (ii) a respective list of queries, where each respective query of the list of respective queries is associated with the respective document, and where each of the respective queries has been used by one or more users to access the respective document; and (iii) a respective plurality of user-interaction parameters per query (if the document was interacted with), which are indicative of user-interactions with the respective document by users having submitted the respective query of the list of queries.
- the user interaction log 166 may further include respective timestamps associated with the respective user-interactions, and other statistical data.
- the search query log 164 and the user interaction log 166 may be implemented as a single log.
- data from the tracking database 162 may be acquired by at least one of the search engine server 140 , and the tracking server 160 , in a synchronous manner (i.e. at predetermined time intervals), or in an asynchronous manner (e.g. upon receipt of an indication).
- the functionality of the tracking server 160 and the search engine server 140 can be implemented by a single server.
- the purpose of the search query log 164 is to maintain a log of searches submitted on the search engine server 140 . More specifically, the search query log 164 may maintain a list of queries, each respective query of the list having respective search terms, the associated documents that were listed by the search engine server 140 in response to the respective search query, a number of submissions over time of the respective query (referred to as a frequency of past use herein below), and may also contain a list of users (or group of users) identified by anonymous IDs (or without an ID altogether) and the respective documents they have clicked on after submitting the respective search query. In some embodiments, the search query log 164 may be updated every time a new search is performed on the search engine server 140 . In other embodiments, the search query log 164 may be updated at predetermined times. In some embodiments, there may be a plurality of copies of the search query log 164 , each corresponding to the search query log 164 at different points in time.
- FIG. 2 depicts a schematic diagram of a structure of the search index database 146 , the search index database 146 being depicted in accordance with non-limiting embodiments of the present technology.
- the search index database 146 includes a main portion 210 , and an update portion 280 .
- the search index database 146 has a size limit 205 , the size limit 205 being an available storage space in a storage medium of the indexing server 120 .
- the main portion 210 of the search index database 146 stores a plurality of indexed documents 250 .
- the main portion 210 is “an active” version of the search index database 146 , the main portion 210 being used by the search engine server 140 , including the one or more MLAs 142 , for retrieving documents of the plurality of indexed documents 250 to be presented as search results to users in response to queries in real-time.
- the main portion 210 of the search index database 146 has a main size limit 212 . Naturally, the main size limit 212 is below the size limit 205 of the search index database 146 .
- the update portion 280 of the search index database 146 is configured to be used for updating documents, such as the plurality of indexed documents 250 , included in the main portion 210 of the search index database 146 .
- “updating” a document in the main portion 210 of the search index database 146 may include: deletion of a given indexed document 260 of the plurality of indexed documents 250 , addition of a new document to the plurality of indexed documents 250 , and modification of the given indexed document 260 in the plurality of indexed documents 250 .
- the update portion 280 may be a “temporary” storage used for updating documents in the main portion 210 of the search index database 146 .
- the update portion 280 has an update size limit 282 , the update size limit 282 being equal to y, the update size limit 282 being below the main size limit 212 .
- the update size limit 282 may be 20% of the main size limit 212 or any other suitable number selected by the operator of the search index database 146 . How the update size limit 282 y is determined will be explained in more detail herein below.
- the update portion 280 may be located in a volatile memory of the indexing server 120 . How the update portion 280 is used to update documents in the main portion 210 will be explained in more detail herein below.
- a number of the plurality of indexed documents 250 in the main portion 210 of the search index database 146 is constant, i.e. the number of documents in the plurality of indexed documents 250 , after addition of new documents and deletion of documents is about the same.
- the number of the plurality of indexed documents 250 is constant due to the size limit 205 of the search index database 146 of the indexing server 120 .
- additional storage space may be added periodically to the indexing server 120 , and consequently the size limit 205 of the search index database 146 may increase, and the number of the plurality of indexed documents 250 may also increase.
- the main portion 210 is divided into a plurality of portions or chunks 220 , the plurality of chunks 220 including a predetermined number of chunks 221 , which may be expressed as n. How the predetermined number of chunks 221 of the plurality of chunks 220 is determined will be explained in more detail herein below.
- the main portion 210 of the search index database 146 includes the plurality of indexed documents 250 , the plurality of indexed documents 250 having been divided among the plurality of chunks 220 .
- a given chunk 222 of the plurality of chunks 220 includes a respective portion of indexed documents 252 of the plurality of indexed documents 250 .
- the given chunk 222 has a respective total size 254 , the respective total size 254 being a sum of individual size of indexed documents in the respective portion of indexed documents 250 .
- the respective total size 254 of the given chunk 222 may be expressed as: s i (1)
- s is the respective total size 254
- i is the chunk identifier 224 of the given chunk 222 .
- the given indexed document 260 in the respective portion of indexed documents 252 of the given chunk 222 has:
- the respective document information 268 includes information such as, but not limited to:
- the respective document information 268 may be acquired at least partially from the search query log 164 , and/or the user interaction log 166 at predetermined intervals of time and stored in the search index database 146 , or may be acquired as a non-limiting example when receiving a request for an update and may be stored in the search index database 146 .
- the indexing server 120 may maintain a list 215 of the plurality of indexed documents 250 , the list 215 including, for each given indexed document 260 of the plurality of indexed documents 250 , the respective document identifier 262 , the respective chunk identifier 224 , the respective document location 266 and the respective document information 268 .
- the list 215 may “map” each given indexed document 260 to the given chunk 222 , and include all information associated with the given chunk 222 , and the given indexed document 260 .
- the respective portion of indexed documents 252 in the given chunk 222 includes a respective set of active documents 230 , and a respective set of inactive documents 240 .
- Each given indexed document 260 of the given chunk 222 is either part of the set of active documents 230 or the respective set of inactive documents 240 .
- the indexing server 120 may determine that the given indexed document 260 is part of the respective set of active documents 230 , or the respective set of inactive documents 240 based on the respective document information 268 .
- the indexing server 120 may “scan” at least a portion of the plurality of chunks 220 to determine if, for the given chunk 222 , each given indexed document 260 is part of the respective set of active documents 230 , or the respective set of inactive documents 240 , based on the respective document information 268 .
- the indexing server 120 may scan at least a portion of the plurality of chunks 220 at predetermined intervals of time, or upon receiving an indication, such as a request for updating one or more documents in the plurality of chunks 220 of the search index database 146 .
- each given chunk 222 may maintain information about the respective set of active documents 230 , and the respective set of inactive documents 240 in metafields (not depicted) of the given chunk 222 .
- the respective set of active documents 230 includes documents that are considered “useful” to the search engine server 140 .
- the indexing server 120 may determine that a document is useful to the search engine server 140 based on the respective document information 268 .
- the given indexed document 260 may be in the respective set of active document 230 if the respective document information 268 of the given indexed document 260 satisfies one or more of the following conditions:
- the respective set of active documents 230 has a respective active size 232 , the respective active size 232 being a sum of the respective sizes 264 of documents in the respective set of active documents 230 .
- the respective active size 232 of the respective set of active documents 230 of the given chunk 222 may be expressed as: u i (2)
- u is the respective active size 232 of the respective set of active documents 230
- i is the chunk identifier 224 of the given chunk 222 .
- the respective set of inactive documents 240 includes documents that are considered inactive or not useful to the search engine server 140 .
- the indexing server 120 may determine that a document is inactive and/or not useful to the search engine server 140 based on the respective document information 268 .
- the given indexed document 260 may be in the respective set of inactive documents 240 if the respective document information 268 of the given indexed document 260 satisfies one or more of the following conditions:
- the respective set of inactive documents 240 has a respective inactive size 242 , the respective inactive size 242 being a difference between the total size 254 and the respective active size 232 which may be expressed as: s i ⁇ u i (3)
- s is the respective total size 254
- u is the respective active size 232
- i is the chunk identifier 224 of the given chunk 222 .
- the given chunk 222 may only have a respective set of active documents 230 , or a respective set of inactive documents 240 .
- FIG. 2 depict schematic diagrams of an update process 300 of the search index database 146 in accordance with non-limiting embodiments of the present technology.
- the request 310 to update the set of documents 320 may be received by the indexing server 120 .
- the request 310 may be transmitted by the search engine server 140 at predetermined intervals of time.
- the request 310 to update the set of documents 320 may be received after completion of a crawling procedure by the plurality of crawlers 122 , where the request 310 may include newly crawled documents, such as newly crawled documents from the plurality of digital documents 117 .
- the request 310 to update the set of documents 320 may be received after the crawling procedure executed by the plurality of crawlers 122 determines that some documents have been updated, deleted or new document have appeared.
- the request 310 to update the set of documents 320 generally includes: deletion of a first subset of documents 330 from the search index database 146 , and addition of a second subset of documents 340 to the search index database 146 .
- documents to be modified i.e. documents present in the search index database 146 that have been modified by their operator(s)
- documents that have been modified may have their “old” version deleted from the search index database 146 , and their “new” version added to the search index database 146 , and the list 215 may be updated accordingly to reflect the changes.
- the request 310 is illustrated with the first subset of documents 330 to delete, and the second subset of documents 340 to add, it should be understood that that the request 310 may only include an indication of the first subset of documents 330 and an indication the second subset of documents 340 in a form of a list of changes, and the second subset of documents 340 to add may be retrieved from the preindexing database 138 or another database (not depicted) and added to the search index database 146 at the end of the update process 300 .
- a request size 312 of the request 310 to update the set of documents 320 may generally be equal to, or below a predetermined size threshold 315 , which may be expressed as: ⁇ x (4)
- ⁇ is the request size 312 of the request 310 to update the set of documents 320
- x is the predetermined size threshold 315 .
- the request size 312 of the request 310 may be determined based on a sum of the sizes of the documents in the set of documents 320 .
- the request size 312 may be a sum of sizes documents in the second subset of documents 340 (which will be added to the search index database 146 ), minus a sum of sizes of documents in the first subset of documents 330 (which will be removed from the search index database 146 ).
- the predetermined size threshold 315 may be expressed as a percentage of the size limit 205 of the search index database 146 . As a non-limiting example, the predetermined size threshold 315 may be 3%. How the predetermined size threshold 315 x is determined will be explained in more detail herein below.
- the request 310 to update the set of documents 320 may be above the predetermined size threshold 315 , and the indexing server 120 may divide the set of documents 320 into a plurality of subsets (not depicted), such that each subset is below the predetermined size threshold 315 . The indexing server 120 may then perform the update for each of the plurality of subsets sequentially.
- a given document to be deleted 332 in the first subset of documents 330 may be included in the request 310 for one or more of the following reasons:
- Each respective document to be deleted 332 in the first subset of documents 330 has a respective document identifier 334 , a respective chunk identifier 336 , and a respective chunk location 338 , the respective chunk location 338 being indicative of a location of the respective document to be deleted 332 in the respective chunk.
- Each respective document to be added 342 included in the second subset of documents 340 may be included in the request 310 for one or more of the following reasons:
- Each respective document to be added 342 in the second subset of documents 340 has a respective document identifier 344 , and respective document information 349 .
- the indexing server 120 may identify, based on at least one of the respective document identifier 334 , the respective chunk identifier 336 , and the respective chunk location 338 of each respective document to be deleted 332 of the first subset of documents 330 , a set of chunks 360 , each respective chunk 362 of the set of chunks 360 including at least one respective document to be deleted 332 .
- the indexing server 120 may compute, for each respective chunk 362 of the set of chunks 360 , a respective updated active size 364 , a respective updated total size 366 , and a respective ratio 368 of the respective updated active size 364 to the respective updated total size 366 .
- the respective updated active size 364 is a size of the respective set of active documents 230 in the respective chunk 362 , the respective updated active size 364 not including sizes of documents of the first subset of documents 330 part of the respective chunk 362 .
- u′ i is the respective updated active size 364
- u i is the respective active size 232
- x is the predetermined size threshold 315
- i is the chunk identifier 224 of the respective chunk 362 .
- the respective updated total size 366 is a size of the respective portion of the plurality of indexed documents 252 in the respective chunk 362 , the respective updated total size 366 not including documents of the first subset of documents 330 part of the respective chunk 362 .
- s′ i the respective updated total size 366
- s i the respective total size 254
- x is the predetermined size threshold 315
- i the chunk identifier 224 of the respective chunk 362 .
- the indexing server 120 may compute, for each respective chunk 362 of the set of chunks 360 , the respective ratio 368 of the respective updated active size 364 to the respective updated total size 366 , which may be expressed as: u′ i /s′ i (7)
- u′ i is the respective updated active size 364
- s′ i the respective updated total size 366 .
- the indexing server 120 may then sort the set of chunks 360 based on the respective ratio 368 .
- the indexing server 120 may select the subset of chunks 370 including a number of k chunks from the set of chunks 360 for duplication in the update portion 280 of the search index database 146 to obtain the subset of candidate chunks 380 .
- the subset of candidate chunks 380 includes a predetermined number of candidate chunks 381 equal to k, the predetermined number of candidate chunks 381 having a minimal respective ratio 368 : u′ i /s′ i ⁇ min (8)
- the subset of candidate chunks 380 may include chunks from the set of chunks 360 having a respective ratio 368 below a predetermined threshold. How the predetermined number of candidate chunks 381 in the subset of candidate chunks 380 is determined will be explained in more detail herein below.
- the subset of candidate chunks 380 may be stored temporarily in the update portion 280 of the search index database 146 .
- the indexing server 120 may delete at least a portion of the first subset of documents 330 from the subset of candidate chunks 380 .
- “At least a portion” of the subset of documents 330 from the subset of candidate chunks 380 is designated to mean deletion of all documents included in the first subset of documents 330 that are also part of the subset of candidate chunks 380 , as some of the documents marked for deletion in the first subset of documents 330 may be in chunks from the set of chunks 360 that are not included in subset of candidate chunks 380 .
- the subset of chunks 370 may not include all chunks having documents marked for deletion in the first subset of documents 330 of the request 310 , as the chunks may have the respective ratio 368 that is not among the k lowest respective ratios 360 .
- remaining documents of the first subset of documents 330 may be added to a subsequent request for updating a set of documents (not depicted).
- the remaining documents may be marked as being inactive documents (i.e. part of the respective set of inactive documents 240 ), and may be deleted in subsequent requests for updating the set of document (not depicted) when the associated ones of the chunks are selected for updating.
- This information about documents to be deleted can be maintained in the list 215 of the plurality of indexed documents 250 (i.e. to show the document to be deleted but not yet deleted as “inactive” or simply remove it from the records of the list 215 ).
- the indexing server 120 may compute a respective updated active size (not depicted) for each respective candidate chunk 382 of the subset of candidate chunks 380 , the respective updated active size (not depicted) being a sum of a respective set of active documents (not depicted) in the respective candidate chunk 382 , which may be expressed as: u′ j (9)
- u′ is the respective updated active size (not depicted)
- j is a chunk identifier of the respective candidate chunk 382 in the k chunks of the subset of candidate chunks 380
- the indexing server 120 may add all documents included in the subset of candidate chunks 380 to the second subset of documents 340 .
- the indexing server may form a new request 390 , the new request 390 including the second subset of documents 340 , the second subset of documents 340 including each respective set of active documents (not depicted) from the subset of candidate chunks 380 .
- the new request 390 may have a new request size 395 , which may be expressed as:
- ⁇ ′ is the new request size 395 of the new request 390
- ⁇ is the request size 312 of the request 310
- u′ is the respective updated active size (not depicted) of the respective candidate chunk 382 in the subset of candidate chunks 380
- j is a respective chunk identifier of a respective candidate chunk 382 from the subset of candidate chunks 380 .
- the indexing server 120 may distribute the second subset of documents 340 in the subset of candidate chunks 380 .
- the second subset of documents 340 may be distributed in the subset of candidate chunks 380 such that an updated total size 405 , and an updated active size 410 of a respective updated set of active documents 402 in each respective chunk 382 of the subset of candidate chunks 380 may be expressed as:
- u′′ j is the updated active size 410 of the respective updated set of active documents 402 in the respective chunk 382
- s′′ j is the updated total size 405
- ⁇ ′ is the new request size 395
- k is the number of chunks in the subset of candidate chunks 380 .
- the second subset of documents 340 is distributed such that each respective candidate chunk 382 of the subset of candidate chunks 380 has an equal updated active size 410 , and an equal updated total size 405 .
- this may not be the case in every embodiment of the present technology, as documents may have different sizes, and consequently, different techniques may be used to distribute the second subset of documents 340 in the subset of candidate chunks 380 .
- documents from the second subset of documents 340 could be distributed sequentially in each respective candidate chunk 382 until each respective candidate chunk 382 is “full”, i.e. a storage size limit of each respective candidate chunk 382 is reached.
- the indexing server 120 may replace the subset of chunks 370 in the main portion 210 with the subset of candidate chunks 380 , the subset of candidate chunks 380 including the second subset of documents 340 of the request 310 .
- documents in the subset of candidate chunks 380 that were part of the subset of chunks 370 (i.e. before receiving the request 310 ) that have been redistributed in the subset of candidate chunks 380 may have their respective document identifier 262 , and their location 266 updated in the list 215 of the plurality of indexed documents 250 maintained by the indexing server 120 . Further, the list 215 of the plurality of indexed documents 250 may be updated to reflect addition of new documents of the second subset of candidate chunks 380 .
- the indexing server 120 may delete the subset of candidate chunks 380 from the update portion 280 of the search index database 146 .
- the predetermined size threshold 315 x, the predetermined number of chunks 221 n, and the predetermined number of candidate chunks 381 k may be chosen such that the update process 300 of the search index database 146 is optimal, i.e. there is a balance between resources required for storage of the search index database 146 , and update of the search index database 146 , as well as computational time.
- a “cost” of the update may be expressed as: min f ( x,k,n ) ⁇ y (12)
- x is the predetermined size threshold 315
- n is the predetermined number of chunks 221
- k is the predetermined number of candidate chunks 381
- y is the update size limit 282 .
- n is the predetermined number of chunks 221
- k is the predetermined number of candidate chunks 381 .
- a MLA may be used to determine, based on past update procedures, the predetermined size threshold 315 x, the predetermined number of chunks 221 n, and the predetermined number of candidate chunks 381 k.
- search index database 146 is structured and updated upon receipt of a request 310 to update a set of documents 320 in accordance with non-limiting embodiments of the present technology
- FIG. 6 a method 600 for dynamically updating the search index database 146 in accordance with non-limiting embodiments of the present technology.
- the method 600 may be executed by the indexing server 120 . In some embodiments, the method 600 may be executed by the indexing server 120 , and the search engine server 140 . In embodiments where at least a portion of the functionality of the indexing server 120 is implemented by the search engine server 140 , the method 600 may be executed by the search engine server 140 .
- the method 600 may begin at step 602 .
- STEP 602 receiving a request to update a set of documents in the search index, the request to update the set of documents including:
- the indexing server 120 may receive a request 310 to update a set of documents 320 , the request 310 to update the set of documents including: deletion of a first subset of documents 330 from the plurality of chunks 220 of the search index database 146 , and addition of a second subset of documents 340 to the plurality of chunks 220 of the search index database 146 .
- the request 310 to update the set of documents 320 may be transmitted by the plurality of crawlers 122 after a crawling procedure.
- the method 600 may then advance to step 604 .
- STEP 604 computing for each given chunk of a set of chunks from the plurality of chunks, each given chunk including at least one document of the first subset of documents:
- the indexing server 120 may compute, for each respective chunk 362 of the set of chunks 360 , a respective updated active size 364 of the respective set of active documents 230 in the respective chunk 362 , the respective updated active size 364 not including documents in the first subset of documents 330 that are part of the respective chunk 362 .
- the indexing server may compute, for each respective chunk 362 of the set of chunks 360 , the respective updated total size 366 of the respective portion of the plurality of indexed documents 252 in the respective chunk 362 , the respective updated total size 366 of the respective portion of the plurality of indexed documents 252 not including documents in the first subset of documents 330 that are part of the respective chunk 362 .
- the indexing server 120 may compute, for each respective chunk 362 of the set of chunks 360 , the respective ratio 368 of the respective updated active size 364 to the respective updated total size 366 .
- the method 600 may then advance to step 606 .
- STEP 606 duplicating, based on the respective ratio, a subset of chunks from the set of chunks to obtain a subset of candidate chunks
- the indexing server 120 may duplicate, a subset of chunks 370 in the update portion 280 of the search index database 146 , to obtain a subset of candidate chunks 380 .
- the subset of chunks 370 may be selected from the set of chunks based on the respective ratio 368 , each chunk in the subset of chunks 370 having a minimal respective ratio 368 .
- the subset of candidate chunks 380 may include a number k predetermined chunks.
- the method 600 may then advance to step 608 .
- STEP 608 updating, by the server, at least a portion of the set of documents in the subset of candidate chunks, the updating comprising:
- the indexing server 120 may delete at least a portion of the first subset of documents 330 from the subset of candidate chunks 380 .
- the indexing server 120 may then add all documents included in the subset of candidate chunks 380 in the second subset of documents 340 .
- the indexing server 120 may then distribute the second subset of documents 340 in the subset of candidate chunks 380 .
- the second subset of documents 340 may be distributed equally in the subset of candidate chunks 380 .
- the method 600 may then advance to step 610 .
- STEP 610 replacing, by the server, the subset of chunks with the subset of candidate chunks.
- the indexing server 120 may replace the subset of chunks 370 with the subset of candidate chunks 380 , the subset of candidate chunks 380 including the second subset of documents 340 .
- the method 600 may then end.
- the method 600 may be repeated every time a request to update a set of documents is received at the indexing server 120 .
- At least some embodiments of the present technology aim to expand a range of technical solutions for addressing a particular technical problem, namely efficiently updating a search index used by a search engine server by dividing the search index into a number of chunks, determining a number of chunks that may be used for updating the search index, a size limit of the updates, while also taking into account a size limit of a search index, which may allow saving storage resources and optimizing the process of updating the search index.
- the signals can be sent-received using optical means (such as a fibre-optic connection), electronic means (such as using wired or wireless connection), and mechanical means (such as pressure-based, temperature based or any other suitable physical parameter based).
- optical means such as a fibre-optic connection
- electronic means such as using wired or wireless connection
- mechanical means such as pressure-based, temperature based or any other suitable physical parameter based
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Business, Economics & Management (AREA)
- General Business, Economics & Management (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
RU2018140528 | 2018-11-16 | ||
RU2018140528A RU2733482C2 (ru) | 2018-11-16 | 2018-11-16 | Способ и система для обновления базы данных поискового индекса |
Publications (2)
Publication Number | Publication Date |
---|---|
US20200159783A1 US20200159783A1 (en) | 2020-05-21 |
US10891340B2 true US10891340B2 (en) | 2021-01-12 |
Family
ID=70727665
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/505,110 Active 2039-09-15 US10891340B2 (en) | 2018-11-16 | 2019-07-08 | Method of and system for updating search index database |
Country Status (2)
Country | Link |
---|---|
US (1) | US10891340B2 (ru) |
RU (1) | RU2733482C2 (ru) |
Families Citing this family (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP3660699A1 (en) * | 2018-11-29 | 2020-06-03 | Tata Consultancy Services Limited | Method and system to extract domain concepts to create domain dictionaries and ontologies |
CN111611492A (zh) * | 2020-05-26 | 2020-09-01 | 北京字节跳动网络技术有限公司 | 一种触发搜索的方法、装置、电子设备及存储介质 |
US11861320B1 (en) * | 2023-02-27 | 2024-01-02 | Casetext, Inc. | Text reduction and analysis interface to a text generation modeling system |
US11995411B1 (en) | 2023-02-28 | 2024-05-28 | Casetext, Inc. | Large language model artificial intelligence text evaluation system |
US11972223B1 (en) | 2023-06-30 | 2024-04-30 | Casetext, Inc. | Query evaluation in natural language processing systems |
Citations (22)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2003141136A (ja) | 2001-11-07 | 2003-05-16 | Toko Seiki Co Ltd | ドキュメント検索用インデックスの更新方法 |
US20030101183A1 (en) | 2001-11-26 | 2003-05-29 | Navin Kabra | Information retrieval index allowing updating while in use |
US20030135495A1 (en) | 2001-06-21 | 2003-07-17 | Isc, Inc. | Database indexing method and apparatus |
WO2003081462A2 (en) | 2002-03-20 | 2003-10-02 | Magus Research Ltd | Selective updating of index in a search engine |
US20090193406A1 (en) * | 2008-01-29 | 2009-07-30 | James Charles Williams | Bulk Search Index Updates |
US7702614B1 (en) * | 2007-03-30 | 2010-04-20 | Google Inc. | Index updating using segment swapping |
RU2398272C2 (ru) | 2004-05-14 | 2010-08-27 | Майкрософт Корпорейшн | Способ и система для индексирования и поиска в базах данных |
US8131691B1 (en) | 2002-12-30 | 2012-03-06 | Symantec Operating Corporation | System and method for updating a search engine index based on which files are identified in a file change log |
US8224841B2 (en) | 2008-05-28 | 2012-07-17 | Microsoft Corporation | Dynamic update of a web index |
US20120317105A1 (en) | 2009-09-21 | 2012-12-13 | Zte Corporation | Method and Apparatus for Updating Index and Sequencing Search Results Based on Updated Index in Terminal |
US8423885B1 (en) | 2009-01-30 | 2013-04-16 | Google Inc. | Updating search engine document index based on calculated age of changed portions in a document |
CN103198108A (zh) | 2013-03-27 | 2013-07-10 | 新浪网技术(中国)有限公司 | 一种索引数据更新方法、检索服务器及*** |
US8655886B1 (en) | 2011-03-25 | 2014-02-18 | Google Inc. | Selective indexing of content portions |
US8949247B2 (en) * | 2007-12-20 | 2015-02-03 | Microsoft International Holdings B.V. | Method for dynamic updating of an index, and a search engine implementing the same |
US20150356314A1 (en) * | 2014-06-10 | 2015-12-10 | Salesforce.Com, Inc. | Systems and methods for implementing an encrypted search index |
US9430543B2 (en) | 2013-03-15 | 2016-08-30 | Wal-Mart Stores, Inc. | Incrementally updating a large key-value store |
US20170068678A1 (en) | 2015-09-03 | 2017-03-09 | Oracle International Corporation | Methods and systems for updating a search index |
US9665605B2 (en) | 2014-09-09 | 2017-05-30 | Kcura Llc | Methods and apparatus for building a search index for a database |
US20170199908A1 (en) * | 2016-01-07 | 2017-07-13 | International Business Machines Corporation | System and Method for Asynchronous Update of a Search Index |
US9836541B2 (en) | 2011-08-30 | 2017-12-05 | Open Text Sa Ulc | System and method of managing capacity of search index partitions |
US20190227878A1 (en) * | 2018-01-19 | 2019-07-25 | Rubrik, Inc. | Cloud instantiation using out-of-order incrementals |
US10417190B1 (en) * | 2014-09-25 | 2019-09-17 | Amazon Technologies, Inc. | Log-structured file system for zone block devices with small zones |
-
2018
- 2018-11-16 RU RU2018140528A patent/RU2733482C2/ru active
-
2019
- 2019-07-08 US US16/505,110 patent/US10891340B2/en active Active
Patent Citations (23)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20030135495A1 (en) | 2001-06-21 | 2003-07-17 | Isc, Inc. | Database indexing method and apparatus |
JP2003141136A (ja) | 2001-11-07 | 2003-05-16 | Toko Seiki Co Ltd | ドキュメント検索用インデックスの更新方法 |
US20030101183A1 (en) | 2001-11-26 | 2003-05-29 | Navin Kabra | Information retrieval index allowing updating while in use |
WO2003081462A2 (en) | 2002-03-20 | 2003-10-02 | Magus Research Ltd | Selective updating of index in a search engine |
US8131691B1 (en) | 2002-12-30 | 2012-03-06 | Symantec Operating Corporation | System and method for updating a search engine index based on which files are identified in a file change log |
RU2398272C2 (ru) | 2004-05-14 | 2010-08-27 | Майкрософт Корпорейшн | Способ и система для индексирования и поиска в базах данных |
US7702614B1 (en) * | 2007-03-30 | 2010-04-20 | Google Inc. | Index updating using segment swapping |
US8949247B2 (en) * | 2007-12-20 | 2015-02-03 | Microsoft International Holdings B.V. | Method for dynamic updating of an index, and a search engine implementing the same |
US20090193406A1 (en) * | 2008-01-29 | 2009-07-30 | James Charles Williams | Bulk Search Index Updates |
US8224841B2 (en) | 2008-05-28 | 2012-07-17 | Microsoft Corporation | Dynamic update of a web index |
US8423885B1 (en) | 2009-01-30 | 2013-04-16 | Google Inc. | Updating search engine document index based on calculated age of changed portions in a document |
US20120317105A1 (en) | 2009-09-21 | 2012-12-13 | Zte Corporation | Method and Apparatus for Updating Index and Sequencing Search Results Based on Updated Index in Terminal |
US8655886B1 (en) | 2011-03-25 | 2014-02-18 | Google Inc. | Selective indexing of content portions |
US9836541B2 (en) | 2011-08-30 | 2017-12-05 | Open Text Sa Ulc | System and method of managing capacity of search index partitions |
US9430543B2 (en) | 2013-03-15 | 2016-08-30 | Wal-Mart Stores, Inc. | Incrementally updating a large key-value store |
US20160342612A1 (en) * | 2013-03-15 | 2016-11-24 | Wal-Mart Stores, Inc. | Incrementally updating a large key-value store |
CN103198108A (zh) | 2013-03-27 | 2013-07-10 | 新浪网技术(中国)有限公司 | 一种索引数据更新方法、检索服务器及*** |
US20150356314A1 (en) * | 2014-06-10 | 2015-12-10 | Salesforce.Com, Inc. | Systems and methods for implementing an encrypted search index |
US9665605B2 (en) | 2014-09-09 | 2017-05-30 | Kcura Llc | Methods and apparatus for building a search index for a database |
US10417190B1 (en) * | 2014-09-25 | 2019-09-17 | Amazon Technologies, Inc. | Log-structured file system for zone block devices with small zones |
US20170068678A1 (en) | 2015-09-03 | 2017-03-09 | Oracle International Corporation | Methods and systems for updating a search index |
US20170199908A1 (en) * | 2016-01-07 | 2017-07-13 | International Business Machines Corporation | System and Method for Asynchronous Update of a Search Index |
US20190227878A1 (en) * | 2018-01-19 | 2019-07-25 | Rubrik, Inc. | Cloud instantiation using out-of-order incrementals |
Non-Patent Citations (3)
Title |
---|
English Abstract for the publication CN103198108 (A) (related to CN103198108 (B), retrieved from Espacenet on Jul. 30, 2018. |
English Abstract for the publication JP2003141136 (A), retrieved from Espacenet on Jul. 30, 2018. |
Russian Search Report dated May 22, 2020 issued in respect of the Russian Patent Application No. RU2018140528. |
Also Published As
Publication number | Publication date |
---|---|
RU2018140528A3 (ru) | 2020-05-25 |
US20200159783A1 (en) | 2020-05-21 |
RU2733482C2 (ru) | 2020-10-01 |
RU2018140528A (ru) | 2020-05-18 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10891340B2 (en) | Method of and system for updating search index database | |
US10296538B2 (en) | Method for matching images with content based on representations of keywords associated with the content in response to a search query | |
US11113291B2 (en) | Method of and system for enriching search queries for ranking search results | |
US10956502B2 (en) | Method of and system for recommending fresh search query suggestions on search engine | |
US9396188B2 (en) | Assigning tags to digital content | |
US8417692B2 (en) | Generalized edit distance for queries | |
US9183499B1 (en) | Evaluating quality based on neighbor features | |
US11194848B2 (en) | Method of and system for building search index using machine learning algorithm | |
RU2744029C1 (ru) | Система и способ формирования обучающего набора для алгоритма машинного обучения | |
US11681713B2 (en) | Method of and system for ranking search results using machine learning algorithm | |
US11194878B2 (en) | Method of and system for generating feature for ranking document | |
US20160110437A1 (en) | Activity stream | |
US11775573B2 (en) | Method of and server for retraining machine learning algorithm | |
US20110161318A1 (en) | Method and apparatus for assigning tags to digital content | |
US9773035B1 (en) | System and method for an annotation search index | |
US11816159B2 (en) | Method of and system for generating a training set for a machine learning algorithm (MLA) | |
US20200401638A1 (en) | Method of and system for generating search query completion suggestion on search engine | |
US20150088859A1 (en) | Click magnet images | |
US11308097B2 (en) | Method of and server for generating meta-feature for ranking documents | |
US20200089714A1 (en) | Method and server for indexing web page in index | |
RU2739554C1 (ru) | Способ и система для формирования карточки объекта | |
US20210374606A1 (en) | Method of and system for generating a training set for a machine learning algorithm (mla) | |
US11334559B2 (en) | Method of and system for identifying abnormal rating activity | |
RU2778385C1 (ru) | Способ и система для формирования обучающего набора для алгоритма машинного обучения | |
RU2790033C2 (ru) | Система и способ для формирования обучающего набора для алгоритма машинного обучения |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
AS | Assignment |
Owner name: DIRECT CURSUS TECHNOLOGY L.L.C, UNITED ARAB EMIRATES Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:YANDEX EUROPE AG;REEL/FRAME:065692/0720 Effective date: 20230912 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |