WO2015103997A1 - 一种基于关键词检索的网络爬虫调度方法及*** - Google Patents
一种基于关键词检索的网络爬虫调度方法及*** Download PDFInfo
- Publication number
- WO2015103997A1 WO2015103997A1 PCT/CN2015/070475 CN2015070475W WO2015103997A1 WO 2015103997 A1 WO2015103997 A1 WO 2015103997A1 CN 2015070475 W CN2015070475 W CN 2015070475W WO 2015103997 A1 WO2015103997 A1 WO 2015103997A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- bucket
- link address
- page
- task
- keyword
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 20
- 230000009193 crawling Effects 0.000 claims description 73
- 238000004458 analytical method Methods 0.000 claims description 51
- 230000000694 effects Effects 0.000 claims description 11
- 238000010586 diagram Methods 0.000 description 4
- 238000007726 management method Methods 0.000 description 4
- 238000013500 data storage Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/162—Delete operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/958—Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
Definitions
- the invention relates to the technical field of web crawlers, in particular to a web crawler scheduling method and system based on keyword retrieval.
- Web crawlers based on keyword retrieval are the basis for obtaining keyword information, but the update of information may lead to repeated crawling or catching of web crawlers.
- the target website information changes very frequently, such as Sina Weibo and other microblogging websites, Baidu and other search websites.
- For popular keywords because the information is updated very quickly, it is difficult for the web crawler to capture the data.
- For the unpopular keywords due to the slower information update, the information will be repeatedly crawled.
- the current practice is to set different enthusiasm for keywords, and then crawl keywords according to keyword popularity, and hot keywords are more frequently crawled.
- a web crawler scheduling method based on keyword retrieval comprising:
- Step 12 The scheduling end receives a task request command from the crawling node.
- Step 13 The scheduling end acquires the secondary download link address generation task from the priority bucket that saves the secondary download link address, and joins the task list. If the number of priority buckets is allowed to be added to the task list, step 16 is performed; otherwise, step 14 is performed.
- the secondary download link address is obtained by the crawling node according to the task in the task list, and the crawling page is analyzed, and the link address that needs to be downloaded twice is analyzed from the crawling page;
- Step 14 The dispatching end acquires a keyword link address from a dynamic bucket that stores a keyword multiple page code link address, and derives a derivative link address of the page number according to the number of page codes corresponding to the keyword link address, according to the number of the page number. Deriving the link address to generate the task of the number of page numbers and adding the task list. If the number of dynamic buckets allowed to join the task list has been reached, step 16 is performed; otherwise, step 15 is performed, and the keyword link address is based on the target website.
- the link result address of the search result page generated by the keyword, and the search result page corresponding to the keyword link address in the dynamic bucket includes a page number greater than or equal to a preset page number threshold, and the page number threshold is greater than or equal to 2;
- Step 15 The dispatching end acquires a keyword link address from the basic bucket that stores the keyword link address to generate a task and joins the task list. If the number of basic buckets is allowed to be added to the task list, step 16 is performed, and the keyword link address is a search result page corresponding to the keyword link address generated in the target bucket, and a search result page corresponding to the keyword link address in the basic bucket includes a page number smaller than a preset page number threshold, and the page number threshold is greater than Or equal to 2;
- Step 16 The dispatcher returns a task list to the crawling node, and the crawling node executes the task in the task list according to the received task list.
- a web crawler scheduling system based on keyword retrieval comprising: a dispatching end and at least one crawling node communicating with the dispatching end;
- the dispatching end includes:
- a task request command receiving module configured to receive a task request command sent by the crawling node
- the priority bucket task generation module is configured to acquire a secondary download link address generation task from the priority bucket that saves the secondary download link address and join the task list, if the task list has been reached If the priority bucket number is allowed to be added, the task list is returned to the module, otherwise the dynamic bucket task generation module is executed, and the secondary download link address is obtained by the crawling node according to the task in the task list, and the crawling page is obtained from the crawling node.
- the crawling page analyzes the link address that needs to be downloaded twice;
- a dynamic bucket task generating module configured to store a dynamic bucket of a keyword multi-page code link address to obtain a keyword link address, and derive a derivative link address of the page number according to a page number corresponding to the keyword link address, according to the page number
- the number of derived link addresses generates the task of the number of page numbers and joins the task list. If the number of dynamic buckets allowed to join the task list is reached, the task list return module is executed, otherwise the dynamic bucket task generation module is executed, and the keyword link address is executed.
- a search result page corresponding to the keyword search link generated in the target website, and a search result page corresponding to the keyword link address in the dynamic bucket includes a page number greater than or equal to a preset page number threshold, the page number The quantity threshold is greater than or equal to 2;
- a basic bucket task generating module configured to acquire a keyword link address generation task from a basic bucket storing a keyword link address and join the task list, and if the task list is allowed to join the basic bucket number, execute a task list return module,
- the keyword link address is a link address of a search result page generated according to the keyword in the target website, and the search result page corresponding to the keyword link address in the basic bucket includes a page number smaller than a preset page number threshold.
- the page number threshold is greater than or equal to 2;
- a task list returning module for returning a task list to the crawling node
- the crawling node includes:
- a task request command sending module configured to send a task request command to the dispatching end
- a task execution module configured to execute a task in the task list according to the received task list.
- the priority bucket, the dynamic bucket, and the basic bucket are all virtual buckets.
- virtual buckets By setting virtual buckets with different priorities, different types of link addresses are respectively stored in the virtual bucket, and the higher-priority link addresses are placed in higher priority.
- Level in the virtual bucket Flexibly adjust the number of calls for different types of link addresses by adjusting the number of allowed tasks for each virtual bucket.
- the heat of the link address is not set manually, but based on the link address. The actual situation of the page number that is captured is set, thus making it more frequent to grab popular keywords, avoiding data omissions, and reducing repeated crawling of unpopular keywords.
- FIG. 1 is a flowchart of a network crawler scheduling method based on keyword retrieval according to the present invention
- FIG. 2 is a schematic block diagram of a web crawler scheduling system based on keyword retrieval according to the present invention
- FIG. 3 is a schematic structural diagram of a system according to an example of the present invention.
- FIG. 4 is a working flow chart of an example bucket scheduling management module according to the present invention.
- FIG. 5 is a flow chart showing the operation of an example bucket updater of the present invention.
- FIG. 1 is a flowchart of a network crawler scheduling method based on keyword retrieval according to the present invention, including:
- Step 11 The crawling node sends a task request command to the dispatching end.
- Step 12 The scheduling end receives the task request command sent by the crawling node.
- Step 13 The scheduling end acquires the secondary download link address generation task from the priority bucket that saves the secondary download link address, and joins the task list. If the number of priority buckets is allowed to be added to the task list, step 16 is performed; otherwise, step 14 is performed.
- the secondary download link address is obtained by the crawling node according to the task in the task list, and the crawling page is analyzed, and the link address that needs to be downloaded twice is analyzed from the crawling page;
- Step 14 The dispatching end acquires a keyword link address from a dynamic bucket that stores a keyword multiple page code link address, and derives a derivative link address of the page number according to the number of page codes corresponding to the keyword link address, according to the number of the page number. Deriving the link address to generate the task of the number of page numbers and adding the task list. If the number of dynamic buckets allowed to join the task list has been reached, step 16 is performed; otherwise, step 15 is performed, and the keyword link address is based on the target website.
- the link address of the search result page generated by the keyword, and The search result page corresponding to the keyword link address in the dynamic bucket includes a page number greater than or equal to a preset page number threshold, and the page number threshold is greater than or equal to 2;
- Step 15 The dispatching end acquires a keyword link address from the basic bucket that stores the keyword link address to generate a task and joins the task list. If the number of basic buckets is allowed to be added to the task list, step 16 is performed, and the keyword link address is a search result page corresponding to the keyword link address generated in the target bucket, and a search result page corresponding to the keyword link address in the basic bucket includes a page number smaller than a preset page number threshold, and the page number threshold is greater than Or equal to 2;
- Step 16 The dispatcher returns a task list to the crawling node, and the crawling node executes the task in the task list according to the received task list.
- step 13 the processing is specifically performed for the secondary download link address, and the second download of the link address means that the crawling node performs the capture to the crawling page according to the task in the task list, and the analysis needs to be performed from the crawling page.
- the link address of the second download for example, searching for "Jingdong" in Baidu, a search result page including multiple search results will appear, each search result corresponding to a link address, and the link address corresponding to the search result is two Download link address. From the crawling page, the link address that needs to be downloaded twice is obtained, and there are already mature technologies for analysis, which will not be described here.
- the keyword link address in steps 14 and 15 refers to the link address of the search result page generated according to the keyword in the target website. For example, if "Jingdong" is searched in Baidu, a key according to the key will appear.
- the search result page generated by the word "Jingdong” if the keyword is a popular keyword, a search result page does not include all the search results, there will be a page number on the search page.
- the number of page numbers is used to indicate that the search result page having the number of page numbers includes the keyword. Therefore, the more the number of page numbers, the more popular the keyword is. Therefore, the keyword link address whose page number exceeds the preset page number threshold is placed in the dynamic bucket, and is preferentially processed in step 14. Since it is judged based on the actual number of page numbers, it is possible to more realistically reflect the popularity of keywords.
- the number of priority buckets allowed to be used is used to indicate the number of secondary download link addresses in the priority bucket allowed to join the task list
- the number of dynamic buckets allowed to be added is used to indicate the number of keyword link addresses in the dynamic bucket allowed to join the task list.
- the number of allowed basic buckets is used to indicate the number of keyword link addresses in the basic bucket that are allowed to join the task list.
- the step 13 specifically includes: acquiring a secondary download link address generation task from the priority bucket holding the secondary download link address and joining the task list, and deleting the secondary download of the generated task from the priority bucket.
- Link address if the number of priority buckets allowed to join the task list is reached, go to step 16. Otherwise, if there is a secondary download link address in the priority bucket, go to step 13. If all secondary download link addresses in the priority bucket are in the priority bucket If they have been deleted, go to step 14.
- the secondary download link address is preferentially executed, and the secondary download link address is deleted only once after execution, thereby avoiding repeated crawling.
- the step 14 specifically includes: acquiring an unscheduled keyword link address from a dynamic bucket storing a keyword link address, and deriving the page number according to the number of page codes corresponding to the keyword link address. Deriving a link address, generating a task of the page number according to the derived link address of the page number, adding a task list, and setting a state of the keyword link address of the generated task to be scheduled, if the task list is allowed to join the dynamic If the number of buckets is the same, go to step 16. Set the status of all the keyword link addresses in the dynamic bucket to unscheduled. Otherwise, if there are unscheduled keyword link addresses in the dynamic bucket, go to step 14. If the bucket is in the dynamic bucket. If the unscheduled keyword link address is not saved, go to step 15.
- the keyword link address in the dynamic bucket is the link address corresponding to the popular keyword, it is called multiple times. Thereby avoiding the omission of popular keyword crawling.
- the basic bucket includes an active bucket and a suspended bucket
- the step 15 specifically includes: obtaining the scheduling time from the active bucket storing the keyword link address.
- the keyword link address generates a task and joins the task list, and increases the scheduling time of the keyword link address of the generated task by a preset scheduling time increase amount, and then moves to the suspended bucket. If the task list has been allowed to join the basic bucket number Then, step 16 is performed. Otherwise, if the keyword link address is still stored in the activity bucket, step 15 is performed. If the keyword link address is not saved in the activity bucket, step 16 is performed.
- the basic bucket since the keyword link address in the basic bucket is the link address corresponding to the unpopular keyword, the basic bucket is further set as the active bucket and the suspended bucket, and the link address in the active bucket is moved to Suspend the bucket and execute it after an interval increase of the scheduling time, so as to avoid repeated crawling of the unpopular keyword.
- the method further includes:
- the crawling node obtains the crawling page according to the task in the task list, and analyzes the crawling page to obtain analysis data including the secondary download link address, the information details or the page number, and sends the analysis data to the dispatching end;
- the dispatcher receives the analysis data:
- the analysis data is a secondary download link address
- the secondary download link address is placed in the priority bucket
- analysis data is information details
- the information details are put into the data bucket
- the keyword link address corresponding to the number of page numbers is adjusted in the dynamic bucket and the basic bucket.
- Different analysis data is placed in different virtual buckets to update the link address of the virtual bucket.
- the scheduling end adjusting the keyword link address corresponding to the number of the page numbers in the dynamic bucket and the basic bucket includes:
- the number of page numbers corresponding to the corresponding keyword link address is modified to be a new page number
- the corresponding keyword link address is moved to the dynamic bucket.
- the keyword link address corresponding to the number of page numbers is adjusted in the dynamic bucket and the basic bucket according to the comparison between the number of old page numbers and the number of new page numbers.
- the crawling node obtains the crawling page according to the task in the task list, and analyzes the crawling page to obtain analysis data including the secondary download link address, the information details or the page number, and sends the analysis data to the dispatching end;
- the dispatcher receives the analysis data:
- the analysis data is a secondary download link address
- the secondary download link address is placed in the priority bucket
- analysis data is information details
- the information details are put into the data bucket
- the analysis data is the number of page numbers, set the number of page numbers currently received to be the number of new page numbers, and set the number of page numbers received by the same keyword link address as the number of old page numbers, and if the number of old page numbers does not match the number of new page numbers, :
- the number of page numbers corresponding to the corresponding keyword link address is modified to be a new page number
- the page number threshold is used to retrieve the suspended bucket and move the keyword link address in the suspended bucket to the current time to the active bucket.
- the keyword link address corresponding to the page number is adjusted in the dynamic bucket and the active bucket according to the comparison between the number of the old page number and the new page number, and the suspended bucket is retrieved at an appropriate time to avoid hanging.
- the link address in the bucket is put on hold for a long time.
- the number of allowed dynamic buckets is greater than the number of allowed basic buckets. By setting the number of allowed dynamic buckets to be larger than the number of allowed basic buckets, more resources are used to capture popular keywords.
- FIG. 2 is a block diagram of a network crawler scheduling system based on keyword retrieval, which is characterized in that it includes: a dispatching end 21 and at least one crawling node 22 communicating with the dispatching end 21;
- the dispatching end 21 includes:
- the task request command receiving module 211 is configured to receive a task request command sent by the crawling node;
- the priority bucket task generating module 212 is configured to acquire a secondary download link address generation task from the priority bucket that saves the secondary download link address and join the task list, and if the task list is allowed to join the priority bucket number, execute the task list return module. Otherwise, the dynamic bucket task generating module is executed, and the secondary download link address is obtained by the crawling node according to the task in the task list, and the crawling page is obtained, and the link that needs to be downloaded twice is analyzed from the crawling page. address;
- the dynamic bucket task generating module 213 is configured to save a dynamic bucket of the keyword multi-page code link address to obtain a keyword link address, and derive a derivative link address of the page number according to the number of page codes corresponding to the keyword link address, according to the The derived link address of the page number generates the task of the page number and joins the task list. If the number of dynamic buckets allowed to join the task list is reached, the task list return module is executed, otherwise the dynamic bucket task generation module is executed, and the keyword link is executed.
- the address is in the target website according to the keyword a search result page of the generated search result page, and the search result page corresponding to the keyword link address in the dynamic bucket includes a page number greater than or equal to a preset page number threshold, and the page number threshold is greater than or equal to 2;
- the basic bucket task generating module 214 is configured to obtain a keyword link address generation task from the basic bucket storing the keyword link address and join the task list. If the task list is allowed to join the basic bucket number, the task list return module is executed.
- the keyword link address is a link address of a search result page generated according to the keyword in the target website, and the search result page corresponding to the keyword link address in the basic bucket includes a page number smaller than a preset page number threshold.
- the page number threshold is greater than or equal to 2;
- the task list returning module 215 is configured to return a task list to the crawling node
- the crawling node 22 includes:
- the task request command sending module 221 is configured to send a task request command to the dispatching end;
- the task execution module 222 is configured to execute a task in the task list according to the received task list.
- the priority bucket task generating module 212 is specifically configured to: acquire a secondary download link address generation task from the priority bucket that saves the secondary download link address, join the task list, and delete the generated from the priority bucket.
- the secondary download link address of the task if the priority bucket number of the task list is allowed to be added, the task list returning module 215 is executed. Otherwise, if the secondary download link address is also stored in the priority bucket, the priority bucket task generating module 212 is executed. If all the secondary download link addresses in the priority bucket have been deleted, the dynamic bucket task generation module 213 is executed.
- the dynamic bucket task generating module 213 is specifically configured to: obtain an unscheduled keyword link address from a dynamic bucket that stores a keyword link address, and derive the page number according to the number of page codes corresponding to the keyword link address. Deriving a link address of the page number, generating a task of the page number according to the derived link address of the page number, adding a task list, and setting a state of the keyword link address of the generated task to be scheduled, if the task has been reached If the number of dynamic buckets is allowed to be added to the list, the task list returning module 215 is executed, and the state of all the keyword link addresses in the dynamic bucket is set to be unscheduled, otherwise If the unscheduled keyword link address is also stored in the dynamic bucket, the dynamic bucket task generation module 213 is executed. If the unscheduled keyword link address is not stored in the dynamic bucket, the basic bucket task generation module 214 is executed.
- the basic bucket includes an active bucket and a suspended bucket
- the basic bucket task generating module 214 is specifically configured to acquire a keyword link address generation with the highest scheduling time from the active bucket that saves the keyword link address.
- the task is added to the task list, and the scheduling time of the keyword link address of the generated task is increased by the preset scheduling time increase amount, and then moved to the suspended bucket. If the task list is allowed to join the basic bucket number, the task list is executed. Returning to the module 215, if the keyword link address is still stored in the activity bucket, the basic bucket task generation module 214 is executed. If the keyword link address is not saved in the activity bucket, the task list return module 215 is executed.
- the task execution module is specifically configured to: perform a capture to the crawling page according to the task in the task list, and analyze the captured page to obtain analysis data including a secondary download link address, information details, or page number, and analyze the data. Data is sent to the dispatcher;
- the dispatching end further includes an analysis data receiving module for receiving the analysis data:
- the analysis data is a secondary download link address
- the secondary download link address is placed in the priority bucket
- analysis data is information details
- the information details are put into the data bucket
- the keyword link address corresponding to the number of page numbers is adjusted in the dynamic bucket and the basic bucket.
- the analyzing the data receiving module to adjust the keyword link address corresponding to the page number in the dynamic bucket and the basic bucket includes:
- the number of page codes corresponding to the corresponding keyword link address is modified to be a new page number
- the corresponding keyword link address is moved to the dynamic bucket.
- the task execution module is specifically configured to: perform a capture to the crawling page according to the task in the task list, and analyze the captured page to obtain analysis data including a secondary download link address, information details, or page number, and analyze the data. Data is sent to the dispatcher;
- the dispatching end further includes an analysis data receiving module for receiving the analysis data:
- the analysis data is a secondary download link address
- the secondary download link address is placed in the priority bucket
- analysis data is information details
- the information details are put into the data bucket
- the analysis data is the number of page numbers, set the number of page numbers currently received to be the number of new page numbers, and set the number of page numbers received by the same keyword link address as the number of old page numbers, and if the number of old page numbers does not match the number of new page numbers, :
- the number of page numbers corresponding to the corresponding keyword link address is modified to be a new page number
- the suspended bucket is retrieved, and the keyword link address in the suspended bucket that reaches the current time in the scheduled time is moved to the active bucket.
- the number of allowed dynamic buckets is greater than the allowed addition. Enter the number of basic buckets.
- FIG. 3 is a schematic structural diagram of a system according to an example of the present invention, including:
- the priority bucket 321 located at the dispatching end 32, the dynamic bucket 322, the basic bucket 323, the active bucket 3231, the suspended bucket 3232, the bucket updater 324, the data bucket 325, and the database 326.
- the bucket scheduling management module (BucketScheduler) includes a priority bucket 321, a dynamic bucket 322, and a basic bucket 323.
- the priority bucket 321 stores a secondary download link address returned by the crawling node, and the URLs are deleted from the bucket after the scheduling.
- the dynamic bucket 322 stores URLs composed of keywords of each website. These URLs will generate new URLs according to the results returned by the statistics module. These derived URLs will not be saved, but will be scheduled together with the URLs in the dynamic bucket.
- the regular bucket also stores URLs composed of keywords of each website, but these URLs do not derive new URLs.
- the regular bucket 323 is divided into two parts: the active bucket 3231 and the suspended bucket 3232, and the URL in the active bucket 3231 is scheduled.
- the bucket scheduling management module extracts the corresponding URL from the bucket and constructs the task as a task and sends the task to the crawling node.
- the URL is obtained from the regular bucket. If the number of URLs of the active bucket in the regular bucket is less than or equal to 200, all the URLs are built into the task, and all the URLs are added to the suspended bucket; if the number of URLs in the active bucket is greater than 200, then The top 200 URLs are scheduled to build tasks, and these URLs are added to the suspended bucket.
- FIG. 4 is a flow chart of a bucket scheduling management module. Its In the priorityBucket is allowed to join the number of priority buckets, dynamicBucket ⁇ the average number of page numbers per URL is allowed to join the dynamic bucket number, the task size size minus the priorityBucket minus the dynamicBucket ⁇ the average number of pages per URL, then the basic number is allowed to join.
- the number of barrels As shown in Figure 4, it includes:
- Step S401 obtaining a URL from the priority bucket, while deleting from the bucket, if the task is full, return, otherwise step S402;
- Step S402 obtaining a URL from the dynamic bucket, and deriving a new URL through the page number, if the task is full, returning, otherwise performing step S403;
- Step S403 obtaining a URL from the activity bucket (conventional bucket), modifying the next scheduling time of the URLs, and inserting the suspended bucket.
- the statistics module is located at the capture end. After the crawling node requests the data, the number of information of the page and the time interval between the first information and the last information of the page are extracted and sent to the statistics module, and the statistics module calculates the keyword according to the data. The number of page numbers that can be generated in 30 minutes and pass the data to the dispatcher.
- the bucket updater 324 (BucketUpdater) is located at the dispatcher 32, and its main function is to update the URLs in the priority bucket 321, the dynamic bucket 322, and the basic bucket 323.
- the data extracted by the crawling node 31, including the secondary download link address, the information details, and the page number, are transmitted to the bucket updater 324 of the dispatcher 32.
- the bucket updater 324 updates different according to the data type.
- Bucket the specific process, as shown in Figure 5, includes:
- step S510 the data type is checked. If the analyzed data is the secondary download link address, step S520 is performed. If the analyzed data is the information details, step S540 is performed, and if the analyzed data is the number of page numbers, Then executing step S540;
- Step S520 inserting the data into a priority bucket (PriorityBucket) for scheduling by the dispatcher;
- step S530 the data is inserted into the data bucket (DataBucket) for the service party to take data.
- DataBucket data bucket
- Step S540 comparing with the number of page numbers obtained last time, if the number of page numbers is Wait, then do nothing and return. If the number of page numbers is not equal, then:
- step S541 the URL is currently located in the dynamic bucket, and there is no need to exchange the bucket, and only the number of page numbers of the barcode is modified.
- step S542 the URL is currently located in the dynamic bucket and needs to be moved to the active bucket.
- step S543 the URL is currently located in the regular bucket and needs to be moved to the dynamic bucket.
- step S544 the URL is currently located in the regular bucket and is likely to be a suspended bucket.
- the suspended bucket needs to be scanned, and the URL of the suspended bucket to reach the scheduling time is added to the active bucket.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Human Computer Interaction (AREA)
- Information Transfer Between Computers (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
Claims (16)
- 一种基于关键词检索的网络爬虫调度方法,其特征在于,包括:步骤(12),调度端从抓取节点接收任务请求命令;步骤(13),调度端从保存二次下载链接地址的优先桶获取二次下载链接地址生成任务并加入任务列表,如果已达到任务列表的允许加入优先桶数量,则执行步骤(16),否则执行步骤(14),所述二次下载链接地址为所述抓取节点根据任务列表中的任务进行抓取得到抓取页面,从抓取页面中分析得到需要进行二次下载的链接地址;步骤(14),调度端从保存关键词多页码链接地址的动态桶获取关键词链接地址,并根据与关键词链接地址对应的页码数量衍生出所述页码数量的衍生链接地址,根据所述页码数量的衍生链接地址生成所述页码数量的任务并加入任务列表,如果已达到任务列表的允许加入动态桶数量,则执行步骤(16),否则执行步骤(15),所述关键词链接地址为在目标网站中根据所述关键词所生成的搜索结果页面的链接地址,且动态桶中的关键词链接地址对应的搜索结果页面包括大于或等于预设页码数量阈值的页码数量,所述页码数量阈值大于或等于2;步骤(15),调度端从保存关键词链接地址的基本桶获取关键词链接地址生成任务并加入任务列表,如果已达到任务列表的允许加入基本桶数量,则执行步骤(16),所述关键词链接地址为在目标网站中根据所述关键词所生成的搜索结果页面的链接地址,且基本桶中的关键词链接地址对应的搜索结果页面包括小于预设页码数量阈值的页码数量,所述页码数量阈值大于或等于2;步骤(16),调度端向抓取节点返回任务列表,所述抓取节点根据所接收到的任务列表执行任务列表中的任务。
- 根据权利要求1所述的基于关键词检索的网络爬虫调度方法,其特征在于,所述步骤(13)具体包括:从保存二次下载链接地址的优先桶获取二次下载链接地址生成任务并加入任务列表,并从优先桶 中删除已生成任务的二次下载链接地址,如果已达到任务列表的允许加入优先桶数量,则执行步骤(16),否则如果优先桶中还保存有二次下载链接地址,则执行步骤(13),如果优先桶中所有的二次下载链接地址均已删除,则执行步骤(14)。
- 根据权利要求1所述的基于关键词检索的网络爬虫调度方法,其特征在于,所述步骤(14)具体包括:从保存关键词链接地址的动态桶获取未调度的关键词链接地址,并根据与关键词链接地址对应的页码数量衍生出所述页码数量的衍生链接地址,根据所述页码数量的衍生链接地址生成所述页码数量的任务并加入任务列表,并设置已生成任务的关键词链接地址的状态为已调度,如果已达到任务列表的允许加入动态桶数量,则执行步骤(16),并设置动态桶中所有的关键词链接地址的状态为未调度,否则如果动态桶中还保存有未调度的关键词链接地址,则执行步骤(14),如果动态桶中未保存有未调度的关键词链接地址,则执行步骤(15)。
- 根据权利要求1所述的基于关键词检索的网络爬虫调度方法,其特征在于,所述基本桶包括活动桶和挂起桶,所述步骤(15)具体包括,从保存关键词链接地址的活动桶获取调度时间最前的关键词链接地址生成任务并加入任务列表,并将已生成任务的关键词链接地址的调度时间增加预设的调度时间增加量后移动到挂起桶,如果已达到任务列表的允许加入基本桶数量,则执行步骤(16),否则如果活动桶中还保存有关键词链接地址,则执行步骤(15),如果活动桶中未保存关键词链接地址,则执行步骤(16)。
- 根据权利要求1所述的基于关键词检索的网络爬虫调度方法,其特征在于,还包括:抓取节点根据任务列表中的任务进行抓取得到抓取页面,对所述抓取页面进行分析得到包括二次下载链接地址、信息详情或页码数量的分析数据,将分析数据发送到调度端;调度端接收到分析数据:如果分析数据为二次下载链接地址,则将二次下载链接地址放入优先桶;如果分析数据为信息详情,则将信息详情放入数据桶;如果分析数据为页码数量,则在动态桶和基本桶中调整与所述页码数量对应的关键词链接地址。
- 根据权利要求5所述的基于关键词检索的网络爬虫调度方法,其特征在于,调度端在动态桶和基本桶中调整与所述页码数量对应的关键词链接地址具体包括:设置当前接收到的页码数量为新页码数量,设置上一次同一关键词链接地址接收到的页码数量为旧页码数量;如果旧页码数量与新页码数量不一致,则:如果旧页码数量大于或等于所述页码数量阈值,且新页码数量大于或等于所述页码数量阈值,则修改对应的关键词链接地址对应的页码数量为新页码数量;如果旧页码数量大于或等于所述页码数量阈值,且新页码数量小于所述页码数量阈值,则将对应的关键词链接地址移动到基本桶;如果旧页码数量小于所述页码数量阈值,且新页码数量大于或等于所述页码数量阈值,则将对应的关键词链接地址移动到动态桶。
- 根据权利要求4所述的基于关键词检索的网络爬虫调度方法,其特征在于:抓取节点根据任务列表中的任务进行抓取得到抓取页面,对所述抓取页面进行分析得到包括二次下载链接地址、信息详情或页码数量的分析数据,将分析数据发送到调度端;调度端接收到分析数据:如果分析数据为二次下载链接地址,则将二次下载链接地址放入优先桶;如果分析数据为信息详情,则将信息详情放入数据桶;如果分析数据为页码数量,则设置当前接收到的页码数量为新页码数量,设置上一次同一关键词链接地址接收到的页码数量为旧页码数量,且如果旧页码数量与新页码数量不一致,则:如果旧页码数量大于或等于所述页码数量阈值,且新页码数量大于或等于所述页码数量阈值,则修改对应的关键词链接地址对应的页码数量为新页码数量;如果旧页码数量大于或等于所述页码数量阈值,且新页码数量小于所述页码数量阈值,则将对应的关键词链接地址移动到活动桶;如果旧页码数量小于所述页码数量阈值,且新页码数量大于或等于所述页码数量阈值,则将对应的关键词链接地址移动到动态桶如果旧页码数量小于所述页码数量阈值,且新页码数量小于所述页码数量阈值,则对挂起桶进行检索,将挂起桶中调度时间到达当前时间的关键词链接地址移动到活动桶。
- 根据权利要求1所述的基于关键词检索的网络爬虫调度方法,其特征在于,所述允许加入动态桶数量大于允许加入基本桶数量。
- 一种基于关键词检索的网络爬虫调度***,其特征在于,包括:调度端以及与调度端通讯的至少一个抓取节点;所述调度端包括:任务请求命令接收模块,用于接收抓取节点发送的任务请求命令;优先桶任务生成模块,用于从保存二次下载链接地址的优先桶获取二次下载链接地址生成任务并加入任务列表,如果已达到任务列表的允许加入优先桶数量,则执行任务列表返回模块,否则执行动态桶任务生成模块,所述二次下载链接地址为所述抓取节点根据任务列表中的任务进行抓取得到抓取页面,从抓取页面中分析得到需要进行二次下载的链接地址;动态桶任务生成模块,用于保存关键词多页码链接地址的动态桶获取关键词链接地址,并根据与关键词链接地址对应的页码数量衍生出所述页码数量的衍生链接地址,根据所述页码数量的衍生链接地址 生成所述页码数量的任务并加入任务列表,如果已达到任务列表的允许加入动态桶数量,则执行任务列表返回模块,否则执行动态桶任务生成模块,所述关键词链接地址为在目标网站中根据所述关键词所生成的搜索结果页面的链接地址,且动态桶中的关键词链接地址对应的搜索结果页面包括大于或等于预设页码数量阈值的页码数量,所述页码数量阈值大于或等于2;基本桶任务生成模块,用于从保存关键词链接地址的基本桶获取关键词链接地址生成任务并加入任务列表,如果已达到任务列表的允许加入基本桶数量,则执行任务列表返回模块,所述关键词链接地址为在目标网站中根据所述关键词所生成的搜索结果页面的链接地址,且基本桶中的关键词链接地址对应的搜索结果页面包括小于预设页码数量阈值的页码数量,所述页码数量阈值大于或等于2;任务列表返回模块,用于向抓取节点返回任务列表;所述抓取节点包括:任务请求命令发送模块,用于向调度端发送任务请求命令;任务执行模块,用于根据所接收到的任务列表执行任务列表中的任务。
- 根据权利要求9所述的基于关键词检索的网络爬虫调度***,其特征在于,所述优先桶任务生成模块具体用于:从保存二次下载链接地址的优先桶获取二次下载链接地址生成任务并加入任务列表,并从优先桶中删除已生成任务的二次下载链接地址,如果已达到任务列表的允许加入优先桶数量,则执行任务列表返回模块,否则如果优先桶中还保存有二次下载链接地址,则执行优先桶任务生成模块,如果优先桶中所有的二次下载链接地址均已删除,则执行动态桶任务生成模块。
- 根据权利要求9所述的基于关键词检索的网络爬虫调度***,其特征在于,所述动态桶任务生成模块具体用于:从保存关键词链接地址的动态桶获取未调度的关键词链接地址,并根据与关键词链接地 址对应的页码数量衍生出所述页码数量的衍生链接地址,根据所述页码数量的衍生链接地址生成所述页码数量的任务并加入任务列表,并设置已生成任务的关键词链接地址的状态为已调度,如果已达到任务列表的允许加入动态桶数量,则执行任务列表返回模块,并设置动态桶中所有的关键词链接地址的状态为未调度,否则如果动态桶中还保存有未调度的关键词链接地址,则执行动态桶任务生成模块,如果动态桶中未保存有未调度的关键词链接地址,则执行基本桶任务生成模块。
- 根据权利要求9所述的基于关键词检索的网络爬虫调度***,其特征在于,所述基本桶包括活动桶和挂起桶,所述基本桶任务生成模块具体用于,从保存关键词链接地址的活动桶获取调度时间最前的关键词链接地址生成任务并加入任务列表,并将已生成任务的关键词链接地址的调度时间增加预设的调度时间增加量后移动到挂起桶,如果已达到任务列表的允许加入基本桶数量,则执行任务列表返回模块,否则如果活动桶中还保存有关键词链接地址,则执行基本桶任务生成模块,如果活动桶中未保存关键词链接地址,则执行任务列表返回模块。
- 根据权利要求9所述的基于关键词检索的网络爬虫调度***,其特征在于:所述任务执行模块具体用于:根据任务列表中的任务进行抓取得到抓取页面,对所述抓取页面进行分析得到包括二次下载链接地址、信息详情或页码数量的分析数据,将分析数据发送到调度端;调度端还包括分析数据接收模块,用于接收到分析数据:如果分析数据为二次下载链接地址,则将二次下载链接地址放入优先桶;如果分析数据为信息详情,则将信息详情放入数据桶;如果分析数据为页码数量,则在动态桶和基本桶中调整与所述页码数量对应的关键词链接地址。
- 根据权利要求13所述的基于关键词检索的网络爬虫调度***,其特征在于,分析数据接收模块在动态桶和基本桶中调整与所述页码数量对应的关键词链接地址具体包括:设置当前接收到的页码数量为新页码数量,设置上一次同一关键词链接地址接收到的页码数量为旧页码数量;如果旧页码数量与新页码数量不一致,则:如果旧页码数量大于或等于所述页码数量阈值,且新页码数量大于或等于所述页码数量阈值,则修改对应的关键词链接地址对应的页码数量为新页码数量;如果旧页码数量大于或等于所述页码数量阈值,且新页码数量小于所述页码数量阈值,则将对应的关键词链接地址移动到基本桶;如果旧页码数量小于所述页码数量阈值,且新页码数量大于或等于所述页码数量阈值,则将对应的关键词链接地址移动到动态桶。
- 根据权利要求12所述的基于关键词检索的网络爬虫调度***,其特征在于:所述任务执行模块具体用于:根据任务列表中的任务进行抓取得到抓取页面,对所述抓取页面进行分析得到包括二次下载链接地址、信息详情或页码数量的分析数据,将分析数据发送到调度端;调度端还包括分析数据接收模块,用于接收到分析数据:如果分析数据为二次下载链接地址,则将二次下载链接地址放入优先桶;如果分析数据为信息详情,则将信息详情放入数据桶;如果分析数据为页码数量,则设置当前接收到的页码数量为新页码数量,设置上一次同一关键词链接地址接收到的页码数量为旧页码数量,且如果旧页码数量与新页码数量不一致,则:如果旧页码数量大于或等于所述页码数量阈值,且新页码数量大于或等于所述页码数量阈值,则修改对应的关键词链接地址对应的页码数量为新页码数量;如果旧页码数量大于或等于所述页码数量阈值,且新页码数量小于所述页码数量阈值,则将对应的关键词链接地址移动到活动桶;如果旧页码数量小于所述页码数量阈值,且新页码数量大于或等于所述页码数量阈值,则将对应的关键词链接地址移动到动态桶如果旧页码数量小于所述页码数量阈值,且新页码数量小于所述页码数量阈值,则对挂起桶进行检索,将挂起桶中调度时间到达当前时间的关键词链接地址移动到活动桶。
- 根据权利要求9所述的基于关键词检索的网络爬虫调度***,其特征在于,所述允许加入动态桶数量大于允许加入基本桶数量。
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/110,564 US10185771B2 (en) | 2014-01-09 | 2015-01-09 | Method and system for scheduling web crawlers according to keyword search |
RU2016132573A RU2645266C1 (ru) | 2014-01-09 | 2015-01-09 | Способ и устройство для планирования web-обходчиков в соответствии с поиском по ключевым словам |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201410011079.8A CN103761279B (zh) | 2014-01-09 | 2014-01-09 | 一种基于关键词检索的网络爬虫调度方法及*** |
CN201410011079.8 | 2014-01-09 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2015103997A1 true WO2015103997A1 (zh) | 2015-07-16 |
Family
ID=50528516
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2015/070475 WO2015103997A1 (zh) | 2014-01-09 | 2015-01-09 | 一种基于关键词检索的网络爬虫调度方法及*** |
Country Status (4)
Country | Link |
---|---|
US (1) | US10185771B2 (zh) |
CN (1) | CN103761279B (zh) |
RU (1) | RU2645266C1 (zh) |
WO (1) | WO2015103997A1 (zh) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111460254A (zh) * | 2020-03-24 | 2020-07-28 | 南阳柯丽尔科技有限公司 | 一种基于多线程的网页爬取方法、装置、存储介质及设备 |
CN113965555A (zh) * | 2021-10-21 | 2022-01-21 | 北京值得买科技股份有限公司 | 一种参数化爬虫下载方法、装置、设备及存储介质 |
Families Citing this family (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103761279B (zh) * | 2014-01-09 | 2017-02-08 | 北京京东尚科信息技术有限公司 | 一种基于关键词检索的网络爬虫调度方法及*** |
CN105989151B (zh) * | 2015-03-02 | 2019-09-06 | 阿里巴巴集团控股有限公司 | 网页抓取方法及装置 |
CN107784036A (zh) * | 2016-08-31 | 2018-03-09 | 北京国双科技有限公司 | 网络爬虫***和基于网络爬虫***的数据处理方法 |
US20180189409A1 (en) | 2016-12-30 | 2018-07-05 | Caavo Inc | Targeted crawler to develop and/or maintain a searchable database of media content across multiple content providers |
CN107480297A (zh) * | 2017-08-30 | 2017-12-15 | 福建中金在线信息科技有限公司 | 一种文章收录方法及装置 |
CN110309403B (zh) * | 2018-03-05 | 2022-11-04 | 百度在线网络技术(北京)有限公司 | 用于抓取数据的方法和装置 |
GB2572544A (en) * | 2018-03-27 | 2019-10-09 | Innoplexus Ag | System and method of crawling a wide area computer network for retrieving contextual information |
CN108647342A (zh) * | 2018-05-14 | 2018-10-12 | 佛山市真觉网络科技有限公司 | 一种优化百度蜘蛛抓取的方法 |
CN111371705B (zh) * | 2020-02-24 | 2023-09-12 | 维沃移动通信有限公司 | 一种下载任务执行方法及电子设备 |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6691108B2 (en) * | 1999-12-14 | 2004-02-10 | Nec Corporation | Focused search engine and method |
US7454430B1 (en) * | 2004-06-18 | 2008-11-18 | Glenbrook Networks | System and method for facts extraction and domain knowledge repository creation from unstructured and semi-structured documents |
CN102646129A (zh) * | 2012-03-09 | 2012-08-22 | 武汉大学 | 一种主题相关的分布式网络爬虫*** |
CN103279492A (zh) * | 2013-04-28 | 2013-09-04 | 乐视网信息技术(北京)股份有限公司 | 一种抓取网页的方法和装置 |
CN103761279A (zh) * | 2014-01-09 | 2014-04-30 | 北京京东尚科信息技术有限公司 | 一种基于关键词检索的网络爬虫调度方法及*** |
Family Cites Families (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7305610B1 (en) * | 2000-04-06 | 2007-12-04 | Google, Inc. | Distributed crawling of hyperlinked documents |
US7512878B2 (en) * | 2004-04-30 | 2009-03-31 | Microsoft Corporation | Modular document format |
US20150066895A1 (en) * | 2004-06-18 | 2015-03-05 | Glenbrook Networks | System and method for automatic fact extraction from images of domain-specific documents with further web verification |
US7769742B1 (en) * | 2005-05-31 | 2010-08-03 | Google Inc. | Web crawler scheduler that utilizes sitemaps from websites |
US8875249B2 (en) * | 2006-03-01 | 2014-10-28 | Oracle International Corporation | Minimum lifespan credentials for crawling data repositories |
CN100520778C (zh) * | 2006-07-25 | 2009-07-29 | 腾讯科技(深圳)有限公司 | 一种互联网主题文件搜索方法、爬虫***和搜索引擎 |
US20090254512A1 (en) * | 2008-04-03 | 2009-10-08 | Yahoo! Inc. | Ad matching by augmenting a search query with knowledge obtained through search engine results |
JP2012502325A (ja) * | 2008-09-10 | 2012-01-26 | ジュンヒュン スン | デバイスインターフェイシングのための多重モード調音統合 |
TW201013433A (en) * | 2008-09-19 | 2010-04-01 | Esobi Inc | Filtering method for the same or similar documents |
US20120072232A1 (en) * | 2010-04-28 | 2012-03-22 | Patrick Frankham | Systems and Methods for Using Online Resources to Design a Clinical Study and Recruit Participants |
US9246932B2 (en) * | 2010-07-19 | 2016-01-26 | Sitelock, Llc | Selective website vulnerability and infection testing |
US8868541B2 (en) * | 2011-01-21 | 2014-10-21 | Google Inc. | Scheduling resource crawls |
EP2812843A1 (en) * | 2012-02-09 | 2014-12-17 | AOL Inc. | Systems and methods for testing online systems and content |
-
2014
- 2014-01-09 CN CN201410011079.8A patent/CN103761279B/zh active Active
-
2015
- 2015-01-09 WO PCT/CN2015/070475 patent/WO2015103997A1/zh active Application Filing
- 2015-01-09 US US15/110,564 patent/US10185771B2/en active Active
- 2015-01-09 RU RU2016132573A patent/RU2645266C1/ru active
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6691108B2 (en) * | 1999-12-14 | 2004-02-10 | Nec Corporation | Focused search engine and method |
US7454430B1 (en) * | 2004-06-18 | 2008-11-18 | Glenbrook Networks | System and method for facts extraction and domain knowledge repository creation from unstructured and semi-structured documents |
CN102646129A (zh) * | 2012-03-09 | 2012-08-22 | 武汉大学 | 一种主题相关的分布式网络爬虫*** |
CN103279492A (zh) * | 2013-04-28 | 2013-09-04 | 乐视网信息技术(北京)股份有限公司 | 一种抓取网页的方法和装置 |
CN103761279A (zh) * | 2014-01-09 | 2014-04-30 | 北京京东尚科信息技术有限公司 | 一种基于关键词检索的网络爬虫调度方法及*** |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111460254A (zh) * | 2020-03-24 | 2020-07-28 | 南阳柯丽尔科技有限公司 | 一种基于多线程的网页爬取方法、装置、存储介质及设备 |
CN111460254B (zh) * | 2020-03-24 | 2023-05-05 | 南阳柯丽尔科技有限公司 | 一种基于多线程的网页爬取方法、装置、存储介质及设备 |
CN113965555A (zh) * | 2021-10-21 | 2022-01-21 | 北京值得买科技股份有限公司 | 一种参数化爬虫下载方法、装置、设备及存储介质 |
CN113965555B (zh) * | 2021-10-21 | 2024-04-12 | 北京值得买科技股份有限公司 | 一种参数化爬虫下载方法、装置、设备及存储介质 |
Also Published As
Publication number | Publication date |
---|---|
CN103761279A (zh) | 2014-04-30 |
RU2645266C1 (ru) | 2018-02-19 |
US20160328475A1 (en) | 2016-11-10 |
US10185771B2 (en) | 2019-01-22 |
CN103761279B (zh) | 2017-02-08 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2015103997A1 (zh) | 一种基于关键词检索的网络爬虫调度方法及*** | |
CN109582660B (zh) | 数据血缘分析方法、装置、设备、***及可读存储介质 | |
KR102133951B1 (ko) | 짧은 링크 처리 방법, 디바이스, 및 서버 | |
CN107590188B (zh) | 一种自动化垂直细分领域的爬虫爬取方法及其管理*** | |
CN107273409B (zh) | 一种网络数据采集、存储及处理方法及*** | |
CN104903894B (zh) | 用于分布式数据库查询引擎的***和方法 | |
CN103902386B (zh) | 一种基于连接代理优化管理的多线程网络爬虫处理方法 | |
US10089143B2 (en) | Dynamic scheduling of tasks for collecting and processing data using job configuration data | |
KR101213930B1 (ko) | 결정 이론 웹 크롤링, 및 웹 페이지 변경의 예측 | |
US9064013B1 (en) | Application of resource limits to request processing | |
WO2018095351A1 (zh) | 搜索处理方法及装置 | |
US20150234927A1 (en) | Application search method, apparatus, and terminal | |
US20160147830A1 (en) | Managing datasets produced by alert-triggering search queries | |
CN104202360A (zh) | 访问网页的方法、装置及路由器 | |
WO2010041517A1 (ja) | 情報収集装置、検索エンジン、情報収集方法およびプログラム | |
CN102355488A (zh) | 爬虫种子获取方法与设备及爬虫爬取方法与设备 | |
WO2018035799A1 (zh) | 数据查询方法、应用和数据库服务器、中间件及*** | |
Dixit et al. | A novel approach to priority based focused crawler | |
CN110908641A (zh) | 基于可视化的流计算平台、方法、设备和存储介质 | |
CN109600385B (zh) | 一种访问控制方法及装置 | |
CN102946320A (zh) | 一种分布式用户行为日志预测网络监管方法及*** | |
Elshater et al. | godiscovery: Web service discovery made efficient | |
CN113656673A (zh) | 面向广告投放的主从分布内容爬取机器人 | |
KR102189127B1 (ko) | 행위 기반 룰 처리 장치 및 그 처리 방법 | |
CN113377791A (zh) | 一种数据处理方法、***及计算设备 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 15735407 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 15110564 Country of ref document: US |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
WWE | Wipo information: entry into national phase |
Ref document number: IDP00201604531 Country of ref document: ID |
|
ENP | Entry into the national phase |
Ref document number: 2016132573 Country of ref document: RU Kind code of ref document: A |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 15735407 Country of ref document: EP Kind code of ref document: A1 |