CN105991699B - 一种互联网爬虫的分布式下载*** - Google Patents
一种互联网爬虫的分布式下载*** Download PDFInfo
- Publication number
- CN105991699B CN105991699B CN201510063839.4A CN201510063839A CN105991699B CN 105991699 B CN105991699 B CN 105991699B CN 201510063839 A CN201510063839 A CN 201510063839A CN 105991699 B CN105991699 B CN 105991699B
- Authority
- CN
- China
- Prior art keywords
- server
- download
- downloading
- dns
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000013507 mapping Methods 0.000 claims description 13
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims description 12
- 238000004891 communication Methods 0.000 claims description 8
- 230000004044 response Effects 0.000 claims description 4
- 238000004806 packaging method and process Methods 0.000 claims description 2
- 238000000151 deposition Methods 0.000 claims 1
- 238000011156 evaluation Methods 0.000 claims 1
- 230000006870 function Effects 0.000 description 10
- 238000000034 method Methods 0.000 description 6
- 230000005540 biological transmission Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
- 238000005303 weighing Methods 0.000 description 1
Landscapes
- Information Transfer Between Computers (AREA)
Abstract
Description
Claims (5)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201510063839.4A CN105991699B (zh) | 2015-02-06 | 2015-02-06 | 一种互联网爬虫的分布式下载*** |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201510063839.4A CN105991699B (zh) | 2015-02-06 | 2015-02-06 | 一种互联网爬虫的分布式下载*** |
Publications (2)
Publication Number | Publication Date |
---|---|
CN105991699A CN105991699A (zh) | 2016-10-05 |
CN105991699B true CN105991699B (zh) | 2019-07-19 |
Family
ID=57037952
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201510063839.4A Expired - Fee Related CN105991699B (zh) | 2015-02-06 | 2015-02-06 | 一种互联网爬虫的分布式下载*** |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN105991699B (zh) |
Families Citing this family (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106502802A (zh) * | 2016-10-12 | 2017-03-15 | 山东浪潮云服务信息科技有限公司 | 一种基于Avro RPC传输的分布式云端并发采集方法 |
CN106941537B (zh) * | 2017-05-05 | 2020-08-07 | 北京奇艺世纪科技有限公司 | 一种负载均衡方法及装置 |
CN110874260B (zh) * | 2018-09-04 | 2023-03-10 | 北京奇虎科技有限公司 | 一种基于多用户机器的文件处理方法和装置 |
CN111324805B (zh) * | 2018-12-13 | 2024-02-13 | 北京搜狗科技发展有限公司 | 查询意图确定方法及装置、搜索方法及搜索引擎 |
CN110365776B (zh) * | 2019-07-17 | 2021-05-04 | 京东方科技集团股份有限公司 | 图片批量下载方法、装置、电子设备及存储介质 |
CN110990675B (zh) * | 2019-11-25 | 2024-07-26 | 爱信诺征信有限公司 | 网页数据爬取方法及*** |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102244810A (zh) * | 2009-07-16 | 2011-11-16 | 北京中传网联数据服务有限公司 | 一种获取数字电视受众信息的方法、装置和*** |
CN102833293A (zh) * | 2011-06-17 | 2012-12-19 | 腾讯科技(深圳)有限公司 | P2sp网络中资源下载的方法及客户端 |
CN103248636A (zh) * | 2012-02-03 | 2013-08-14 | 深圳市腾讯计算机***有限公司 | 离线下载的***及方法 |
CN103338279A (zh) * | 2013-07-18 | 2013-10-02 | 上海数讯信息技术有限公司 | 基于域名解析的优化排序方法及*** |
CN104112207A (zh) * | 2014-07-29 | 2014-10-22 | 浪潮软件集团有限公司 | 一种基于互联网数据的电子商务交易监测方法 |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20060288087A1 (en) * | 2005-06-17 | 2006-12-21 | Jiehyeong Sun | Web-based method and system for providing content-driven service to internet users |
-
2015
- 2015-02-06 CN CN201510063839.4A patent/CN105991699B/zh not_active Expired - Fee Related
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102244810A (zh) * | 2009-07-16 | 2011-11-16 | 北京中传网联数据服务有限公司 | 一种获取数字电视受众信息的方法、装置和*** |
CN102833293A (zh) * | 2011-06-17 | 2012-12-19 | 腾讯科技(深圳)有限公司 | P2sp网络中资源下载的方法及客户端 |
CN103248636A (zh) * | 2012-02-03 | 2013-08-14 | 深圳市腾讯计算机***有限公司 | 离线下载的***及方法 |
CN103338279A (zh) * | 2013-07-18 | 2013-10-02 | 上海数讯信息技术有限公司 | 基于域名解析的优化排序方法及*** |
CN104112207A (zh) * | 2014-07-29 | 2014-10-22 | 浪潮软件集团有限公司 | 一种基于互联网数据的电子商务交易监测方法 |
Also Published As
Publication number | Publication date |
---|---|
CN105991699A (zh) | 2016-10-05 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN105991699B (zh) | 一种互联网爬虫的分布式下载*** | |
Jalaparti et al. | Network-aware scheduling for data-parallel jobs: Plan when you can | |
Biran et al. | A stable network-aware vm placement for cloud systems | |
CN103530189B (zh) | 一种面向流式数据的自动伸缩及迁移的方法及装置 | |
JP4527976B2 (ja) | ホストされるアプリケーションのためのサーバ・リソース管理 | |
CN104486445B (zh) | 一种基于云平台的分布式可扩展资源监控*** | |
US10560544B2 (en) | Data caching in a collaborative file sharing system | |
CN107241384A (zh) | 一种基于多云架构的内容分发服务资源优化调度方法 | |
Wang et al. | Exploiting spot and burstable instances for improving the cost-efficacy of in-memory caches on the public cloud | |
CN101501669A (zh) | 域名解析资源分配 | |
CN103607424B (zh) | 一种服务器连接方法及服务器*** | |
CN111225059A (zh) | 网络请求资源调度方法、装置、电子设备及存储介质 | |
CN103747274B (zh) | 一种增设缓存集群的视频数据中心及其缓存资源调度方法 | |
McClatchey et al. | Data intensive and network aware (DIANA) grid scheduling | |
CN104679594B (zh) | 一种中间件分布式计算方法 | |
US10346367B1 (en) | Load shedding techniques for distributed services with persistent client connections to ensure quality of service | |
EP2791819A1 (en) | Content delivery network | |
CN105159775A (zh) | 基于负载均衡器的云计算数据中心的管理***和管理方法 | |
Zhang et al. | Netkv: Scalable, self-managing, load balancing as a network function | |
Wang et al. | Bandwidth guaranteed virtual network function placement and scaling in datacenter networks | |
Hajjat et al. | Dealer: application-aware request splitting for interactive cloud applications | |
CN106254561A (zh) | 一种网络资源文件的实时离线下载方法及*** | |
CN108667744A (zh) | 流量控制方法及装置 | |
Simoncelli et al. | Stream-monitoring with blockmon: convergence of network measurements and data analytics platforms | |
Sundarrajan et al. | Midgress-aware traffic provisioning for content delivery |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
C06 | Publication | ||
PB01 | Publication | ||
C10 | Entry into substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
TA01 | Transfer of patent application right |
Effective date of registration: 20170426 Address after: 100086 Beijing, Haidian District, North Third Ring Road West, No. 43, building 5, floor 08-09, No. 2 Applicant after: BEIJING ZHONGSOU CLOUD BUSINESS NETWORK TECHNOLOGY Co.,Ltd. Address before: Shou Heng Technology Building No. 51 Beijing 100191 Haidian District Xueyuan Road room 0902 Applicant before: BEIJING ZHONGSOU NETWORK TECHNOLOGY Co.,Ltd. |
|
TA01 | Transfer of patent application right | ||
TA01 | Transfer of patent application right | ||
TA01 | Transfer of patent application right |
Effective date of registration: 20180130 Address after: 102400 Beijing city Fangshan District Chenguang Road No. 16 Building No. 16 hospital 6 layer 612 Applicant after: Beijing Zhongsoyunyue Network Technology Co.,Ltd. Address before: 100086 Beijing, Haidian District, North Third Ring Road West, No. 43, building 5, floor 08-09, No. 2 Applicant before: BEIJING ZHONGSOU CLOUD BUSINESS NETWORK TECHNOLOGY Co.,Ltd. |
|
GR01 | Patent grant | ||
GR01 | Patent grant | ||
CP01 | Change in the name or title of a patent holder |
Address after: 102400 Beijing city Fangshan District Chenguang Road No. 16 Building No. 16 hospital 6 layer 612 Patentee after: Beijing Yunyue Gongchuang Network Technology Co.,Ltd. Address before: 102400 Beijing city Fangshan District Chenguang Road No. 16 Building No. 16 hospital 6 layer 612 Patentee before: Beijing Zhongsoyunyue Network Technology Co.,Ltd. |
|
CP01 | Change in the name or title of a patent holder | ||
CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20190719 Termination date: 20220206 |
|
CF01 | Termination of patent right due to non-payment of annual fee |