CN110287395A - A kind of crawler method, apparatus, system, equipment and readable storage medium storing program for executing - Google Patents
A kind of crawler method, apparatus, system, equipment and readable storage medium storing program for executing Download PDFInfo
- Publication number
- CN110287395A CN110287395A CN201910584929.6A CN201910584929A CN110287395A CN 110287395 A CN110287395 A CN 110287395A CN 201910584929 A CN201910584929 A CN 201910584929A CN 110287395 A CN110287395 A CN 110287395A
- Authority
- CN
- China
- Prior art keywords
- crawler
- mobile terminal
- platform
- result
- address
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mobile Radio Communication Systems (AREA)
- Telephonic Communication Services (AREA)
Abstract
This application discloses a kind of crawler methods, applied to any one mobile terminal for establishing communication connection with crawler platform, comprising: receive the crawler task that crawler platform is sent, crawler task includes at least: target to be crawled and crawler rule；Target to be crawled is crawled using the IP address and crawler rule of mobile terminal, obtains crawler result corresponding with crawler task；IP address is that the mobile communications network of mobile terminal connection is the IP address of mobile terminal distribution；Crawler result is transmitted to crawler platform, so that crawler platform stores crawler result.IP address used in this application is the dynamic IP addressing that mobile communications network is mobile terminal distribution；And mobile terminal be it is multiple, therefore reduce by crawler target interception or block crawler operation used in IP address a possibility that, also provide possibility for the high concurrent of crawler.A kind of crawler device, system, equipment and readable storage medium storing program for executing disclosed in the present application, similarly have above-mentioned technique effect.
Description
Technical field
This application involves Internet technical field, in particular to a kind of crawler method, apparatus, system, equipment and readable deposit
Storage media.
Background technique
Crawler is the mode for automatically grabbing web message according to certain rules, but due to usually being set by the target of crawler
There is the security mechanism of anti-crawler, therefore in existing crawler scheme, it will usually utilize the IP agent pool for being equipped with multiple IP address
Carry out crawler, the IP address that replacement crawls some target can be changed frequently in this way, to be avoided as far as possible by crawler target interception or
Block IP address used in crawler operation.
But the IP address in IP agent pool needs constantly maintenance, quantity is limited after all, when more by the target of crawler
When, the IP address quantity in IP agent pool just seems awkward, so existing IP agent pool can not be well solved and be climbed
The problem of IP address used in worm target interception or block crawler operation.Meanwhile in the IP agent pool in existing crawler scheme
IP address be only used as Agent IP, needs could be completed to crawl process by other equipment, this undoubtedly increases crawler operation
Complexity, be unfavorable for the execution of the crawler operation of high concurrent.
Therefore, the IP address used in crawler target interception or block crawler operation how is avoided, is art technology
Personnel's problem to be solved.
Summary of the invention
In view of this, the application's is designed to provide a kind of crawler method, apparatus, system, equipment and readable storage medium
Matter, to avoid the IP address used in crawler target interception or block crawler operation.Its concrete scheme is as follows:
In a first aspect, this application provides a kind of crawler method, applied to any for establishing communication connection with crawler platform
A mobile terminal, comprising:
The crawler task that crawler platform is sent is received, crawler task includes at least: target to be crawled and crawler rule；
Target to be crawled is crawled using the IP address and crawler rule of mobile terminal, obtains climb corresponding with crawler task
Worm result；IP address are as follows: the mobile communications network of mobile terminal connection is the IP address of mobile terminal distribution；
Crawler result is transmitted to crawler platform, so that crawler platform stores crawler result.
Preferably, crawler result is transmitted to crawler platform, so that crawler platform stores crawler result, comprising:
The ID of crawler result and mobile terminal is transmitted to crawler platform, so that crawler platform stores crawler result and movement
The ID of terminal.
Preferably, mobile communications network 2G, 3G, 4G or 5G network.
Second aspect, this application provides a kind of crawler devices, applied to any for establishing communication connection with crawler platform
A mobile terminal, comprising:
Receiving module, for receiving the crawler task of crawler platform transmission, crawler task is included at least: target to be crawled
With crawler rule；
Crawler module, for using mobile terminal IP address and crawler rule crawl target to be crawled, obtain and climb
The corresponding crawler result of worm task；IP address are as follows: the IP that the mobile communications network of mobile terminal connection distributes for mobile terminal
Location；
Transmission module, for crawler result to be transmitted to crawler platform, so that crawler platform stores crawler result.
The third aspect, this application provides a kind of crawler systems, comprising: crawler platform and multiple crawler dresses disclosed above
It sets.
Preferably, crawler platform includes:
Task management center, target to be crawled and crawler rule for being inputted according to user generate crawler task；
Control centre, for sending crawler task to any one crawler device；
Storage center, for storing the corresponding crawler result of each crawler task.
Preferably, storage center stores crawler result using non-relational database.
Preferably, crawler platform is built based on WEB server.
Fourth aspect, this application provides a kind of crawler equipment, comprising:
Memory, for storing computer program；
Processor, for executing computer program, to realize aforementioned disclosed crawler method.
5th aspect, this application provides a kind of readable storage medium storing program for executing, for saving computer program, wherein computer
Aforementioned disclosed crawler method is realized when program is executed by processor.
By above scheme it is found that this application provides a kind of crawler method, communication link is established applied to crawler platform
Any one mobile terminal connect, comprising: receive the crawler task that crawler platform is sent, crawler task includes at least: to be crawled
Target and crawler rule；Target to be crawled is crawled using the IP address and crawler rule of mobile terminal, is obtained and crawler task
Corresponding crawler result；IP address are as follows: the mobile communications network of mobile terminal connection is the IP address of mobile terminal distribution；It will climb
Worm result is transmitted to crawler platform, so that crawler platform stores crawler result.
Wherein, the application utilizes any one mobile terminal execution crawler task that communication connection is established with crawler platform, by
It is distributed in the mobile communications network that the IP address of mobile terminal is mobile terminal connection for mobile terminal, therefore the IP address is
Dynamic IP addressing.So it is dynamic for accessing the IP address of target to be crawled for a mobile terminal；And with crawler
Platform establish communication connection mobile terminal be it is multiple, therefore execute crawler task IP address by flexible transformation, so also
A possibility that reducing the IP address used in crawler target interception or block crawler operation.Meanwhile the movement in the application
Terminal can use itself and complete entire crawler process, carrys out transmission back so as to avoid the data during crawler, reduces and climb
The complexity of worm operation, also provides possibility for the high concurrent of crawler.
Correspondingly, a kind of crawler device, system, equipment and readable storage medium storing program for executing provided by the present application similarly have upper
State technical effect.
Detailed description of the invention
In order to illustrate the technical solutions in the embodiments of the present application or in the prior art more clearly, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
The embodiment of application for those of ordinary skill in the art without creative efforts, can also basis
The attached drawing of offer obtains other attached drawings.
Fig. 1 is a kind of crawler method flow diagram disclosed in the present application；
Fig. 2 is a kind of crawler schematic device disclosed in the present application；
Fig. 3 is a kind of crawler system schematic diagram disclosed in the present application；
Fig. 4 is a kind of crawler equipment schematic diagram disclosed in the present application.
Specific embodiment
Below in conjunction with the attached drawing in the embodiment of the present application, technical solutions in the embodiments of the present application carries out clear, complete
Site preparation description, it is clear that described embodiments are only a part of embodiments of the present application, instead of all the embodiments.It is based on
Embodiment in the application, it is obtained by those of ordinary skill in the art without making creative efforts every other
Embodiment shall fall in the protection scope of this application.
Currently, the IP address in IP agent pool needs constantly maintenance, quantity is limited after all, when more by the target of crawler
When, the IP address quantity in IP agent pool just seems awkward, so existing IP agent pool can not be well solved and be climbed
The problem of IP address used in worm target interception or block crawler operation.Meanwhile in the IP agent pool in existing crawler scheme
IP address be only used as Agent IP, can not really crawl data, need by other equipment assist could complete to crawl
Journey, this undoubtedly increases the complexity of crawler operation, is unfavorable for the execution of the crawler operation of high concurrent.For this purpose, the application provides
A kind of crawler scheme, a possibility that can reduce the IP address used in crawler target interception or block crawler operation,
Possibility is provided for the high concurrent of crawler.
Shown in Figure 1, the embodiment of the present application discloses a kind of crawler method, establishes communication link applied to crawler platform
Any one mobile terminal connect, comprising:
S101, the crawler task that crawler platform is sent is received, crawler task includes at least: target to be crawled and crawler rule
Then；
S102, target to be crawled is crawled using the IP address and crawler rule of mobile terminal, obtain and crawler task pair
The crawler result answered；IP address are as follows: the mobile communications network of mobile terminal connection is the IP address of mobile terminal distribution；
S103, crawler result is transmitted to crawler platform, so that crawler platform stores crawler result.
In the present embodiment, crawler result is transmitted to crawler platform, so that crawler platform stores crawler result, comprising:
The ID of crawler result and mobile terminal is transmitted to crawler platform, so that crawler platform stores crawler result and mobile terminal
ID。
Specifically, self ID can be also transmitted to crawler platform, climbed in this way by mobile terminal while transmitting crawler result
Worm platform can be by crawler result and the corresponding storage of ID, in order to the inquiry and management of crawler result.
It should be noted that target to be crawled can be specially web site url.Crawler rule can be used canonical sentence and set
It is fixed, and crawler rule is write using any programming language (such as python).Crawler task at least further includes following information: creation should
ID, ID of crawler task of the account of crawler task etc., these information can be sent to mobile terminal together.Wherein, user can be with
Crawler task is created in crawler platform based on crawler platform login account, and based on the account, to can incite somebody to action in crawler platform
The crawler task of different user creation is distributed to different mobile terminals, to utilize each crawler task of mobile terminal execution.
In the present embodiment, mobile communications network 2G, 3G, 4G or 5G network.It should be noted that mobile communications network
IP address can be distributed to access the mobile terminal of the network, and be allocated in a dynamic fashion.That is: it is directed to a certain mobile terminal,
Its used IP address is dynamic change.Rule change can be with are as follows: if mobile terminal accessing mobile communication network for the first time,
IP address is selected in preset IP address Chi Zhongwei current mobile terminal.In mobile terminal accessing mobile communication network again
When, if the time interval for being currently accessed time and last time suspension time is more than preset time threshold, for current mobile terminal
Redistribute IP address；Otherwise, continue to continue to use the IP address of last time.This is a kind of exemplary rule change of the application, can also be with
According to the IP address of other rule changes transformation mobile terminal, other rule changes can refer to the prior art, therefore the present embodiment
No longer illustrate one by one herein.
Mobile terminal can utilize SIM card (Subscriber for mobile phone, wrist-watch, bracelet, plate, computer etc.
Identification Module, client identification module) accessing mobile communication network equipment.
As it can be seen that the embodiment of the present application utilizes any one the mobile terminal execution crawler for establishing communication connection with crawler platform to appoint
Business is the IP address of mobile terminal distribution since the IP address of mobile terminal is the mobile communications network of mobile terminal connection, because
This IP address is dynamic IP addressing.So the IP address for accessing target to be crawled is dynamic for a mobile terminal
's；And it is multiple for establishing the mobile terminal of communication connection with crawler platform, therefore the IP address for executing crawler task will be flexible
Transformation, a possibility that also just reducing by crawler target interception or block IP address used in crawler operation.Meanwhile the application
In mobile terminal can use itself and complete entire crawler process, carry out transmission back so as to avoid the data during crawler,
The complexity of crawler operation is reduced, also provides possibility for the high concurrent of crawler.
A kind of crawler device provided by the embodiments of the present application is introduced below, a kind of crawler device described below with
A kind of above-described crawler method can be cross-referenced.
Shown in Figure 2, the embodiment of the present application discloses a kind of crawler device, establishes communication link applied to crawler platform
Any one mobile terminal connect, comprising:
Receiving module 201, for receiving the crawler task of crawler platform transmission, crawler task is included at least: to be crawled
Target and crawler rule；
Crawler module 202, for using mobile terminal IP address and crawler rule crawl target to be crawled, obtain with
The corresponding crawler result of crawler task；IP address are as follows: the mobile communications network of mobile terminal connection is the IP of mobile terminal distribution
Address；
Transmission module 203, for crawler result to be transmitted to crawler platform, so that crawler platform stores crawler result.
In a specific embodiment, transmission module is specifically used for:
The ID of crawler result and mobile terminal is transmitted to crawler platform, so that crawler platform stores crawler result and movement
The ID of terminal.
Wherein, previous embodiment can be referred to by closing the more specifical course of work of modules, unit in this present embodiment
Disclosed in corresponding contents, no longer repeated herein.
It should be noted that the device can be specially the application program of installation on mobile terminals.
As it can be seen that present embodiments providing a kind of crawler device, comprising: receiving module, crawler module and transmission module.It is first
The crawler task that crawler platform is sent first is received by receiving module, crawler task includes at least: target to be crawled and crawler rule
Then；Then crawler module crawls target to be crawled using the IP address and crawler rule of mobile terminal, obtains and crawler task
Corresponding crawler result；Crawler result is transmitted to crawler platform by last transmission module, so that crawler platform stores crawler result.
Share out the work and help one another between such modules, Each performs its own functions, so that reducing is made by crawler target interception or block crawler operation
A possibility that IP address, also provides possibility for the high concurrent of crawler.
A kind of crawler system provided by the embodiments of the present application is introduced below, a kind of crawler system described below with
A kind of above-described crawler method and device can be cross-referenced.
Shown in Figure 3, the embodiment of the present application discloses a kind of crawler system, comprising: crawler platform and multiple above-mentioned public affairs
The crawler device opened, such as crawler device 1, crawler device 2 ... the crawler device N in Fig. 3；
Wherein, crawler platform includes:
Task management center 301, target to be crawled and crawler rule for being inputted according to user generate crawler task；
Control centre 302, for sending crawler task to any one crawler device；
Storage center 303, for storing the corresponding crawler result of each crawler task.
In a specific embodiment, storage center stores crawler result using non-relational database.Non-relational
Database can optimize storage organization, improve query performance, such as elasticsearch PostgreSQL database.
In a specific embodiment, crawler platform is built based on WEB server, build environment can for linux,
Apache, mysql or php etc..
It should be noted that crawler platform and mobile terminal pass through hypertext transfer protocol (HTTP, Hyper Text
Transfer Protocol) carry out data interaction, that is to say, that and crawler task and crawler result are carried out in the form of HTTP message
Transmission.It establishes and communicates to connect when mobile terminal and crawler platform, crawler platform can detect the mobile terminal, and be the shifting
Dynamic terminal distribution crawler task simultaneously issues.After mobile terminal completes a crawler task, crawler platform can be mobile terminal point
With new crawler task.If desired the IP address for replacing mobile terminal, then make according to the connected mobile communications network of the mobile terminal
The IP of IP address rule change transformation current mobile terminal.Specifically, mobile terminal by mobile communications network with
Crawler platform establishes communication connection.
When the crawler of user query some crawler tasks is as a result, the inquiry request that then crawler platform is inputted according to user is being deposited
Corresponding crawler is inquired as a result, and visualizing the crawler result inquired in storage center.
As it can be seen that crawler platform is collected for collecting, distributing crawler task in crawler system provided in this embodiment
The crawler result of mobile terminal transmission.Wherein, complete crawler task using mobile terminal, can reduce by crawler target interception or
A possibility that blocking IP address used in crawler operation, also provides possibility for the high concurrent of crawler.
A kind of crawler equipment provided by the embodiments of the present application is introduced below, a kind of crawler equipment described below with
A kind of above-described crawler method, apparatus and system can be cross-referenced.
Shown in Figure 4, the embodiment of the present application discloses a kind of crawler equipment, comprising:
Memory 401, for saving computer program；
Processor 402, for executing the computer program, to perform the steps of
The crawler task that crawler platform is sent is received, crawler task includes at least: target to be crawled and crawler rule；Benefit
Target to be crawled is crawled with the IP address of mobile terminal and crawler rule, obtains crawler result corresponding with crawler task；IP
Address are as follows: the mobile communications network of mobile terminal connection is the IP address of mobile terminal distribution；Crawler result is transmitted to crawler
Platform, so that crawler platform stores crawler result.
It in the present embodiment, can be specific when the processor executes the computer subprogram saved in the memory
It performs the steps of and the ID of crawler result and mobile terminal is transmitted to crawler platform, so that crawler platform stores crawler result
With the ID of mobile terminal.
A kind of readable storage medium storing program for executing provided by the embodiments of the present application is introduced below, one kind described below is readable to deposit
Storage media can be cross-referenced with a kind of above-described crawler method, apparatus, system and equipment.
A kind of readable storage medium storing program for executing, for saving computer program, wherein when the computer program is executed by processor
Realize crawler method disclosed in previous embodiment.Specific steps about this method can be with reference to phase disclosed in previous embodiment
Content is answered, is no longer repeated herein.
This application involves " first ", " second ", " third ", the (if present)s such as " the 4th " be for distinguishing similar right
As without being used to describe a particular order or precedence order.It should be understood that the data used in this way in the appropriate case can be with
It exchanges, so that the embodiments described herein can be implemented with the sequence other than the content for illustrating or describing herein.In addition,
Term " includes " and " having " and their any deformation, it is intended that cover it is non-exclusive include, for example, containing a system
The process, method or equipment of column step or unit those of are not necessarily limited to be clearly listed step or unit, but may include not having
There are other step or units being clearly listed or intrinsic for these process, methods or equipment.
It should be noted that the description for being related to " first ", " second " etc. in this application is used for description purposes only, and cannot
It is interpreted as its relative importance of indication or suggestion or implicitly indicates the quantity of indicated technical characteristic.Define as a result, " the
One ", the feature of " second " can explicitly or implicitly include at least one of the features.In addition, the skill between each embodiment
Art scheme can be combined with each other, but must be based on can be realized by those of ordinary skill in the art, when technical solution
Will be understood that the combination of this technical solution is not present in conjunction with there is conflicting or cannot achieve when, also not this application claims
Protection scope within.
Each embodiment in this specification is described in a progressive manner, the highlights of each of the examples are with it is other
The difference of embodiment, same or similar part may refer to each other between each embodiment.
The step of method described in conjunction with the examples disclosed in this document or algorithm, can directly be held with hardware, processor
The combination of capable software module or the two is implemented.Software module can be placed in random access memory (RAM), memory, read-only deposit
Reservoir (ROM), electrically programmable ROM, electrically erasable ROM, register, hard disk, moveable magnetic disc, CD-ROM or technology
In any other form of readable storage medium storing program for executing well known in field.
Specific examples are used herein to illustrate the principle and implementation manner of the present application, and above embodiments are said
It is bright to be merely used to help understand the present processes and its core concept；At the same time, for those skilled in the art, foundation
The thought of the application, there will be changes in the specific implementation manner and application range, in conclusion the content of the present specification is not
It is interpreted as the limitation to the application.
Claims (10)
1. a kind of crawler method, which is characterized in that applied to any one mobile terminal for establishing communication connection with crawler platform, packet
It includes:
The crawler task that the crawler platform is sent is received, the crawler task includes at least: target to be crawled and crawler rule
Then；
The target to be crawled is crawled using the IP address and the crawler rule of the mobile terminal, is obtained and the crawler
The corresponding crawler result of task；The IP address are as follows: the mobile communications network of the mobile terminal connection is the mobile terminal
The IP address of distribution；
The crawler result is transmitted to the crawler platform, so that the crawler platform stores the crawler result.
2. crawler method according to claim 1, which is characterized in that described the crawler result is transmitted to crawler to put down
Platform, so that the crawler platform stores the crawler result, comprising:
The ID of the crawler result and the mobile terminal is transmitted to the crawler platform, so that the crawler platform stores institute
State the ID of crawler result and the mobile terminal.
3. crawler method according to claim 1 or 2, which is characterized in that the mobile communications network be 2G, 3G, 4G or
5G network.
4. a kind of crawler device, which is characterized in that applied to any one mobile terminal for establishing communication connection with crawler platform, packet
It includes:
Receiving module, the crawler task sent for receiving the crawler platform, the crawler task include at least: to be crawled
Target and crawler rule；
Crawler module, for using the mobile terminal IP address and the crawler rule crawl the target to be crawled,
Obtain crawler result corresponding with the crawler task；The IP address are as follows: the mobile communications network of the mobile terminal connection
For the IP address of mobile terminal distribution；
Transmission module, for the crawler result to be transmitted to the crawler platform, to be climbed described in crawler platform storage
Worm result.
5. a kind of crawler system characterized by comprising crawler platform and multiple crawler devices as claimed in claim 4.
6. crawler system according to claim 5, which is characterized in that the crawler platform includes:
Task management center, target to be crawled and crawler rule for being inputted according to user generate crawler task；
Control centre, for sending the crawler task to any one crawler device；
Storage center, for storing the corresponding crawler result of each crawler task.
7. crawler system according to claim 6, which is characterized in that the storage center is deposited using non-relational database
Store up crawler result.
8. according to the described in any item crawler systems of claim 5-7, which is characterized in that the crawler platform is based on WEB service
Device is built.
9. a kind of crawler equipment characterized by comprising
Memory, for storing computer program；
Processor, for executing the computer program, to realize crawler method as described in any one of claims 1 to 3.
10. a kind of readable storage medium storing program for executing, which is characterized in that for saving computer program, wherein the computer program is located
Reason device realizes crawler method as described in any one of claims 1 to 3 when executing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910584929.6A CN110287395A (en) | 2019-07-01 | 2019-07-01 | A kind of crawler method, apparatus, system, equipment and readable storage medium storing program for executing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910584929.6A CN110287395A (en) | 2019-07-01 | 2019-07-01 | A kind of crawler method, apparatus, system, equipment and readable storage medium storing program for executing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110287395A true CN110287395A (en) | 2019-09-27 |
Family
ID=68021407
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910584929.6A Pending CN110287395A (en) | 2019-07-01 | 2019-07-01 | A kind of crawler method, apparatus, system, equipment and readable storage medium storing program for executing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110287395A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110826001A (en) * | 2019-10-30 | 2020-02-21 | 深圳前海环融联易信息科技服务有限公司 | Data acquisition request processing method and device, computer equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2007259408A (en) * | 2006-02-21 | 2007-10-04 | Klab Inc | Access method for network by mobile telephone terminal |
| US20090204478A1 (en) * | 2008-02-08 | 2009-08-13 | Vertical Acuity, Inc. | Systems and Methods for Identifying and Measuring Trends in Consumer Content Demand Within Vertically Associated Websites and Related Content |
| CN108551452A (en) * | 2018-04-18 | 2018-09-18 | 平安科技（深圳）有限公司 | Web crawlers method, terminal and storage medium |
| CN108595510A (en) * | 2018-03-22 | 2018-09-28 | 成都数聚城堡科技有限公司 | A kind of reptile based on browser end, distributed reptile system and method |
| CN109815385A (en) * | 2019-01-31 | 2019-05-28 | 无锡火球普惠信息科技有限公司 | Crawler and crawling method based on APP client |
-
2019
- 2019-07-01 CN CN201910584929.6A patent/CN110287395A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2007259408A (en) * | 2006-02-21 | 2007-10-04 | Klab Inc | Access method for network by mobile telephone terminal |
| US20090204478A1 (en) * | 2008-02-08 | 2009-08-13 | Vertical Acuity, Inc. | Systems and Methods for Identifying and Measuring Trends in Consumer Content Demand Within Vertically Associated Websites and Related Content |
| CN108595510A (en) * | 2018-03-22 | 2018-09-28 | 成都数聚城堡科技有限公司 | A kind of reptile based on browser end, distributed reptile system and method |
| CN108551452A (en) * | 2018-04-18 | 2018-09-18 | 平安科技（深圳）有限公司 | Web crawlers method, terminal and storage medium |
| CN109815385A (en) * | 2019-01-31 | 2019-05-28 | 无锡火球普惠信息科技有限公司 | Crawler and crawling method based on APP client |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110826001A (en) * | 2019-10-30 | 2020-02-21 | 深圳前海环融联易信息科技服务有限公司 | Data acquisition request processing method and device, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6095106B2 (en) | System and method for adaptive selection of bank cards for payment | |
| CN108427581A (en) | System micro services method and terminal device | |
| CN102365890B (en) | Verifiable service billing for intermediate networking devices | |
| CN107767265A (en) | A kind of data sharing method and server | |
| CN104079630A (en) | Business server side load balancing method, client side, server side and system | |
| CN110554927A (en) | Micro-service calling method based on block chain | |
| CN104333567A (en) | Web caching with security as a service | |
| CN105409284A (en) | Data bandwidth management system and method | |
| CN106933548A (en) | Global information is obtained, processes and updated, methods, devices and systems | |
| CN107818013A (en) | A kind of application scheduling method thereof and device | |
| CN106528578A (en) | An information display method and device | |
| CN109636307B (en) | River chang APP system | |
| CN110287266A (en) | A kind of distributed system and data processing method | |
| CN102508773A (en) | Method and device for monitoring WEB service system simulation based on Internet explorer (IE) kernel | |
| CN1640087B (en) | Policy-enabled contract-based management of network operational support systems | |
| CN105956422A (en) | Method and system for controlling usage of software programs on mobile computing devices | |
| CN103532932B (en) | A kind of ability open system, method for processing business and equipment | |
| CN110413918A (en) | Data transmission method for uplink, device, equipment and storage medium | |
| CN110287395A (en) | A kind of crawler method, apparatus, system, equipment and readable storage medium storing program for executing | |
| CN106941680A (en) | Base station monitoring method, terminal, vpn server and system | |
| Wang et al. | A comprehensive study on managing strategies in the fog environments | |
| CN105871659A (en) | Monitoring method and device of newly added server | |
| CN112037023B (en) | Bank outlet management method and related device | |
| CN108831531A (en) | A kind of long-range processing method of adaptive medical image and application system based on cloud computing | |
| CN111127706B (en) | Intelligent lock control method, intelligent lock, cloud server and computing equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190927 |