CN106657057A - Anti-crawler system and method - Google Patents
Anti-crawler system and method Download PDFInfo
- Publication number
- CN106657057A CN106657057A CN201611183559.8A CN201611183559A CN106657057A CN 106657057 A CN106657057 A CN 106657057A CN 201611183559 A CN201611183559 A CN 201611183559A CN 106657057 A CN106657057 A CN 106657057A
- Authority
- CN
- China
- Prior art keywords
- access
- behavior
- access behavior
- blacklist rule
- abnormal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/10—Network architectures or network communication protocols for network security for controlling access to devices or network resources
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/02—Network architectures or network communication protocols for network security for separating internal from external traffic, e.g. firewalls
- H04L63/0227—Filtering policies
- H04L63/0236—Filtering by address, protocol, port number or service, e.g. IP-address or URL
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/10—Network architectures or network communication protocols for network security for controlling access to devices or network resources
- H04L63/105—Multiple levels of security
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
- H04L63/1416—Event detection, e.g. attack signature detection
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
- H04L63/1425—Traffic logging, e.g. anomaly detection
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1433—Vulnerability analysis
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/20—Network architectures or network communication protocols for network security for managing network security; network security policies in general
- H04L63/205—Network architectures or network communication protocols for network security for managing network security; network security policies in general involving negotiation or determination of the one or more network security mechanisms to be used, e.g. by negotiation between the client and the server or between peers or by selection according to the capabilities of the entities involved
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses an anti-crawler system. The system comprises an analysis module, an acquisition module, a learning module, and a filter module, wherein the analysis module judges whether an access behavior is normal; the acquisition module acquires an access log which is judged as abnormal access behavior by the analysis module; the learning module has an updatable blacklist rule base and extracts a new blacklist rule according to the access log which is the abnormal access behavior so as to update the blacklist rule base; and the filter module prohibits the access behaviors included in the blacklist rule base. The invention further provides an anti-crawler method. On the one hand, the system can perform the anti-crawler according to the analysis of the analysis module; on the other hand, the learning module can extract new blacklist rule from the access log which is the abnormal access behavior so as to continuously update the blacklist rule base to assist the anti-crawler. Therefore, the anti-crawler system disclosed by the invention has the accuracy and the speed at the same time.
Description
Technical field
The present invention relates to anti-crawler technology field.It is more particularly related to be capable of the anti-crawler system of self study
And method.
Background technology
Climb data to refer to without the main permission of server, by the operation of software program models people, capture the data on the page.
The common anti-method for climbing data has two：1) by being verified to page setup identifying code, because these identifying codes are calculated
Machine is difficult to.But some computers are still able to identification, and some meetings are identified otherwise by employing people to know, therefore
Can not solve problem at all；2) by monitoring the abnormal behaviour of IP address, such as the client of certain IP address is not with browsing
Device is called, and calls excessive velocities in other words, and the amount of calling is excessive etc..Rule is set, these IP are sealed.But can so cause reaction speed
Degree is slow, after noting abnormalities, needs to be manually set rule, could realize counter climbing.And user is often acted on behalf of by IP
Mode, call various pseudo- IP to climb data.Therefore, it is necessary to design it is a kind of can oneself study it is counter climb rule system and side
Method.
The content of the invention
It is an object of the present invention to provide it is a kind of can extract from the access log of abnormal access behavior it is new black
List rule, to constantly update blacklist rule base, and then come the anti-system and method climbed.
In order to realize these purposes of the invention and further advantage, there is provided a kind of anti-crawler system, including：
Analysis module, it judges whether access behavior is normal；
Acquisition module, analysis module described in its acquisition Jing is judged as the access log of abnormal access behavior；
Study module, it has renewable blacklist rule base, and the study module is according to abnormal access behavior
Access log extracts new blacklist rule, to update the blacklist rule base；
Filtering module, it forbids the access behavior that the blacklist rule place is included.
Preferably, described anti-crawler system, the filtering module stores renewable IP blacklists, the filter module
The corresponding IP address of access behavior that blacklist rule place is included is added the IP blacklists by block, and forbids the IP ground
The access behavior of location.
Preferably, described anti-crawler system, if an access behavior is not included by the blacklist rule base, calls
The analysis module is analyzed to the access behavior, if the access behavior is abnormal, the filtering module forbids this
Access behavior；If an access behavior is included by the blacklist rule base, never call the analysis module and continue to the access
Behavior is analyzed.
Preferably, described anti-crawler system, the analysis module judges that the whether normal method of access behavior includes：
Access times of the access behavior in the first preset time period are obtained, and detects whether there is mouse behavior；
If the access times in the first Preset Time exceed predetermined threshold value, and are not detected by mouse behavior, then judging should
Access behavior is abnormal.
Preferably, described anti-crawler system, the study module is carried according to the access log of abnormal access behavior
Taking out the method for new blacklist rule includes：
Calculate unit interval access times and unit interval of the abnormal access behavior in the first preset time period to preset
Threshold value；Extract new blacklist rule be：The unit interval access times of access behavior are higher than unit interval predetermined threshold value.
Preferably, described anti-crawler system, the analysis module judges that the whether normal method of access behavior includes：
Access times and each access times corresponding time point of the access behavior in each second preset time period are obtained,
Then the second preset time period is divided into N number of time segment, the access frequency of each time segment is calculated respectively；
If the access frequency of N number of time segment is below first threshold, next second preset time period is divided into
N/2 time segment, then calculates respectively the access frequency of each time segment；
If the access frequency of N number of time segment is above Second Threshold, next second preset time period is divided into
2N time segment, then calculates respectively the access frequency of each time segment；
If the access frequency of N number of time segment is above first threshold and less than Second Threshold, pre- by next second
If the time period is divided into N number of time segment, the access frequency of each time segment is then calculated respectively；
Wherein, if access frequency of the behavior of access in arbitrary time segment is higher than frequency threshold, behavior is accessed for not
Normally；
Wherein, N >=10；
Wherein, first threshold is the 1/4 of frequency threshold, and Second Threshold is the 3/4 of frequency threshold.
A kind of anti-reptile method, including：
Judge whether access behavior is normal；
Obtain the access log that Jing is judged as abnormal access behavior；
New blacklist rule is extracted according to the access log of abnormal access behavior, to update blacklist rule base；
The access behavior for forbidding the blacklist rule place to include.
Preferably, described anti-reptile method, also includes：
The corresponding IP address of access behavior that blacklist rule place is included is added into the IP blacklists, and is forbidden
The access behavior of the IP address.
Preferably, described anti-reptile method, also includes：
If an access behavior is not included by the blacklist rule base, the access behavior is analyzed, if the visit
Ask that behavior is abnormal, then forbid the access behavior；If an access behavior is included by the blacklist rule base, do not continue right
The access behavior is analyzed.
Preferably, described anti-reptile method, judges that the whether normal method of access behavior includes：
Access times of the access behavior in the first preset time period are obtained, and detects whether there is mouse behavior；
If the access times in the first Preset Time exceed predetermined threshold value, and are not detected by mouse behavior, then judging should
Access behavior is abnormal.
The present invention at least includes following beneficial effect：
One aspect of the present invention can be climbed according to the analysis of analysis module is counter, and another aspect study module can never just
Frequentation asks that the access log of behavior extracts new blacklist rule, constantly updates blacklist rule base, and then can direct root
Access behavior is closed according to blacklist rule base, both combine have concurrently it is counter climb speed and it is counter climb the degree of accuracy, compared to only using
Analysis module, anti-speed of climbing is obviously improved.
The further advantage of the present invention, target and feature embody part by description below, and part will also be by this
The research of invention and practice and be understood by the person skilled in the art.
Specific embodiment
With reference to example, the present invention is described in further detail, to make those skilled in the art with reference to specification text
Word can be implemented according to this.
The present invention provides a kind of anti-crawler system, including：
Analysis module, it judges whether access behavior is normal；
Acquisition module, analysis module described in its acquisition Jing is judged as the access log of abnormal access behavior；
Study module, it has renewable blacklist rule base, and the study module is according to abnormal access behavior
Access log extracts new blacklist rule, to update the blacklist rule base；
Filtering module, it forbids the access behavior that the blacklist rule place is included.
In above-mentioned technical proposal, analysis module, acquisition module, study module and filtering module are independent server
Group, so having very high analytical performance and efficiency on hardware.Analysis method and existing skill of the analysis module to access behavior
Analysis method in art is identical, such as determine whether positive frequentation according to the IP of behavior, access time and access times are accessed
Behavior is asked, for example, accession page number of times is more than 3000 times in times of the IP more than three hours, and can't detect mouse
Mark, is taken as abnormal access.Acquisition module is used to obtain the access log of client.Study module is obtained according to analysis module
The abnormal access for going out, extracts new blacklist rule, and the new blacklist rule can be directly non-to this with analysis module
The normal judgment rule for accessing is identical, or improvement of judgment rule of the abnormal access.So, analysis module, obtain
Delivery block and study module repeated work, constantly update blacklist rule base.Filtering module is by next access behavior and black name
Blacklist rule in single rule base is compared, if access behavior meets one of blacklist rule, forbids the access
Behavior.
In another kind of example, described anti-crawler system, the filtering module stores renewable IP blacklists, described
The corresponding IP address of access behavior that blacklist rule place is included is added the IP blacklists by filtering module, and is forbidden
The access behavior of the IP address.Here, it is also possible to IP blacklists are constantly enriched according to blacklist rule base, makes system black according to IP
List directly closes the IP for abnormal access occurred, without comparing access behavior and blacklist rule base again.
In another kind of example, described anti-crawler system, if an access behavior is not included by the blacklist rule base,
The analysis module is then called to be analyzed the access behavior, if the access behavior is abnormal, the filtering module
Forbid the access behavior；If an access behavior is included by the blacklist rule base, never call the analysis module and continue right
The access behavior is analyzed.Here, optimization has been made to calling for analysis module, if an access behavior meets a kind of blacklist
Rule, then close the access behavior, and do not recall analysis module and be analyzed, if an access behavior does not meet any one
Blacklist rule, then call analysis module to be analyzed whether it should close.
In another kind of example, described anti-crawler system, the analysis module judges the whether normal side of access behavior
Method includes：
Access times of the access behavior in the first preset time period are obtained, and detects whether there is mouse behavior；
If the access times in the first Preset Time exceed predetermined threshold value, and are not detected by mouse behavior, then judging should
Access behavior is abnormal.
Above-mentioned technical proposal provides one kind and judges the whether normal feasible method of access behavior, that is, judge that access behavior exists
Whether the access times in certain period of time exceed predetermined threshold value, if exceeding, and are not detected by mouse behavior, then judge the access
Illegal act is normal.For example, the judgment rule of analysis module is accession page number of times in times of the IP more than three hours
More than 3000 times, and can't detect mouse, it is believed that be abnormal access.
In another kind of example, described anti-crawler system, access of the study module according to abnormal access behavior
Daily record extracts the method for new blacklist rule to be included：
Calculate unit interval access times and unit interval of the abnormal access behavior in the first preset time period to preset
Threshold value；Extract new blacklist rule be：The unit interval access times of access behavior are higher than unit interval predetermined threshold value.
The method that above-mentioned technical proposal provides the blacklist rule for extracting new from abnormal access behavior, i.e., with access
Number of times and predetermined threshold value are respectively divided by the first Preset Time, obtain unit interval access times and unit interval predetermined threshold value, newly
Blacklist rule be：The unit interval access times of access behavior are higher than unit interval predetermined threshold value, and are not detected by mouse
Behavior.The blacklist rule extracted is compared more flexible with the judgment rule of analysis module, easy.For example, certain analysis mould
It is final to divide also above 3000 times that block detects accession page number of times in mouse, but times of this IP more than three hours
Analysis module assert that this access falls within abnormal access, then analysis module these access logs can be issued study module as
Negative sample (before study module can also obtain some positive samples and negative sample for study).Study module can be according to these negative samples
This, extracts new blacklist rule, uses for next time.For example at second day, the access frequency of another IP has also reached often
Hour 1,000 times, although be also not reaching to three hours, analysis module does not also assert that this access belongs to abnormal access, and
The blacklist rule base of study module has enumerated this abnormal access.
In another kind of example, described anti-crawler system, the analysis module judges the whether normal side of access behavior
Method includes：
Access times and each access times corresponding time point of the access behavior in each second preset time period are obtained,
Then the second preset time period is divided into N number of time segment, the access frequency of each time segment is calculated respectively；
If the access frequency of N number of time segment is below first threshold, next second preset time period is divided into
N/2 time segment, then calculates respectively the access frequency of each time segment；
If the access frequency of N number of time segment is above Second Threshold, next second preset time period is divided into
2N time segment, then calculates respectively the access frequency of each time segment；
If the access frequency of N number of time segment is above first threshold and less than Second Threshold, pre- by next second
If the time period is divided into N number of time segment, the access frequency of each time segment is then calculated respectively；
Wherein, if access frequency of the behavior of access in arbitrary time segment is higher than frequency threshold, behavior is accessed for not
Normally；
Wherein, N >=10；
Wherein, first threshold is the 1/4 of frequency threshold, and Second Threshold is the 3/4 of frequency threshold.
Above-mentioned technical proposal judges the whether normal method of access behavior there is provided a kind of analysis module, will second preset
Time period is divided into some time segments, is calculated divided by the time span of time segment with the access times of each time segment respectively
To the access frequency of each time segment, then the access frequency of each time segment and frequency threshold are compared, if one of visit
Ask that frequency higher than frequency threshold, then judges that the access illegal act is normal；So can more accurately recognize reptile, it is to avoid reptile profit
Access frequency is reduced when improving access frequency when being moved with system leakage.In order to reduce amount of calculation, the present invention is also between each timesharing
When the access frequency of section is less than first threshold, access frequency technology number of times is reduced, when access frequency is higher than Second Threshold, improved
Access frequency calculation times, prevent from failing to report reptile.Present invention also offers the preferred value of first threshold and Second Threshold and subtracting
The preferred value of access frequency number of times is calculated less.
The present invention also provides a kind of anti-reptile method, including：
Judge whether access behavior is normal；
Obtain the access log that Jing is judged as abnormal access behavior；
New blacklist rule is extracted according to the access log of abnormal access behavior, to update blacklist rule base；
The access behavior for forbidding the blacklist rule place to include.
It is identical with analysis method of the prior art to accessing the analysis method of behavior first in above-mentioned technical proposal,
Determine whether normally to access behavior such as according to the IP of behavior, access time and access times are accessed, for example, an IP exists
Accession page number of times is more than 3000 times in time more than three hours, and can't detect mouse, is taken as abnormal access.
Then the access log of client is obtained.Subsequently new blacklist rule, the new blacklist are extracted according to abnormal access
Rule directly judgment rule identical with the judgment rule of the abnormal access, or the abnormal access can change
Enter.In this manner it is possible to constantly update blacklist rule base, and by the blacklist in next access behavior and blacklist rule base
Rule is compared, if access behavior meets one of blacklist rule, forbids the access behavior.
In another kind of example, described anti-reptile method also includes：
The corresponding IP address of access behavior that blacklist rule place is included is added into the IP blacklists, and is forbidden
The access behavior of the IP address.Here, it is also possible to IP blacklists are constantly enriched according to blacklist rule base, makes system black according to IP
List directly closes the IP for abnormal access occurred, without comparing access behavior and blacklist rule base again.
In another kind of example, described anti-reptile method also includes：
If an access behavior is not included by the blacklist rule base, the access behavior is analyzed, if the visit
Ask that behavior is abnormal, then forbid the access behavior；If an access behavior is included by the blacklist rule base, do not continue right
The access behavior is analyzed.Here, optimization has been made to calling for analysis, if an access behavior meets a kind of blacklist rule
Then, then the access behavior is closed, and is no longer analyzed, if an access behavior does not meet any blacklist rule,
Proceed to analyze whether it should close.
In another kind of example, described anti-reptile method judges that the whether normal method of access behavior includes：
Access times of the access behavior in the first preset time period are obtained, and detects whether there is mouse behavior；
If the access times in the first Preset Time exceed predetermined threshold value, and are not detected by mouse behavior, then judging should
Access behavior is abnormal.Above-mentioned technical proposal provides one kind and judges the whether normal feasible method of access behavior, that is, judge
Whether access times of the access behavior in certain period of time exceed predetermined threshold value, if exceeding, and are not detected by mouse behavior, then
Judge that the access illegal act is normal.For example, judgment rule is accession page number of times in times of the IP more than three hours
More than 3000 times, and mouse is can't detect, meet the rule and be taken as abnormal access.
Although embodiment of the present invention is disclosed as above, it is not restricted to listed in specification and embodiment
With, it can be applied to completely various suitable the field of the invention, for those skilled in the art, can be easily
Other modification is realized, therefore under the universal limited without departing substantially from claim and equivalency range, the present invention is not limited
In specific details and shown here as the example with description.
Claims (10)
1. a kind of anti-crawler system, it is characterised in that include：
Analysis module, it judges whether access behavior is normal；
Acquisition module, analysis module described in its acquisition Jing is judged as the access log of abnormal access behavior；
Study module, it has renewable blacklist rule base, access of the study module according to abnormal access behavior
Daily record extracts new blacklist rule, to update the blacklist rule base；
Filtering module, it forbids the access behavior that the blacklist rule place is included.
2. anti-crawler system as claimed in claim 1, it is characterised in that the filtering module stores the black names of renewable IP
Single, the corresponding IP address of access behavior that blacklist rule place is included is added the black names of the IP by the filtering module
It is single, and forbid the access behavior of the IP address.
3. anti-crawler system as claimed in claim 2, it is characterised in that if accesses behavior not by the blacklist rule base
Include, then call the analysis module to be analyzed the access behavior, if the access behavior is abnormal, the filtration
Module forbids the access behavior；If an access behavior is included by the blacklist rule base, never call the analysis module after
It is continuous that the access behavior is analyzed.
4. anti-crawler system as claimed in claim 1, it is characterised in that the analysis module judges whether access behavior is normal
Method include：
Access times of the access behavior in the first preset time period are obtained, and detects whether there is mouse behavior；
If the access times in the first Preset Time exceed predetermined threshold value, and are not detected by mouse behavior, then the access is judged
Behavior is abnormal.
5. anti-crawler system as claimed in claim 4, it is characterised in that the study module is according to abnormal access behavior
Access log extracts the method for new blacklist rule to be included：
Calculate unit interval access times and unit interval predetermined threshold value of the abnormal access behavior in the first preset time period；
Extract new blacklist rule be：The unit interval access times of access behavior are higher than unit interval predetermined threshold value.
6. anti-crawler system as claimed in claim 1, it is characterised in that the analysis module judges whether access behavior is normal
Method include：
Access times and each access times corresponding time point of the access behavior in each second preset time period are obtained, then
Second preset time period is divided into N number of time segment, the access frequency of each time segment is calculated respectively；
If the access frequency of N number of time segment is below first threshold, next second preset time period is divided into N/2
Time segment, then calculates respectively the access frequency of each time segment；
If the access frequency of N number of time segment is above Second Threshold, next second preset time period is divided into 2N
Time segment, then calculates respectively the access frequency of each time segment；
If the access frequency of N number of time segment is above first threshold and less than Second Threshold, when next second is preset
Between section be divided into N number of time segment, the access frequency of each time segment is then calculated respectively；
Wherein, if access frequency of the behavior of access in arbitrary time segment is higher than frequency threshold, the behavior that accesses is abnormal；
Wherein, N >=10；
Wherein, first threshold is the 1/4 of frequency threshold, and Second Threshold is the 3/4 of frequency threshold.
7. a kind of anti-reptile method, it is characterised in that include：
Judge whether access behavior is normal；
Obtain the access log that Jing is judged as abnormal access behavior；
New blacklist rule is extracted according to the access log of abnormal access behavior, to update blacklist rule base；
The access behavior for forbidding the blacklist rule place to include.
8. anti-reptile method as claimed in claim 7, it is characterised in that also include：
The corresponding IP address of access behavior that blacklist rule place is included is added into the IP blacklists, and forbids the IP
The access behavior of address.
9. anti-reptile method as claimed in claim 8, it is characterised in that also include：
If an access behavior is not included by the blacklist rule base, the access behavior is analyzed, if the access row
For abnormal, then forbid the access behavior；If an access behavior is included by the blacklist rule base, do not continue to the visit
The behavior of asking is analyzed.
10. anti-reptile method as claimed in claim 1, it is characterised in that judge that the whether normal method of access behavior includes：
Access times of the access behavior in the first preset time period are obtained, and detects whether there is mouse behavior；
If the access times in the first Preset Time exceed predetermined threshold value, and are not detected by mouse behavior, then the access is judged
Behavior is abnormal.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611183559.8A CN106657057B (en) | 2016-12-20 | 2016-12-20 | Anti-crawler system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611183559.8A CN106657057B (en) | 2016-12-20 | 2016-12-20 | Anti-crawler system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106657057A true CN106657057A (en) | 2017-05-10 |
| CN106657057B CN106657057B (en) | 2020-09-29 |
Family
ID=58833462
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611183559.8A Active CN106657057B (en) | 2016-12-20 | 2016-12-20 | Anti-crawler system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106657057B (en) |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107196968A (en) * | 2017-07-12 | 2017-09-22 | 深圳市活力天汇科技股份有限公司 | A kind of reptile recognition methods |
| CN107547548A (en) * | 2017-09-05 | 2018-01-05 | 北京京东尚科信息技术有限公司 | Data processing method and system |
| CN108133140A (en) * | 2017-12-08 | 2018-06-08 | 成都数聚城堡科技有限公司 | A kind of mode of the anti-reptile of dynamic |
| CN109246141A (en) * | 2018-10-26 | 2019-01-18 | 电子科技大学 | A kind of anti-excessive crawler method based on SDN |
| CN109246064A (en) * | 2017-07-11 | 2019-01-18 | 阿里巴巴集团控股有限公司 | Safe access control, the generation method of networkaccess rules, device and equipment |
| CN109241733A (en) * | 2018-08-07 | 2019-01-18 | 北京神州绿盟信息安全科技股份有限公司 | Crawler Activity recognition method and device based on web access log |
| CN109818949A (en) * | 2019-01-17 | 2019-05-28 | 济南浪潮高新科技投资发展有限公司 | A kind of anti-crawler method neural network based |
| CN110020512A (en) * | 2019-04-12 | 2019-07-16 | 重庆天蓬网络有限公司 | A kind of method, apparatus, equipment and the storage medium of anti-crawler |
| CN110781366A (en) * | 2019-09-09 | 2020-02-11 | 深圳壹账通智能科技有限公司 | Webpage data processing method and device, computer equipment and storage medium |
| CN111355728A (en) * | 2020-02-27 | 2020-06-30 | 紫光云技术有限公司 | Malicious crawler protection method |
| CN111625700A (en) * | 2020-05-25 | 2020-09-04 | 北京世纪家天下科技发展有限公司 | Anti-grabbing method, device, equipment and computer storage medium |
| CN112003833A (en) * | 2020-07-30 | 2020-11-27 | 瑞数信息技术（上海）有限公司 | Abnormal behavior detection method and device |
| CN112688919A (en) * | 2020-12-11 | 2021-04-20 | 杭州安恒信息技术股份有限公司 | APP interface-based crawler-resisting method, device and medium |
| CN113536301A (en) * | 2021-07-19 | 2021-10-22 | 北京计算机技术及应用研究所 | Behavior characteristic analysis-based anti-crawling method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103107948A (en) * | 2011-11-15 | 2013-05-15 | 阿里巴巴集团控股有限公司 | Flow control method and flow control device |
| CN103297435A (en) * | 2013-06-06 | 2013-09-11 | 中国科学院信息工程研究所 | Abnormal access behavior detection method and system on basis of WEB logs |
| CN103475637A (en) * | 2013-04-24 | 2013-12-25 | 携程计算机技术（上海）有限公司 | Network access control method and system based on IP access behaviors |
| US20140373139A1 (en) * | 2013-06-13 | 2014-12-18 | Alibaba Group Holding Limited | Method and system of distinguishing between human and machine |
| CN104902008A (en) * | 2015-04-26 | 2015-09-09 | 成都创行信息科技有限公司 | Crawler data processing method |
-
2016
- 2016-12-20 CN CN201611183559.8A patent/CN106657057B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103107948A (en) * | 2011-11-15 | 2013-05-15 | 阿里巴巴集团控股有限公司 | Flow control method and flow control device |
| CN103475637A (en) * | 2013-04-24 | 2013-12-25 | 携程计算机技术（上海）有限公司 | Network access control method and system based on IP access behaviors |
| CN103297435A (en) * | 2013-06-06 | 2013-09-11 | 中国科学院信息工程研究所 | Abnormal access behavior detection method and system on basis of WEB logs |
| US20140373139A1 (en) * | 2013-06-13 | 2014-12-18 | Alibaba Group Holding Limited | Method and system of distinguishing between human and machine |
| CN104902008A (en) * | 2015-04-26 | 2015-09-09 | 成都创行信息科技有限公司 | Crawler data processing method |
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109246064A (en) * | 2017-07-11 | 2019-01-18 | 阿里巴巴集团控股有限公司 | Safe access control, the generation method of networkaccess rules, device and equipment |
| CN107196968A (en) * | 2017-07-12 | 2017-09-22 | 深圳市活力天汇科技股份有限公司 | A kind of reptile recognition methods |
| CN107547548A (en) * | 2017-09-05 | 2018-01-05 | 北京京东尚科信息技术有限公司 | Data processing method and system |
| CN107547548B (en) * | 2017-09-05 | 2020-06-30 | 北京京东尚科信息技术有限公司 | Data processing method and system |
| CN108133140A (en) * | 2017-12-08 | 2018-06-08 | 成都数聚城堡科技有限公司 | A kind of mode of the anti-reptile of dynamic |
| CN109241733A (en) * | 2018-08-07 | 2019-01-18 | 北京神州绿盟信息安全科技股份有限公司 | Crawler Activity recognition method and device based on web access log |
| CN109246141B (en) * | 2018-10-26 | 2021-03-12 | 电子科技大学 | SDN-based excessive crawler prevention method |
| CN109246141A (en) * | 2018-10-26 | 2019-01-18 | 电子科技大学 | A kind of anti-excessive crawler method based on SDN |
| CN109818949A (en) * | 2019-01-17 | 2019-05-28 | 济南浪潮高新科技投资发展有限公司 | A kind of anti-crawler method neural network based |
| CN110020512A (en) * | 2019-04-12 | 2019-07-16 | 重庆天蓬网络有限公司 | A kind of method, apparatus, equipment and the storage medium of anti-crawler |
| CN110781366A (en) * | 2019-09-09 | 2020-02-11 | 深圳壹账通智能科技有限公司 | Webpage data processing method and device, computer equipment and storage medium |
| CN111355728A (en) * | 2020-02-27 | 2020-06-30 | 紫光云技术有限公司 | Malicious crawler protection method |
| CN111625700A (en) * | 2020-05-25 | 2020-09-04 | 北京世纪家天下科技发展有限公司 | Anti-grabbing method, device, equipment and computer storage medium |
| CN111625700B (en) * | 2020-05-25 | 2023-04-07 | 北京世纪家天下科技发展有限公司 | Anti-grabbing method, device, equipment and computer storage medium |
| CN112003833A (en) * | 2020-07-30 | 2020-11-27 | 瑞数信息技术（上海）有限公司 | Abnormal behavior detection method and device |
| CN112688919A (en) * | 2020-12-11 | 2021-04-20 | 杭州安恒信息技术股份有限公司 | APP interface-based crawler-resisting method, device and medium |
| CN113536301A (en) * | 2021-07-19 | 2021-10-22 | 北京计算机技术及应用研究所 | Behavior characteristic analysis-based anti-crawling method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106657057B (en) | 2020-09-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106657057A (en) | Anti-crawler system and method | |
| CN103297435B (en) | A kind of abnormal access behavioral value method and system based on WEB daily record | |
| CN106936781B (en) | A kind of determination method and device of user's operation behavior | |
| CN105809035B (en) | The malware detection method and system of real-time behavior is applied based on Android | |
| CN106295349A (en) | Risk Identification Method, identification device and the anti-Ore-controlling Role that account is stolen | |
| CN104391979A (en) | Malicious web crawler recognition method and device | |
| CN111738549A (en) | Food safety risk assessment method, device, equipment and storage medium | |
| CN106209862A (en) | A kind of steal-number defence implementation method and device | |
| CN107566390B (en) | Industrial control system network security analysis system and method based on threat information | |
| CN107016298B (en) | Webpage tampering monitoring method and device | |
| CN110139075A (en) | Video data handling procedure, device, computer equipment and storage medium | |
| CN105721406A (en) | Method and device for obtaining IP black list | |
| CN108111463A (en) | The self study of various dimensions baseline and abnormal behaviour analysis based on average value and standard deviation | |
| CN106327619A (en) | In/out management method and system thereof | |
| CN110135162A (en) | The recognition methods of the back door WEBSHELL, device, equipment and storage medium | |
| CN108366274B (en) | Method and device for detecting brushing playing amount | |
| CN105825130B (en) | A kind of information security method for early warning and device | |
| CN107172033B (en) | WAF misjudgment identification method and device | |
| CN112380126B (en) | Web system health prediction device and method | |
| CN104426836A (en) | Invasion detection method and device | |
| CN109190408B (en) | Data information security processing method and system | |
| CN111222162A (en) | Industry cloud resource access control method and device | |
| CN111049685A (en) | Network security sensing system, network security sensing method and device of power system | |
| CN105487936A (en) | Information system security evaluation method for classified protection under cloud environment | |
| CN115659324A (en) | Multi-device security management method and system for data security |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |