US20170243003A1 - Identifying bots - Google Patents
Identifying bots Download PDFInfo
- Publication number
- US20170243003A1 US20170243003A1 US15/590,689 US201715590689A US2017243003A1 US 20170243003 A1 US20170243003 A1 US 20170243003A1 US 201715590689 A US201715590689 A US 201715590689A US 2017243003 A1 US2017243003 A1 US 2017243003A1
- Authority
- US
- United States
- Prior art keywords
- prid
- machine
- client
- web
- script
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1441—Countermeasures against malicious traffic
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
- G06F21/55—Detecting local intrusion or implementing counter-measures
- G06F21/552—Detecting local intrusion or implementing counter-measures involving long-term monitoring or reporting
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/08—Network architectures or network communication protocols for network security for authentication of entities
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/12—Applying verification of the received information
- H04L63/126—Applying verification of the received information the source of the received data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
- H04L63/1416—Event detection, e.g. attack signature detection
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/16—Implementing security features at a particular protocol layer
- H04L63/168—Implementing security features at a particular protocol layer above the transport layer
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
- H04L67/025—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP] for remote control or remote monitoring of applications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W12/00—Security arrangements; Authentication; Protecting privacy or anonymity
- H04W12/06—Authentication
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L2463/00—Additional details relating to network architectures or network communication protocols for network security covered by H04L63/00
- H04L2463/144—Detection or countermeasures against botnets
Definitions
- the present invention relates to robotic internet surfing done by so-called bots and, in particular, to detecting suspicious behavior of bots and barring or restricting such behavior.
- Bots also known as web robots, spiders or web crawlers, are software applications that run automated tasks over the Internet. Typically, bots perform tasks that are both simple and structurally repetitive, at a much higher rate than would be possible for a human being, The largest usage of bots is web crawling, in which an automated script fetches, analyses and files information from web servers. Bots are used for many purposes; mainly for browsing, mapping and indexing data; monitoring the behavior of sites; advertising purposes; and, for commercial or academic research.
- bots may also be implemented where a response speed faster than that of human's is required (for example, gaming bots and auction-site robots) or, less commonly, in situations where the emulation of human activity is required (for example, chat bots).
- malicious bots such as spam bots, that harvest email addresses from contact forms or guestbook pages; downloader programs that suck bandwidth by downloading entire web sites; web site scrapers that grab the content of web sites and re-use it without permission on automatically generated doorway pages; and custom crawlers, tailored for specific websites to steal information (typically regarding index sites, classifieds and large database sites) or spam (typically regarding forums, web mail and social networks) and the like.
- bots can be divided into three main types:
- the first type is protocol based bots. These bots continuously generate a request using a certain protocol (such as, for example, HTTP or FTP) and receive a response, which is typically sent to a parser for analysis. These bots are simple and usually operate fast. They do not render the content they receive and hence have no browser capabilities.
- the second type is application bots which are based on protocol based bots but have more sophisticated parsing tools that render and interpret portions of the response (typically by having JavaScript capabilities).
- the third type is browser bots which are browsers (such as, for example, Internet Explorer, Firefox etc.) or browser platforms (such as, for example, Webkit) being controlled by an automation script. Browser bots are mechanically operated rather than being controlled by a human user.
- CAPTCHA Completely Automated Public Turing test to tell Computers and Humans Apart
- a CAPTCHA Completely Automated Public Turing test to tell Computers and Humans Apart
- the process usually involves one computer (a server) asking a user to complete a simple test which the computer is able to generate and grade. Because other computers are unable to solve the CAPTCHA, any user entering a correct solution is presumed to be human.
- CAPTCHAs are vulnerable to hackers, both by sophisticated custom made OCR systems which recognize the distorted text, or by simple relay hack (A bot displays the CAPTCHA to a human user who fills it in, in order to let the bot carry on its crawling activity).
- CAPTCHAs are typically presented to users only in the event of form filling, in order to avoid interruption to web application flow; thus, any activity done before or after filling the form can be easily driven by a bot.
- Another attempt to identify bots is honey pots or spider traps, which are normally web pages accessible only from transparent links (e.g. white text on white background). Such honey pots assume those who browse these hidden pages are bots. Honey pots are only useful for identifying generic bots such as email harvesters.
- bot identification methods can identify a suspicious activity by the user IP level or by the Session ID level. If a session is blocked, the bot can easily restart another session (typically by deleting a cookie file), while if an IP is blocked, legitimate users who may try to access the site from the same IP are blocked as well. Further more, these methods tend to yield too many false-positives (false identification of bots), or, if applied too carefully, too many false-negatives.
- a client having browser capabilities means any client having script interpreter or runtime environment and also having capabilities of accessing page and browser objects and rendering them into a document and a document object model (DOM).
- DOM document object model
- a mode for monitoring requests while continuing to serve the requested content by the web server This mode is termed hereinafter the “passive mode”.
- the requests are monitored for distinguishing between requests generated by clients having browser capabilities and any other client requests.
- “Monitoring” means capturing and analyzing traffic in a non-intrusive manner, as opposed to blocking which takes place on the “active mode”.
- the “passive mode” optionally enables decision making rules before taking any active step.
- Such decision making rules can be, for example and without wishing to be limited, allowing the handling of specific requests from a client not having browser capabilities which are classified as being legitimate clients.
- Such legitimate users can be, for example and without wishing to be limited, a search engine indexing spider.
- Such decision making rules can, optionally and without wishing to be limited, classify the user according to level of suspiciousness for assigning one or more restrictions according to the level of suspiciousness and by the identity of the originator of the requests.
- a mode restricting requests that are detected as non-browser requests from receiving any web site page contents This mode is termed hereinafter the “active mode”.
- the active mode filters out any HTTP page request not generated by a browser, by not serving the desired contents to that request.
- non-browser clients which are considered as protocol bots or application bots, are detected by generating a unique page request id (hereinafter “PRID”) per each client HTTP page request, embedding a client-side script within the response that is capable of regenerating and returning the same PRID only on a client having browser capabilities.
- Page request id means a unique identification of the page request, which is optionally composed of a random and/or unique combination of numbers and/or characters/letters, and is dynamically generated per each page request received from a client.
- PRID generation is optionally performed by the web server to which the request was sent, via, for example and without wishing to be limited, a server-side code (such as PHP, .Net framework languages, Python, and the like) or a server filter (such as ISAPI).
- a PRID can also be generated, by a logical or physical third party server or by any logical or physical network machine positioned in the network of the web server to monitor incoming traffic, such as a firewall, a web application firewall and the like.
- the machine in which PRID is generated is termed hereinafter “the originating machine”.
- the generated PRID string is optionally stored on the originating machine, optionally in a session scope variable.
- the originating machine either generates or outputs a pre-generated client-side script (optionally, but not limited to JavaScript) which is embedded into the page returned to the client.
- the client-side script is intended, when interpreted and ran by a client having browser capabilities, to regenerate and return, on a separate request (optionally HTTP get/post, Ajax or on a cookie sent with the next request), the same PRID to the originating machine.
- the script sent to the client optionally comprises a sequence of functions or small “sub-scripts” which will generate the sequence of characters that comprise, altogether, the PRID.
- the client-side code which regenerates the PRID on a client that has browser capabilities is termed herein after “regeneration challenge”.
- the set of client-side functions or “sub scripts” that are used to compose PRIDs for the regeneration challenge can optionally be changed from time to time, either generated by dedicated software which resides on the originating machine or by a third party machine (also referred to hereinafter as “controller machine”) serving client-side code templates to the originating machine.
- controller machine serving client-side code templates to the originating machine.
- the client that receives the embedded regeneration challenge returns the PRID to the originating machine
- the machine compares between the PRID it generated for the same page request and the PRID returned by the client. In case a match is found, it is determined that the original request was made by a client having browser capabilities, since only such a client would have run the embedded code, regenerated the PRID correctly and returned it on a separate request.
- the generated PRIDs and the returned regenerated PRIDs can be sent to the controller machine which inspects them and constantly checks match ratios and decides if a certain client is a bot, based on configurable match ratio rules. Match ratios may need to be set and calculated, since it is possible that even on browser clients, certain requests will not return a regenerated PRID if, for example, the request was quickly followed by another request, before the client-side regeneration challenge was run.
- a controller machine produces banks of functions or “sub-scripts”, each capable, when run on a browser, of generating a unique character or a certain string which can serve as a part of a PRID.
- the controller machine once in a while, updates the functions or “sub-scripts” bank that it generates on the originating machine.
- a service machine may produce a bank of ten different client-side functions (or “sub-scripts”) that yield, when run on a browser, the ten digits 0-9.
- This bank of functions will be copied to the originating machine (typically, a web server, though a network appliance may serve as both the originating machine and controller machine), which may, for a certain page request, generate a random PRID such as “194726”.
- the originating machine will collect, from its functions bank, the six functions that generate the six digits 1, 9, 4, 7, 2, 6, and create a single script which comprises all of them in a sequence.
- This script generates the string “194726” when ran on a browser.
- This script is the regeneration challenge.
- the originating machine will embed the regeneration challenge into the response sent to the client.
- the originating machine stores the string “194726” locally and expects to get it on a separate request during the same user session.
- the PRIDs received during a session may be sent back to the controller machine, where match ratios can be calculated and sophisticated rules can be applied to decide if a certain session is generated by a bot, or if a certain IP contains sessions generated by a bot, and whether to take action against this bot.
- the regeneration challenge script is comprised both of small document portions (typically, and without wishing to be limited, HTML or XML) and a script, which runs certain queries on the document object model (DOM), which is supposed to be generated on a browser which renders the document.
- the query can inspect certain objects, attributes and values in the DOM, and return characters or strings, which will, altogether, serve to regenerate the PRID.
- clients having a script runtime environment, but not having document rendering capabilities will not he able to create the DOM and return the strings that will comprise the PRID.
- a set of small documents will be generated on the controller machine, each having document objects (such as, but without wishing to be limited, layers, tables, style sheets—each having IDs, attributes and values generated by the controller machine) with script functions that each yield a known character or string when run on a browser which rendered the document.
- document objects such as, but without wishing to be limited, layers, tables, style sheets—each having IDs, attributes and values generated by the controller machine
- script functions that each yield a known character or string when run on a browser which rendered the document.
- a document portion can be a DIV tag, containing text, styles, tables and other HTML elements with attributes and values, which can be read through the document or the DOM by a script.
- originating machine, or controller machine receives and stores, on a database, a memory collection or array or any other means of storage, a list of the generated PRIDs. Per each PRID originating machine or controller machine also stores an indication whether a regenerated PRID was returned on a separate request to the originating machine. If a certain user session or a certain IP address has more than a predetermined number and/or ratio of PRIDs not followed by client-side PRID submissions, this session or IP is optionally suspected one in which a client not having browser capabilities is engaged, and therefore optionally as having protocol or application bot. Human decision or state machine rules can optionally be further applied to decide what measures, if at all, should be taken against this session or IP.
- the “active mode” can optionally restrict a client not having browser capabilities from receiving contents from web application by presenting the PRID regeneration challenge to the client before serving the requested content, as opposed to the “passive mode”, in which the PRID regeneration challenge is served with the contents.
- the originating machine first searches any request received from that IP or session for an embedded PRID (for example, within query-string, within any HTTP header value or any other collection of values normally sent as part of a request).
- the originating machine or web server collects all request information such as, and without wishing to be limited, the requested URL/URI, the HTTP header values, and query-string values, which are required to duplicate the request.
- the originating machine generates a PRID, and then in response to the request, instead of running the server-side code which resides at the requested URL and releasing the requested content to the client, the originating machine releases a page with client-side code, that is intended, if interpreted and run correctly by a client having browser capabilities, to reproduce the same request sent before with the regenerated PRID embedded to it (either as an additional HTTP header or query string value or with any collection of values typically sent as part of a request).
- This response serves as a gateway, validating the presence of browser capabilities before rendering and serving the requested contents.
- the gateway response will include the PRID regeneration challenge and a form which includes all the HTTP get/post keys and values from the previous requests, and another field into which the PRID will be inserted.
- the form will auto submit itself (via client-side script) to the web server, hence the original request is duplicated with a regenerated PRID embedded to it (as a get or post value or as a cookie).
- the gateway response may include additional functions to check the validity of the request and the client requesting it.
- a script which checks if the page is positioned at top of the browsers documents hierarchy, or as a child (such as iFrame) and whether the top documents belongs to the same site domain.
- the script may be set not to submit the form if some conditions are found dissatisfying, such as if the document is not a top document, or blurred (not focused), and thus block unaware automation, such as Cross Site Request Forgery, and non-human operated browsers and browser objects.
- Recognized PRID means a PRID which was generated by originating machine for a HTTP request received recently from that same IP or session.
- a client may enter a request-loop if it does not have full browser and/or session capabilities, since if it generates a wrong PRID, it gets a new gateway with a new PRID and so forth.
- the next generated PRID may contain a flag (optionally, an extra character), which signals the originating server not to send another gateway response (regeneration challenge) when another consecutive request with an unrecognized PRID is made from the same client.
- the web server can optionally reply with a message asking the user to turn sessions/cookies on.
- a state machine rule engine may score the suspicious internet users and instantly restrict the internet behavior of such users according to the scores.
- Such state machine can be set to recognize harmless bots and keep their activity uninterrupted, while restricting suspicious users.
- Such a restriction can be for example and without wishing to be limited by raising a “gateway challenge”, a CAPTCHA or even blocking entirely bad IPs and sessions according to the parameters gathered and according to the state machine policy rules.
- the state machine logic optionally resides separately from the web server, on a controller machine.
- the present invention provides a controller machine, which, according to state machine rules or any rule-set, can trigger the web-server to apply certain challenges or sanctions against certain sessions or certain IP addresses. If the controller machine concludes that a certain IP or session is suspected, it sends to the web-server, or alternatively to a central storage unit which is accessible to all application web servers (typically: mem-cache), or to a network appliance, an instruction to apply the chosen sanction or challenge on the certain IP or session (such as, but not limited to, regeneration challenge, CAPTCHA, complete block, or Sandbox mode).
- the web-server, central storage unit or network appliance will store the instructions telling which sanctions or challenges are applied to which IPs and/or sessions.
- a new request from a listed IP and/or session will be made to a web server, it will first lookup its instructions collection and if an instruction is applicable, it will respond with the sanction or challenge that is listed for the IP or session.
- Implementation of the method and system of the present invention involves performing or completing certain selected tasks or stages manually, automatically, or a combination thereof.
- several selected stages could be implemented by hardware or by software on any operating system of any firmware or a combination thereof.
- selected stages of the invention could be implemented as a chip or a circuit.
- selected stages of the invention could be implemented as a plurality of software instructions being executed by a computer using any suitable operating system.
- selected stages of the method and system of the invention could be described as being performed by a data processor, such as a computing platform for executing a plurality of instructions.
- any device featuring a data processor and/or the ability to execute one or more instructions may be described as a computer, including but not limited to a PC (personal computer), a server, a minicomputer, a cellular telephone, a smart phone, a PDA (personal data assistant), a pager, TV decoder, game console, digital music player, ATM (machine for dispensing cash), POS credit card terminal (point of sale), electronic cash register. Any two or more of such devices in communication with each other, and/or any computer in communication with any other computer may optionally comprise a “computer network”.
- FIGS. 1 a -1 c are schematic drawing of scenarios illustrating the process of identifying a client
- FIG. 2 is an exemplary scenario illustrating the passive mode
- FIG. 3 is an exemplary scenario illustrating the active mode
- FIG. 4 is an exemplary structure of a PRID script
- FIG. 5 is an exemplary description of the system according to one embodiment of the present invention.
- the present invention is of a method for detecting bots and in particular for detecting application and protocol based bots and for restricting malicious bots. It does this by identifying whether a web page request comes from a web client with browser capabilities.
- the invention is also of machines that run the method.
- FIGS. 1 a -1 c are schematic drawings of scenarios illustrating the process of identifying clients.
- FIG. 1 a illustrates a scenario in which the client is a protocol bot. In such a scenario, the client is not capable of generating a PRID at all and thus the system identifies this client as a client not having browser capabilities.
- the client requests a page.
- the web server generates a PRID, after receiving the request.
- the server generates client side functions to construct PRID upon execution. These functions are embedded in the reply and returned to the client in stage 4 .
- the client analyses the reply and, since the client is a protocol bot, the client is not capable of executing the Java script that is embedded in the response, and thus is identified by the server as a client not having browser capabilities.
- FIG. 1 b illustrates a scenario in which the client is an application bot.
- the client is not capable of generating a correct PRID.
- the client is capable of executing the java scripts the client is not able to access the required objects; thus, the system identifies this client as a client not having browser capabilities.
- the client requests a page.
- the web servers generates a PRID, after receiving the request.
- the server generates client side functions to construct PRID upon execution. These functions are embedded in the reply and returned to the client in stage 4 .
- the client analyses the reply and executes Java script, since the client is an application bot, the client is not capable for accessing the required objects and thus generates deformed PRID.
- the client sends the generated PRID to the server.
- the server compares the original PRID to the PRID sent by a client. The server does not find a match between the two PRIDs and thus the server identifies the client as a client not having browser capabilities.
- FIG. 1 c illustrates a scenario in which the client is a browser.
- the client is capable of generating a correct PRID.
- the client is capable of executing the java scripts and to access the required objects; thus, the system identifies the client as a client having browser capabilities.
- stage 1 the client requests a page.
- stage 2 the web server generates a PRID, after receiving the request.
- stage 3 the server generates client side functions to construct PRID upon execution. These functions are embedded in the reply and returned to the client in stage 4 .
- the client analyses the reply and executes Java script.
- the client generates a correct PRID.
- the client sends the generated PRID to the server.
- the server compares the original PRID to the PRID sent by a client. The server finds a match between the two PRIDs and thus identifies the client as a client having browser capabilities.
- FIG. 2 is a high level flow diagram of the process of identifying HTTP requests from clients not having browser capabilities [passive mode].
- a mode for monitoring requests while keep serving the requested contents by the web server This mode is termed herein after as “passive mode”.
- the requests are monitored for distinguishing between requests generated by clients having browser capabilities and any other client requests.
- monitoring is meant capturing and analyzing traffic in a non intrusive manner.
- the “passive mode” optionally enables to apply decision making rules before taking any active step.
- decision making rules can be, for example and without wishing to be limited, allowing the handling of specific requests from a client not having browser capabilities which are classified as being legitimate clients.
- Such legitimate users can be, for example and without wishing to be limited, a search engine indexing spider.
- Such decision making rules can (optionally and without wishing to be limited) classify the user according to level of suspiciousness for assigning one or more restrictions according to the level of suspiciousness and by the identity of the originator of the requests.
- a web client issues a request for a page from a browser.
- the module in the web server which is responsible for detecting suspicious bots generates a page request id, which is a unique identification of the request.
- the page request id along with the requested content is sent from the web server to the controller machine as a page request event .
- an Iframe is generated and calls additional client-side files from the controller machine, which are being embedded in the returned page.
- dynamic data gathering functions can be embedded to collect data that will be sent further on the process at stage 9 .
- page request id reconstruction functions are generated.
- Page request id reconstruction functions are functions which, when invoked, generate the page request id.
- the code is embedded in the reply and is sent to the web client.
- stage 7 which is performed after receiving a reply from the web server, the reply is processed.
- stage 8 which is performed only if the client has browser capabilities the page request id is generated out of the PRID regeneration functions.
- stage 9 the page request id is sent back to the web server with an additional request for example and without wishing to limit to, Ajax, HTTP get or HTTP post.
- the PRID can be sent directly to the controller machine and not to the originating machine.
- stage 13 which is performed when the message sent in stage 9 arrives in the web server, an asynchronous message is sent to the controller machine, with the regenerated PRID, to report that PRID was regenerated and returned.
- stage 14 which is performed when the controller machine receives the message sent in stage 13 , the regenerated PRID is stored on the controller machine. If the regenerated PRID matches the originally generated PRID, a pair of same PRIDs are received and stored on the controller machine.
- stage 10 which is performed by the controller machine after receiving the first event with the page request id, the event is logged for performing a parity check process. The parity check is performed for all the requests comprising the same IP address as the current request.
- Such a parity check does not comprise the current request but only the requests prior to this request.
- the number of page request events that are not paired with a corresponding page request event for the same IP and thus indicate a request from a client not having browser capabilities are compared to a predefined threshold.
- a message for restricting the IP is sent to the web server.
- the message can be for blocking the IP.
- the message can be for restricting the IP according to predefined rules. In such a case the message can optionally be sent to another server which can optionally apply the rules and sent the restriction command to the web server.
- the IP is blocked, if required.
- FIG. 3 is an exemplary scenario describing the active mode.
- a page request is sent from a WEB client to a Web server.
- the web browser checks if there is a PRID variable in the request. If there is not, this is assumed to be a first request to the URL/URI.
- a PRID flag which can be, but is not limited to, a character within the PRID generated string, is set to false in stage 10 . The flag is used for preventing endless looping which can occur if the browser is set to work in a mode in which cookies are disabled.
- a PRID is generated and the session is marked as a veteran session in stage 13 , indicating that this session is not newly generated when a next request from the same client arrives.
- the PRID is saved as a key in a PRID session array. This array is used for storing request related information to be used when a response is performed. Such information can be, for example and without wishing to be limited, the POST or GET method variables that are sent by the request.
- request variables are stored in the key session array.
- PRID reconstruction random functions are generated. Such functions, when activated, generate the PRID.
- an HTML shell with the PRID reconstruction random functions and a JS redirect to itself is generated and a response is returned to the web client.
- Stages 18 - 21 are performed by the web client when the first response comprising functions for generating PRID is received by web client.
- HTTP reply is processed.
- PRID is generated out of the reconstruction random functions.
- the system checks whether that PRID is a key in the PRID session array. If so, then the request variables are restored from the PRID session array in stage 22 .
- the PRID key in the PRID session array is cleared in stage 23 and in stage 24 the requested HTTP page is sent to the client and is processed by the client in stage 21 . If the answer to the question posted in stage 5 is “no” (PRID is not a key in session array) then this is a specific case in which the PRID is a leftover from a previous request.
- the system extracts the PRID flag in stage 4 .
- the system checks if the flag variable is true. If it is not true, it sets it to true in stage 25 in order to mark the situation in which there was a PRID request variable but it was not found in the session array. Then in stages 10 - 17 it resends the HTTP page shell again to generate a response comprising PRID reconstruction random functions. If it comes back again with the PRID flag as true, then the system checks in stage 7 if this is a veteran session. If so, then stages 10 - 17 are performed to generate a response comprising PRID reconstruction random functions.
- FIG. 4 is an exemplary structure of a PRID script.
- the script requires an interpreter in order to run and never has the PRID explicitly embedded in its code, hence “scraping” of the PRID is not possible.
- Such attributes ensures that a correct PRID is only generated on clients having the script interpreter or runtime environment.
- the scripts additionally and optionally use page and browser objects in the code that generates the PRID.
- the requirement for using the objects ensures that only browsers which render the page and hold object references (typically DOM—Document Object Model and browser objects like navigator, window, location and the like) are able to regenerate the correct PRID.
- the PRID that is generated is KXqe.
- FIG. 5 is a schematic drawing of the system.
- a system 100 features, a plurality of user computers shown as user computer A 101 , B 102 and C 103 , which may optionally be of any suitable number but of which only three are shown for the purpose of illustration only and without any intention of being limiting in any way; a plurality web servers shown as web server A 105 and web server B 106 which may optionally be of any suitable number but of which only two are shown for the sake of illustration only and without any intention of being limiting in any way; cluster dedicated servers 109 , corporate servers 110 , cluster server tier 1 104 , tier 2 central DB 107 and a user computer 109 .
- User computers A 101 , B 102 and C 103 are connected to web server A 105 and web server B 106 , preferably via the internet for retrieving information from the web server. It should be noted that more than one cluster servers can be operated and only one is shown for the purpose of illustration only. User computers A 101 , B 102 and C 103 usually operate legitimate queries to servers A 105 and web server B 106 ; such queries can be, for example queries initiated by a legitimate browser, and the like. However, user computers A 101 , B 102 and C 103 might operate illegal queries that are initiated by malicious Bots.
- Corporate servers 110 are also connected to the internet, usually via a singular IP address and operate queries from a plurality of users using the singular IP address.
- System 100 secures the web servers from malicious bots by using a dedicated software module which is installed on each web servers and by using tier 1 servers shown as 104 and 109 and tier 2 servers shown as 107 .
- Cluster servers Tier 1 104 and cluster dedicated servers 109 communicate with the web servers in order to identify malicious bots. The communication is preferably done via the internet. The method for suspecting malicious bots is explained in great details in FIGS. 2A and FIG. 2B .
- Tier 2 servers 107 perform further operations for classifying the bot and for assigning restrictions, if required.
- the information regarding suspicious bots is preferably kept in the database of the tier 2 server 107 for the future assessment of severity level.
- the process of classifying the suspiciousness and assigning restrictions is based on parameters which are configurable by an operator via a user computer shown as user computer A 109 . Configuration is preferably done via dashboard web interface 108 .
- Tier 2 server 107 preferably communicates with the relevant web servers for updating the web servers regarding the suspicious bot and the assigned restrictions.
- Web servers shown as web server A 105 and web server B 106 perform retractions on queries from the suspicious bots as required by tier 2 server 107 .
- Tier 2 server 107 and web servers preferably communicate via the internet.
- Tier 1 servers and Tier 2 servers are optional and the functionally detection of malicious bots can be embedded in the web servers shown as web server A 105 and web server B 106 . It should also be noted that the classification of the suspicious bots is optional and the system can operate without such classification. The classification, if used can be embedded in any server such as tier 1 server, web server and the like.
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
- Computer And Data Communications (AREA)
Abstract
A method of identifying if a web client has browser capabilities. An originating machine receives a web page request from the web client. The originating machine generates a page request id (PRID) and a script which, when executed by a web client with a browser, regenerates a PRID, and embeds the script in a response. The originating machine sends the response to the web client tor the web client to process and, if the web client is capable, to execute the embedded script, thereby to regenerate a PRID, and to return the regenerated PRID to the originating machine. The originating machine compares the returned regenerated PRID with the generated PRID, a match indicating that the web client has browser capabilities.
Description
- The present invention relates to robotic internet surfing done by so-called bots and, in particular, to detecting suspicious behavior of bots and barring or restricting such behavior.
- Bots, also known as web robots, spiders or web crawlers, are software applications that run automated tasks over the Internet. Typically, bots perform tasks that are both simple and structurally repetitive, at a much higher rate than would be possible for a human being, The largest usage of bots is web crawling, in which an automated script fetches, analyses and files information from web servers. Bots are used for many purposes; mainly for browsing, mapping and indexing data; monitoring the behavior of sites; advertising purposes; and, for commercial or academic research. In addition to their uses outlined above, bots may also be implemented where a response speed faster than that of human's is required (for example, gaming bots and auction-site robots) or, less commonly, in situations where the emulation of human activity is required (for example, chat bots). Unfortunately, there are also malicious bots, such as spam bots, that harvest email addresses from contact forms or guestbook pages; downloader programs that suck bandwidth by downloading entire web sites; web site scrapers that grab the content of web sites and re-use it without permission on automatically generated doorway pages; and custom crawlers, tailored for specific websites to steal information (typically regarding index sites, classifieds and large database sites) or spam (typically regarding forums, web mail and social networks) and the like.
- From a technical aspect, bots can be divided into three main types: The first type is protocol based bots. These bots continuously generate a request using a certain protocol (such as, for example, HTTP or FTP) and receive a response, which is typically sent to a parser for analysis. These bots are simple and usually operate fast. They do not render the content they receive and hence have no browser capabilities. The second type is application bots which are based on protocol based bots but have more sophisticated parsing tools that render and interpret portions of the response (typically by having JavaScript capabilities). The third type is browser bots which are browsers (such as, for example, Internet Explorer, Firefox etc.) or browser platforms (such as, for example, Webkit) being controlled by an automation script. Browser bots are mechanically operated rather than being controlled by a human user.
- There have been many attempts to identify and filter out malicious bots, such as, for example, by analyzing log files and/or by analyzing the frequency of HTTP requests per IP or by using a CAPTCHA. A CAPTCHA (“Completely Automated Public Turing test to tell Computers and Humans Apart”) is a type of challenge-response test used in computing to ensure that the response is not generated by a computer. The process usually involves one computer (a server) asking a user to complete a simple test which the computer is able to generate and grade. Because other computers are unable to solve the CAPTCHA, any user entering a correct solution is presumed to be human. Thus, it is sometimes described as a reverse Turing test because it is administered by a machine and targeted to a human, in contrast to the standard Turing test that is typically administered by a human and targeted to a machine. A common type of CAPTCHA requires that the user types letters or digits from a distorted image that appears on the screen.
- CAPTCHAs are vulnerable to hackers, both by sophisticated custom made OCR systems which recognize the distorted text, or by simple relay hack (A bot displays the CAPTCHA to a human user who fills it in, in order to let the bot carry on its crawling activity). CAPTCHAs are typically presented to users only in the event of form filling, in order to avoid interruption to web application flow; thus, any activity done before or after filling the form can be easily driven by a bot. Another attempt to identify bots is honey pots or spider traps, which are normally web pages accessible only from transparent links (e.g. white text on white background). Such honey pots assume those who browse these hidden pages are bots. Honey pots are only useful for identifying generic bots such as email harvesters.
- Unfortunately, known bot identification methods can identify a suspicious activity by the user IP level or by the Session ID level. If a session is blocked, the bot can easily restart another session (typically by deleting a cookie file), while if an IP is blocked, legitimate users who may try to access the site from the same IP are blocked as well. Further more, these methods tend to yield too many false-positives (false identification of bots), or, if applied too carefully, too many false-negatives.
- There is thus a need in the art for more efficient and reliable method for identifying bots and blocking them with less interruption to genuine human users.
- This need is met by the method according to
claim 1. Other features of the invention are set out in the subsequent claims. - According to one embodiment of the present invention, there is provided a method which differentiates an HTTP request from a client having browser capabilities from any other machine's request or software driven HTTP requests. “A client having browser capabilities” means any client having script interpreter or runtime environment and also having capabilities of accessing page and browser objects and rendering them into a document and a document object model (DOM).
- According to another embodiment of the present invention, there is provided a mode for monitoring requests while continuing to serve the requested content by the web server. This mode is termed hereinafter the “passive mode”. According to this embodiment, the requests are monitored for distinguishing between requests generated by clients having browser capabilities and any other client requests. “Monitoring” means capturing and analyzing traffic in a non-intrusive manner, as opposed to blocking which takes place on the “active mode”. The “passive mode” optionally enables decision making rules before taking any active step. Such decision making rules can be, for example and without wishing to be limited, allowing the handling of specific requests from a client not having browser capabilities which are classified as being legitimate clients. Such legitimate users can be, for example and without wishing to be limited, a search engine indexing spider. Such decision making rules can, optionally and without wishing to be limited, classify the user according to level of suspiciousness for assigning one or more restrictions according to the level of suspiciousness and by the identity of the originator of the requests.
- According to another embodiment of the present invention, there is provided a mode restricting requests that are detected as non-browser requests from receiving any web site page contents. This mode is termed hereinafter the “active mode”. The active mode filters out any HTTP page request not generated by a browser, by not serving the desired contents to that request.
- According to some embodiments of the invention, non-browser clients, which are considered as protocol bots or application bots, are detected by generating a unique page request id (hereinafter “PRID”) per each client HTTP page request, embedding a client-side script within the response that is capable of regenerating and returning the same PRID only on a client having browser capabilities. “Page request id” means a unique identification of the page request, which is optionally composed of a random and/or unique combination of numbers and/or characters/letters, and is dynamically generated per each page request received from a client. PRID generation is optionally performed by the web server to which the request was sent, via, for example and without wishing to be limited, a server-side code (such as PHP, .Net framework languages, Python, and the like) or a server filter (such as ISAPI). Alternatively, a PRID can also be generated, by a logical or physical third party server or by any logical or physical network machine positioned in the network of the web server to monitor incoming traffic, such as a firewall, a web application firewall and the like. The machine in which PRID is generated is termed hereinafter “the originating machine”. The generated PRID string is optionally stored on the originating machine, optionally in a session scope variable. The originating machine either generates or outputs a pre-generated client-side script (optionally, but not limited to JavaScript) which is embedded into the page returned to the client. The client-side script is intended, when interpreted and ran by a client having browser capabilities, to regenerate and return, on a separate request (optionally HTTP get/post, Ajax or on a cookie sent with the next request), the same PRID to the originating machine. The script sent to the client optionally comprises a sequence of functions or small “sub-scripts” which will generate the sequence of characters that comprise, altogether, the PRID. The client-side code which regenerates the PRID on a client that has browser capabilities is termed herein after “regeneration challenge”. The set of client-side functions or “sub scripts” that are used to compose PRIDs for the regeneration challenge can optionally be changed from time to time, either generated by dedicated software which resides on the originating machine or by a third party machine (also referred to hereinafter as “controller machine”) serving client-side code templates to the originating machine. When the client that receives the embedded regeneration challenge returns the PRID to the originating machine, the machine compares between the PRID it generated for the same page request and the PRID returned by the client. In case a match is found, it is determined that the original request was made by a client having browser capabilities, since only such a client would have run the embedded code, regenerated the PRID correctly and returned it on a separate request. The generated PRIDs and the returned regenerated PRIDs can be sent to the controller machine which inspects them and constantly checks match ratios and decides if a certain client is a bot, based on configurable match ratio rules. Match ratios may need to be set and calculated, since it is possible that even on browser clients, certain requests will not return a regenerated PRID if, for example, the request was quickly followed by another request, before the client-side regeneration challenge was run.
- According to some embodiments, a controller machine produces banks of functions or “sub-scripts”, each capable, when run on a browser, of generating a unique character or a certain string which can serve as a part of a PRID. The controller machine, once in a while, updates the functions or “sub-scripts” bank that it generates on the originating machine. For example only, a service machine may produce a bank of ten different client-side functions (or “sub-scripts”) that yield, when run on a browser, the ten digits 0-9. This bank of functions will be copied to the originating machine (typically, a web server, though a network appliance may serve as both the originating machine and controller machine), which may, for a certain page request, generate a random PRID such as “194726”. For this specific PRID, the originating machine will collect, from its functions bank, the six functions that generate the six
digits - According to another embodiment of the present invention, the regeneration challenge script is comprised both of small document portions (typically, and without wishing to be limited, HTML or XML) and a script, which runs certain queries on the document object model (DOM), which is supposed to be generated on a browser which renders the document. The query can inspect certain objects, attributes and values in the DOM, and return characters or strings, which will, altogether, serve to regenerate the PRID. In this manner, clients having a script runtime environment, but not having document rendering capabilities, will not he able to create the DOM and return the strings that will comprise the PRID. Typically, a set of small documents will be generated on the controller machine, each having document objects (such as, but without wishing to be limited, layers, tables, style sheets—each having IDs, attributes and values generated by the controller machine) with script functions that each yield a known character or string when run on a browser which rendered the document. For example only, a document portion can be a DIV tag, containing text, styles, tables and other HTML elements with attributes and values, which can be read through the document or the DOM by a script. Furthermore, since different web browser versions may generate slightly different DOMs from the same document (for example, certain object attributes may have capital letters on one browser and non-capital on another browser), this approach can serve to identify and verify the web browser type and version according to minor differences in the regenerated and returned PRID.
- According to another embodiment of the present invention, originating machine, or controller machine, receives and stores, on a database, a memory collection or array or any other means of storage, a list of the generated PRIDs. Per each PRID originating machine or controller machine also stores an indication whether a regenerated PRID was returned on a separate request to the originating machine. If a certain user session or a certain IP address has more than a predetermined number and/or ratio of PRIDs not followed by client-side PRID submissions, this session or IP is optionally suspected one in which a client not having browser capabilities is engaged, and therefore optionally as having protocol or application bot. Human decision or state machine rules can optionally be further applied to decide what measures, if at all, should be taken against this session or IP.
- According to some embodiments, the “active mode” can optionally restrict a client not having browser capabilities from receiving contents from web application by presenting the PRID regeneration challenge to the client before serving the requested content, as opposed to the “passive mode”, in which the PRID regeneration challenge is served with the contents. When it is decided to filter-out clients without browser capabilities from a certain IP or session, or from specific sections of a website, the originating machine first searches any request received from that IP or session for an embedded PRID (for example, within query-string, within any HTTP header value or any other collection of values normally sent as part of a request). In case the request does not have a PRID, the originating machine or web server collects all request information such as, and without wishing to be limited, the requested URL/URI, the HTTP header values, and query-string values, which are required to duplicate the request. The originating machine generates a PRID, and then in response to the request, instead of running the server-side code which resides at the requested URL and releasing the requested content to the client, the originating machine releases a page with client-side code, that is intended, if interpreted and run correctly by a client having browser capabilities, to reproduce the same request sent before with the regenerated PRID embedded to it (either as an additional HTTP header or query string value or with any collection of values typically sent as part of a request). This response serves as a gateway, validating the presence of browser capabilities before rendering and serving the requested contents. Typically, but without wishing to be limited, the gateway response will include the PRID regeneration challenge and a form which includes all the HTTP get/post keys and values from the previous requests, and another field into which the PRID will be inserted. The form will auto submit itself (via client-side script) to the web server, hence the original request is duplicated with a regenerated PRID embedded to it (as a get or post value or as a cookie). The gateway response may include additional functions to check the validity of the request and the client requesting it. For instance, a script which checks if the page is positioned at top of the browsers documents hierarchy, or as a child (such as iFrame) and whether the top documents belongs to the same site domain. The script may be set not to submit the form if some conditions are found dissatisfying, such as if the document is not a top document, or blurred (not focused), and thus block unaware automation, such as Cross Site Request Forgery, and non-human operated browsers and browser objects.
- Any request on that specific IP or session, which arrives at the originating machine with a recognized PRID embedded in it, will he responded to with the requested content, while if there is no PRID or unrecognized one, the PRID regeneration challenge will be served instead. “Recognized PRID” means a PRID which was generated by originating machine for a HTTP request received recently from that same IP or session.
- In some scenarios a client may enter a request-loop if it does not have full browser and/or session capabilities, since if it generates a wrong PRID, it gets a new gateway with a new PRID and so forth. In order to avoid this situation, optionally, if a request is received with a PRID not recognized as one generated before for that IP or session, the next generated PRID may contain a flag (optionally, an extra character), which signals the originating server not to send another gateway response (regeneration challenge) when another consecutive request with an unrecognized PRID is made from the same client. To such a second response, the web server can optionally reply with a message asking the user to turn sessions/cookies on.
- According to some embodiments, a state machine rule engine may score the suspicious internet users and instantly restrict the internet behavior of such users according to the scores. Such state machine can be set to recognize harmless bots and keep their activity uninterrupted, while restricting suspicious users. Such a restriction can be for example and without wishing to be limited by raising a “gateway challenge”, a CAPTCHA or even blocking entirely bad IPs and sessions according to the parameters gathered and according to the state machine policy rules. The state machine logic optionally resides separately from the web server, on a controller machine.
- According to some embodiments, the present invention provides a controller machine, which, according to state machine rules or any rule-set, can trigger the web-server to apply certain challenges or sanctions against certain sessions or certain IP addresses. If the controller machine concludes that a certain IP or session is suspected, it sends to the web-server, or alternatively to a central storage unit which is accessible to all application web servers (typically: mem-cache), or to a network appliance, an instruction to apply the chosen sanction or challenge on the certain IP or session (such as, but not limited to, regeneration challenge, CAPTCHA, complete block, or Sandbox mode). The web-server, central storage unit or network appliance will store the instructions telling which sanctions or challenges are applied to which IPs and/or sessions. When a new request from a listed IP and/or session will be made to a web server, it will first lookup its instructions collection and if an instruction is applicable, it will respond with the sanction or challenge that is listed for the IP or session.
- Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The materials, methods, and examples provided herein are illustrative only and not intended to be limiting.
- Implementation of the method and system of the present invention involves performing or completing certain selected tasks or stages manually, automatically, or a combination thereof. Moreover, according to actual instrumentation and equipment of preferred embodiments of the method and system of the present invention, several selected stages could be implemented by hardware or by software on any operating system of any firmware or a combination thereof. For example, as hardware, selected stages of the invention could be implemented as a chip or a circuit. As software, selected stages of the invention could be implemented as a plurality of software instructions being executed by a computer using any suitable operating system. In any case, selected stages of the method and system of the invention could be described as being performed by a data processor, such as a computing platform for executing a plurality of instructions.
- Although the present invention is described with regard to a “computer” on a “computer network”, it should be noted that optionally any device featuring a data processor and/or the ability to execute one or more instructions may be described as a computer, including but not limited to a PC (personal computer), a server, a minicomputer, a cellular telephone, a smart phone, a PDA (personal data assistant), a pager, TV decoder, game console, digital music player, ATM (machine for dispensing cash), POS credit card terminal (point of sale), electronic cash register. Any two or more of such devices in communication with each other, and/or any computer in communication with any other computer may optionally comprise a “computer network”.
- The invention is herein described, by way of example only, with reference to the accompanying drawings. With specific reference now to the drawings in detail, it is stressed that the particulars shown are by way of example and for purposes of illustrative discussion of the preferred embodiments of the present invention only, and are presented in order to provide what is believed to be the most useful and readily understood description of the principles and conceptual aspects of the invention. In this regard, no attempt is made to show structural details of the invention in more detail than is necessary for a fundamental understanding of the invention, the description taken with the drawings making apparent to those skilled in the art how the several forms of the invention may be embodied in practice.
- In the drawings:
-
FIGS. 1a-1c are schematic drawing of scenarios illustrating the process of identifying a client; -
FIG. 2 is an exemplary scenario illustrating the passive mode; -
FIG. 3 . is an exemplary scenario illustrating the active mode; -
FIG. 4 is an exemplary structure of a PRID script; and -
FIG. 5 is an exemplary description of the system according to one embodiment of the present invention. - The present invention is of a method for detecting bots and in particular for detecting application and protocol based bots and for restricting malicious bots. It does this by identifying whether a web page request comes from a web client with browser capabilities. The invention is also of machines that run the method.
-
FIGS. 1a-1c are schematic drawings of scenarios illustrating the process of identifying clients.FIG. 1a illustrates a scenario in which the client is a protocol bot. In such a scenario, the client is not capable of generating a PRID at all and thus the system identifies this client as a client not having browser capabilities. Instage 1, the client requests a page. Instage 2 the web server generates a PRID, after receiving the request. Instage 3 the server generates client side functions to construct PRID upon execution. These functions are embedded in the reply and returned to the client instage 4. Instage 5, the client analyses the reply and, since the client is a protocol bot, the client is not capable of executing the Java script that is embedded in the response, and thus is identified by the server as a client not having browser capabilities. -
FIG. 1b illustrates a scenario in which the client is an application bot. In such a scenario, the client is not capable of generating a correct PRID. Though the client is capable of executing the java scripts the client is not able to access the required objects; thus, the system identifies this client as a client not having browser capabilities. Instage 1, the client requests a page. Instage 2 the web servers generates a PRID, after receiving the request. Instage 3 the server generates client side functions to construct PRID upon execution. These functions are embedded in the reply and returned to the client instage 4. Instage 5, the client analyses the reply and executes Java script, since the client is an application bot, the client is not capable for accessing the required objects and thus generates deformed PRID. Instage 7, the client sends the generated PRID to the server. Instage 8, the server compares the original PRID to the PRID sent by a client. The server does not find a match between the two PRIDs and thus the server identifies the client as a client not having browser capabilities. -
FIG. 1c illustrates a scenario in which the client is a browser. In such a scenario, the client is capable of generating a correct PRID. The client is capable of executing the java scripts and to access the required objects; thus, the system identifies the client as a client having browser capabilities. Instage 1, the client requests a page. Instage 2 the web server generates a PRID, after receiving the request. Instage 3 the server generates client side functions to construct PRID upon execution. These functions are embedded in the reply and returned to the client instage 4. Instage 5, the client analyses the reply and executes Java script. Instage 6 the client generates a correct PRID. Instage 7, the client sends the generated PRID to the server. Instage 8, the server compares the original PRID to the PRID sent by a client. The server finds a match between the two PRIDs and thus identifies the client as a client having browser capabilities. -
FIG. 2 is a high level flow diagram of the process of identifying HTTP requests from clients not having browser capabilities [passive mode]. - According to another embodiment of the present invention, there is provided a mode for monitoring requests while keep serving the requested contents by the web server. This mode is termed herein after as “passive mode”. According to this embodiment, the requests are monitored for distinguishing between requests generated by clients having browser capabilities and any other client requests. By monitoring is meant capturing and analyzing traffic in a non intrusive manner. The “passive mode” optionally enables to apply decision making rules before taking any active step. Such decision making rules can be, for example and without wishing to be limited, allowing the handling of specific requests from a client not having browser capabilities which are classified as being legitimate clients. Such legitimate users can be, for example and without wishing to be limited, a search engine indexing spider. Such decision making rules can (optionally and without wishing to be limited) classify the user according to level of suspiciousness for assigning one or more restrictions according to the level of suspiciousness and by the identity of the originator of the requests.
- Referring now to the drawing, in stage 1 a web client issues a request for a page from a browser. In
stage 2, the module in the web server which is responsible for detecting suspicious bots generates a page request id, which is a unique identification of the request. Instage 3, the page request id along with the requested content is sent from the web server to the controller machine as a page request event . In stage 4 [optional], an Iframe is generated and calls additional client-side files from the controller machine, which are being embedded in the returned page. In this controller client-side script file, dynamic data gathering functions can be embedded to collect data that will be sent further on the process atstage 9. Instage 5, page request id reconstruction functions are generated. Page request id reconstruction functions are functions which, when invoked, generate the page request id. Instage 6, the code is embedded in the reply and is sent to the web client. Instage 7, which is performed after receiving a reply from the web server, the reply is processed. Instage 8, which is performed only if the client has browser capabilities the page request id is generated out of the PRID regeneration functions. Instage 9, the page request id is sent back to the web server with an additional request for example and without wishing to limit to, Ajax, HTTP get or HTTP post. Alternatively, the PRID can be sent directly to the controller machine and not to the originating machine. Instage 13, which is performed when the message sent instage 9 arrives in the web server, an asynchronous message is sent to the controller machine, with the regenerated PRID, to report that PRID was regenerated and returned. Instage 14, which is performed when the controller machine receives the message sent instage 13, the regenerated PRID is stored on the controller machine. If the regenerated PRID matches the originally generated PRID, a pair of same PRIDs are received and stored on the controller machine. Instage 10, which is performed by the controller machine after receiving the first event with the page request id, the event is logged for performing a parity check process. The parity check is performed for all the requests comprising the same IP address as the current request. Such a parity check does not comprise the current request but only the requests prior to this request. Instage 12, the number of page request events that are not paired with a corresponding page request event for the same IP and thus indicate a request from a client not having browser capabilities are compared to a predefined threshold. Instage 16, which is performed when the disparity value exceeds the predefined threshold, a message for restricting the IP is sent to the web server. According to one embodiment the message can be for blocking the IP. According to another embodiment, the message can be for restricting the IP according to predefined rules. In such a case the message can optionally be sent to another server which can optionally apply the rules and sent the restriction command to the web server. Instage 15, the IP is blocked, if required. -
FIG. 3 is an exemplary scenario describing the active mode. Instage 1, a page request is sent from a WEB client to a Web server. Instage 2, the web browser checks if there is a PRID variable in the request. If there is not, this is assumed to be a first request to the URL/URI. A PRID flag which can be, but is not limited to, a character within the PRID generated string, is set to false instage 10. The flag is used for preventing endless looping which can occur if the browser is set to work in a mode in which cookies are disabled. Instage 12, a PRID is generated and the session is marked as a veteran session instage 13, indicating that this session is not newly generated when a next request from the same client arrives. Instage 14, the PRID is saved as a key in a PRID session array. This array is used for storing request related information to be used when a response is performed. Such information can be, for example and without wishing to be limited, the POST or GET method variables that are sent by the request. Instage 15, request variables are stored in the key session array. Instage 16, PRID reconstruction random functions are generated. Such functions, when activated, generate the PRID. Instage 17, an HTML shell with the PRID reconstruction random functions and a JS redirect to itself is generated and a response is returned to the web client. Stages 18-21 are performed by the web client when the first response comprising functions for generating PRID is received by web client. Instage 18, HTTP reply is processed. Instage 19, PRID is generated out of the reconstruction random functions. Instage 20, page redirect itself with a PRID as a request parameter. - If the answer to the question “is there a PRID variable in the request?” is “yes”, the system assumes that this is a second request and the PRID is extracted from the request and saved in
stage 3. Instage 5, the system checks whether that PRID is a key in the PRID session array. If so, then the request variables are restored from the PRID session array instage 22. The PRID key in the PRID session array is cleared instage 23 and instage 24 the requested HTTP page is sent to the client and is processed by the client instage 21. If the answer to the question posted instage 5 is “no” (PRID is not a key in session array) then this is a specific case in which the PRID is a leftover from a previous request. In this case the system extracts the PRID flag instage 4. Instage 6, the system checks if the flag variable is true. If it is not true, it sets it to true in stage 25 in order to mark the situation in which there was a PRID request variable but it was not found in the session array. Then in stages 10-17 it resends the HTTP page shell again to generate a response comprising PRID reconstruction random functions. If it comes back again with the PRID flag as true, then the system checks instage 7 if this is a veteran session. If so, then stages 10-17 are performed to generate a response comprising PRID reconstruction random functions. If the session is not marked as veteran instage 7, it means that something is wrong as this request has been processed but there was no trace for it in the session. This situation can happen if the request was processed by a browser that has its cookies mode turned off. In this case, a proper message is sent to the client. This flag is used for preventing infinite loops in the case that there are no cookies in the browser. 5 -
FIG. 4 is an exemplary structure of a PRID script. According to one embodiment, the script requires an interpreter in order to run and never has the PRID explicitly embedded in its code, hence “scraping” of the PRID is not possible. Such attributes ensures that a correct PRID is only generated on clients having the script interpreter or runtime environment. The scripts additionally and optionally use page and browser objects in the code that generates the PRID. The requirement for using the objects ensures that only browsers which render the page and hold object references (typically DOM—Document Object Model and browser objects like navigator, window, location and the like) are able to regenerate the correct PRID. In the exemplary script the PRID that is generated is KXqe. -
FIG. 5 is a schematic drawing of the system. As shown, a system 100 features, a plurality of user computers shown asuser computer A 101,B 102 andC 103, which may optionally be of any suitable number but of which only three are shown for the purpose of illustration only and without any intention of being limiting in any way; a plurality web servers shown asweb server A 105 andweb server B 106 which may optionally be of any suitable number but of which only two are shown for the sake of illustration only and without any intention of being limiting in any way; cluster dedicatedservers 109,corporate servers 110,cluster server tier 1 104,tier 2central DB 107 and auser computer 109. User computers A 101,B 102 andC 103 are connected toweb server A 105 andweb server B 106, preferably via the internet for retrieving information from the web server. It should be noted that more than one cluster servers can be operated and only one is shown for the purpose of illustration only. User computers A 101,B 102 andC 103 usually operate legitimate queries to servers A 105 andweb server B 106; such queries can be, for example queries initiated by a legitimate browser, and the like. However, user computers A 101,B 102 andC 103 might operate illegal queries that are initiated by malicious Bots.Corporate servers 110 are also connected to the internet, usually via a singular IP address and operate queries from a plurality of users using the singular IP address. In the exemplary diagram corporate servers are connected to clusterdedicated tier 1servers 109 for serving thecorporate servers 110 only. System 100 secures the web servers from malicious bots by using a dedicated software module which is installed on each web servers and by usingtier 1 servers shown as 104 and 109 andtier 2 servers shown as 107.Cluster servers Tier 1 104 and cluster dedicatedservers 109 communicate with the web servers in order to identify malicious bots. The communication is preferably done via the internet. The method for suspecting malicious bots is explained in great details inFIGS. 2A andFIG. 2B . When a suspected bot is found, a message is sent to theTier 2servers 107,Tier 2servers 107 perform further operations for classifying the bot and for assigning restrictions, if required. The information regarding suspicious bots is preferably kept in the database of thetier 2server 107 for the future assessment of severity level. The process of classifying the suspiciousness and assigning restrictions is based on parameters which are configurable by an operator via a user computer shown asuser computer A 109. Configuration is preferably done viadashboard web interface 108.Tier 2server 107 preferably communicates with the relevant web servers for updating the web servers regarding the suspicious bot and the assigned restrictions. Web servers shown asweb server A 105 andweb server B 106 perform retractions on queries from the suspicious bots as required bytier 2server 107.Tier 2server 107 and web servers preferably communicate via the internet. It should be noted thatTier 1 servers andTier 2 servers are optional and the functionally detection of malicious bots can be embedded in the web servers shown asweb server A 105 andweb server B 106. It should also be noted that the classification of the suspicious bots is optional and the system can operate without such classification. The classification, if used can be embedded in any server such astier 1 server, web server and the like. - While the invention has been described with respect to a limited number of embodiments, it will be appreciated that many variations, modifications and other applications of the invention may be made.
Claims (22)
1. A method of identifying if a web client has browser capabilities, comprising:
an originating machine receiving a web page request from the web client;
the originating machine generating a page request id (PRID) and a script which, when executed by a web client with a browser, regenerates a PRID, and embedding the script in a response;
the originating machine sending the response to the web client for the web client to process and, if the web client is capable, to execute the embedded script, thereby to regenerate a PRID, and to return the regenerated PRID to the originating machine;
the originating machine or a controller machine comparing the returned regenerated PRID with the generated PRID, a match indicating that the web client has browser capabilities.
2. A method according to claim 1 wherein the PRID comprises a combination of numbers and/or characters.
3. A method according to claim 1 or claim 2 wherein the PRID is generated using a server-side code such as PHP, .Net framework or Python.
4. A method according to claim 1 or claim 2 wherein the PRID is generated using a web-server filter (such as ISAPI) or a network appliance.
5. A method according to any of claims 1 to 4 where the embedded script comprises a sequence of functions or sub-scripts.
6. A method according to claim 5 wherein the controller machine produces banks of functions or sub-scripts for the originating machine, and the controller machine updates the banks on the originating machine from time-to-time.
7. A method according to any of claims 1 to 4 wherein the embedded script comprises a portion of a document which can be interpreted by a browser and rendered into a document object model, and client side script which is used to query the document and the document object model and yield known characters or strings in return.
8. A method according to claim 7 wherein the embedded script yields different PRIDs on different browsers, depending on the way they run scripts and render documents into a document object model, and in which the different results serve to further identify the browser type and version and validate them against the HTTP headers, including User-Agent.
9. A method according to any preceding claim wherein the response comprises the requested page and the script is embedded within the content of the requested page.
10. A method according to any preceding claim wherein the regenerated PRID is returned with the next page request or in a separate request designated for the purpose.
11. A method according to any preceding claim wherein the originating machine and further comprises storage means for storing generated PRIDs and an indication of returned regenerated PRIDs.
12. A method according to any preceding claim wherein the originating machine further sends the generated PRIDs and the returned PRIDs to the controller machine which comprises storage means for storing the generated and returned PRIDs and computes matches and match ratios per IP address and session.
13. A method according to any preceding claim wherein, in the event that in excess of a predetermined number and/or ratio of PRIDs are generated in response to page requests from the same IP address without the web client returning regenerated PRIDs or returning only deformed PRIDs, the web client is suspected to be a bot.
14. A method according to claim 13 wherein a suspected bot or a chosen IP or session is set a challenge, such as a CAPTCHA, that it must satisfy before a requested web page is delivered.
15. A method according to claim 14 wherein the challenge comprises:
the originating machine generating a PRID and a script which, when executed by a web client with a browser, regenerates a PRID and reproduces the previous page request, and embeds the script in a response;
the originating machine sending the response to the web client for the web client to process and, if it is capable, to execute the embedded script, thereby to regenerate a PRID and the previous page request, and to return the regenerated PRID and the previous page request to the originating machine;
the originating machine comparing the returned regenerated PRID with the generated PRID and, in response to a match, sending the previously requested page.
16. A method according to claim 14 wherein the challenge comprises additional functions to condition the re-submission of the page request with the embedded PRID on further tests that ensure that the page is being viewed and displayed on a web browser and is not run in the background on an iFrame, a child window or a browser running as a background service.
17. A method according to any of claims 13 to 16 wherein page requests from a suspected bot are either blocked or restricted according to rules, optionally held on a rule engine.
18. A method according to any preceding claim wherein the originating machine comprises a web server or anything in front (to the web client side) of a web server, such as a logical or physical third party server or any logical or physical network machine.
19. A method according any preceding claim wherein the controller machine performs one or more other of the steps of the method.
20. An originating machine for executing the method according to any of claims 1 to 19 .
21. A controller machine for executing the method according to any of claims 1 to 19 .
22. A computer program product comprising computer code which when run on an originating machine and/or a controller machine causes it to execute the method according to any of claims 1 to 19 .
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/590,689 US20170243003A1 (en) | 2009-06-10 | 2017-05-09 | Identifying bots |
Applications Claiming Priority (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US18593809P | 2009-06-10 | 2009-06-10 | |
PCT/IB2010/052588 WO2010143152A2 (en) | 2009-06-10 | 2010-06-10 | Identifying bots |
US201113376413A | 2011-12-06 | 2011-12-06 | |
US14/987,823 US9680850B2 (en) | 2009-06-10 | 2016-01-05 | Identifying bots |
US15/590,689 US20170243003A1 (en) | 2009-06-10 | 2017-05-09 | Identifying bots |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/987,823 Continuation US9680850B2 (en) | 2009-06-10 | 2016-01-05 | Identifying bots |
Publications (1)
Publication Number | Publication Date |
---|---|
US20170243003A1 true US20170243003A1 (en) | 2017-08-24 |
Family
ID=43309296
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/376,413 Active US9300683B2 (en) | 2009-06-10 | 2010-06-10 | Identifying bots |
US14/987,823 Active US9680850B2 (en) | 2009-06-10 | 2016-01-05 | Identifying bots |
US15/590,689 Abandoned US20170243003A1 (en) | 2009-06-10 | 2017-05-09 | Identifying bots |
Family Applications Before (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/376,413 Active US9300683B2 (en) | 2009-06-10 | 2010-06-10 | Identifying bots |
US14/987,823 Active US9680850B2 (en) | 2009-06-10 | 2016-01-05 | Identifying bots |
Country Status (5)
Country | Link |
---|---|
US (3) | US9300683B2 (en) |
AU (1) | AU2010258278A1 (en) |
CA (1) | CA2764815A1 (en) |
DE (1) | DE112010002445T9 (en) |
WO (1) | WO2010143152A2 (en) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10326783B2 (en) * | 2014-03-28 | 2019-06-18 | Amazon Technologies, Inc. | Token based automated agent detection |
US10929878B2 (en) * | 2018-10-19 | 2021-02-23 | International Business Machines Corporation | Targeted content identification and tracing |
US11303695B2 (en) | 2020-04-27 | 2022-04-12 | Imperva, Inc. | Forced identification with automated POST resubmission |
Families Citing this family (65)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP2266252B1 (en) | 2008-04-01 | 2018-04-25 | Nudata Security Inc. | Systems and methods for implementing and tracking identification tests |
US9842204B2 (en) | 2008-04-01 | 2017-12-12 | Nudata Security Inc. | Systems and methods for assessing security risk |
KR101109669B1 (en) * | 2010-04-28 | 2012-02-08 | 한국전자통신연구원 | Virtual server and method for identifying zombies and Sinkhole server and method for managing zombie information integrately based on the virtual server |
US9356951B2 (en) * | 2010-07-09 | 2016-05-31 | Hewlett Packard Enterprise Development Lp | Responses to server challenges included in a hypertext transfer protocol header |
US8930896B1 (en) * | 2010-07-23 | 2015-01-06 | Amazon Technologies, Inc. | Data anonymity and separation for user computation |
WO2012107879A2 (en) * | 2011-02-10 | 2012-08-16 | Site Black Box Ltd. | DISTINGUISH VALID USERS FROM BOTS, OCRs AND THIRD PARTY SOLVERS WHEN PRESENTING CAPTCHA |
CN102737019B (en) | 2011-03-31 | 2016-08-24 | 阿里巴巴集团控股有限公司 | Machine behavior determines method, web browser and web page server |
GB2491101B (en) * | 2011-04-15 | 2013-07-10 | Bluecava Inc | Detection of spoofing of remote client system information |
WO2013097243A1 (en) * | 2011-12-31 | 2013-07-04 | Thomson Licensing | Method and apparatus for providing web pages |
EP2847680A4 (en) * | 2012-05-08 | 2016-02-17 | Fireblade Ltd | Centralized device reputation center |
US8613089B1 (en) | 2012-08-07 | 2013-12-17 | Cloudflare, Inc. | Identifying a denial-of-service attack in a cloud-based proxy service |
US9660993B2 (en) * | 2012-10-25 | 2017-05-23 | Facebook, Inc. | Event reporting and handling |
US20150156084A1 (en) * | 2012-12-02 | 2015-06-04 | Bot Or Not, Llc | System and method for reporting on automated browser agents |
DE202013100910U1 (en) | 2013-03-01 | 2013-04-09 | Alicius Schröder | CAPTCHA for distinguishing computers and people in the interaction of computers |
DE102013102068A1 (en) | 2013-03-01 | 2014-09-04 | Alicius Schröder | Method for distinguishing between humans and computers in interaction of computers to prevent unintended effects of external computer, involves constructing graphic, which has sequence of numbers, letters and characters |
US9225737B2 (en) | 2013-03-15 | 2015-12-29 | Shape Security, Inc. | Detecting the introduction of alien content |
EP3058545B1 (en) * | 2013-10-18 | 2017-08-09 | White Ops, Inc. | System for detecting classes of automated browser agents |
US9223997B2 (en) * | 2014-01-03 | 2015-12-29 | Juniper Networks, Inc. | Detecting and breaking CAPTCHA automation scripts and preventing image scraping |
US9225729B1 (en) | 2014-01-21 | 2015-12-29 | Shape Security, Inc. | Blind hash compression |
US9424414B1 (en) | 2014-03-28 | 2016-08-23 | Amazon Technologies, Inc. | Inactive non-blocking automated agent detection |
US10097583B1 (en) | 2014-03-28 | 2018-10-09 | Amazon Technologies, Inc. | Non-blocking automated agent detection |
US8997226B1 (en) | 2014-04-17 | 2015-03-31 | Shape Security, Inc. | Detection of client-side malware activity |
US9083739B1 (en) | 2014-05-29 | 2015-07-14 | Shape Security, Inc. | Client/server authentication using dynamic credentials |
US10103961B2 (en) * | 2014-07-25 | 2018-10-16 | Teacher Insurance & Annuity Association of America | Collecting client-side application performance monitoring information |
US10298599B1 (en) | 2014-09-19 | 2019-05-21 | Shape Security, Inc. | Systems for detecting a headless browser executing on a client computer |
US9954893B1 (en) | 2014-09-23 | 2018-04-24 | Shape Security, Inc. | Techniques for combating man-in-the-browser attacks |
US10380627B2 (en) * | 2015-02-12 | 2019-08-13 | Kenshoo Ltd. | Identification of software robot activity |
US9608975B2 (en) | 2015-03-30 | 2017-03-28 | Shape Security, Inc. | Challenge-dynamic credential pairs for client/server request validation |
US9986058B2 (en) | 2015-05-21 | 2018-05-29 | Shape Security, Inc. | Security systems for mitigating attacks from a headless browser executing on a client computer |
US9866545B2 (en) | 2015-06-02 | 2018-01-09 | ALTR Solutions, Inc. | Credential-free user login to remotely executed applications |
WO2017007705A1 (en) | 2015-07-06 | 2017-01-12 | Shape Security, Inc. | Asymmetrical challenges for web security |
US10230718B2 (en) | 2015-07-07 | 2019-03-12 | Shape Security, Inc. | Split serving of computer code |
WO2017015276A1 (en) * | 2015-07-22 | 2017-01-26 | Avg Netherlands B.V | Content access validation system and method |
US10375026B2 (en) | 2015-10-28 | 2019-08-06 | Shape Security, Inc. | Web transaction status tracking |
CN106686151B (en) * | 2015-11-11 | 2020-05-15 | 阿里巴巴集团控股有限公司 | IP address obtaining method and device |
US10212130B1 (en) | 2015-11-16 | 2019-02-19 | Shape Security, Inc. | Browser extension firewall |
US10402555B2 (en) * | 2015-12-17 | 2019-09-03 | Google Llc | Browser attestation challenge and response system |
US10270792B1 (en) * | 2016-01-21 | 2019-04-23 | F5 Networks, Inc. | Methods for detecting malicious smart bots to improve network security and devices thereof |
EP3414695B1 (en) | 2016-02-12 | 2021-08-11 | Shape Security, Inc. | Reverse proxy computer: deploying countermeasures in response to detecting an autonomous browser executing on a client computer |
US10855696B2 (en) * | 2016-03-02 | 2020-12-01 | Shape Security, Inc. | Variable runtime transpilation |
US10567363B1 (en) | 2016-03-03 | 2020-02-18 | Shape Security, Inc. | Deterministic reproduction of system state using seeded pseudo-random number generators |
US9917850B2 (en) | 2016-03-03 | 2018-03-13 | Shape Security, Inc. | Deterministic reproduction of client/server computer state or output sent to one or more client computers |
US10129289B1 (en) | 2016-03-11 | 2018-11-13 | Shape Security, Inc. | Mitigating attacks on server computers by enforcing platform policies on client computers |
WO2018005893A1 (en) | 2016-06-30 | 2018-01-04 | Shape Security, Inc. | Client-side security key generation |
US10135852B2 (en) * | 2016-09-09 | 2018-11-20 | Ca, Inc. | Bot detection based on behavior analytics |
WO2018203098A1 (en) * | 2017-05-05 | 2018-11-08 | Mastercard Technologies Canada ULC | Systems and methods for distinguishing among human users and software robots |
US11038869B1 (en) | 2017-05-12 | 2021-06-15 | F5 Networks, Inc. | Methods for managing a federated identity environment based on application availability and devices thereof |
US9781160B1 (en) * | 2017-05-31 | 2017-10-03 | KnowBe4, Inc. | Systems and methods for discovering suspect bot IP addresses and using validated bot IP address to ignore actions in a simulated phishing environment |
US10802453B2 (en) * | 2017-06-02 | 2020-10-13 | Bank Of America Corporation | Robotics process automation macro bot |
US10505992B2 (en) * | 2017-07-07 | 2019-12-10 | T-Mobile Usa, Inc. | Bot profile discovery |
EP3438866A1 (en) * | 2017-08-02 | 2019-02-06 | Thomson Licensing | Network device and method for determining security problems in such a network device |
GB201715801D0 (en) * | 2017-09-29 | 2017-11-15 | Intechnica Ltd | Method of processing web requests directed to a website |
US10931691B1 (en) | 2017-10-09 | 2021-02-23 | F5 Networks, Inc. | Methods for detecting and mitigating brute force credential stuffing attacks and devices thereof |
US10965683B1 (en) * | 2017-12-07 | 2021-03-30 | Wells Fargo Bank, N.A. | Login and authentication methods and systems |
US11368483B1 (en) | 2018-02-13 | 2022-06-21 | Akamai Technologies, Inc. | Low touch integration of a bot detection service in association with a content delivery network |
US11245722B1 (en) * | 2018-02-13 | 2022-02-08 | Akamai Technologies, Inc. | Content delivery network (CDN)-based bot detection service with stop and reset protocols |
US11157571B2 (en) | 2018-07-12 | 2021-10-26 | Bank Of America Corporation | External network system for extracting external website data using generated polymorphic data |
US11349981B1 (en) | 2019-10-30 | 2022-05-31 | F5, Inc. | Methods for optimizing multimedia communication and devices thereof |
FR3104781B1 (en) | 2019-12-17 | 2023-04-07 | Atos Consulting | Device for detecting fake accounts on social networks |
US11748460B2 (en) * | 2020-04-27 | 2023-09-05 | Imperva, Inc. | Procedural code generation for challenge code |
US11611629B2 (en) * | 2020-05-13 | 2023-03-21 | Microsoft Technology Licensing, Llc | Inline frame monitoring |
US11558183B2 (en) * | 2020-05-15 | 2023-01-17 | Bank Of America Corporation | System for exchanging symmetric cryptographic keys using computer network port knocking |
US11445003B1 (en) * | 2021-06-22 | 2022-09-13 | Citrix Systems, Inc. | Systems and methods for autonomous program detection |
US11977779B2 (en) | 2022-05-11 | 2024-05-07 | Bank Of America Corporation | Smart queue for distributing user requests to automated response generating systems |
US11889153B2 (en) | 2022-05-11 | 2024-01-30 | Bank Of America Corporation | System and method for integration of automatic response generating systems with non-API applications |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20060265493A1 (en) * | 2005-05-20 | 2006-11-23 | Richard Brindley | Fraud prevention and detection for online advertising |
US20070260520A1 (en) * | 2006-01-18 | 2007-11-08 | Teracent Corporation | System, method and computer program product for selecting internet-based advertising |
US20080301808A1 (en) * | 2007-05-31 | 2008-12-04 | International Business Machines Corporation | Internet robot detection for network distributable markup |
US20090241174A1 (en) * | 2008-02-19 | 2009-09-24 | Guru Rajan | Handling Human Detection for Devices Connected Over a Network |
US20100031315A1 (en) * | 2003-08-26 | 2010-02-04 | Wu-Chang Feng | Systems and methods for protecting against denial of service attacks |
Family Cites Families (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20060190561A1 (en) * | 2002-06-19 | 2006-08-24 | Watchfire Corporation | Method and system for obtaining script related information for website crawling |
US20060136249A1 (en) * | 2004-12-17 | 2006-06-22 | Kagan Lloyd D | Method for recycling used items |
US20070255821A1 (en) * | 2006-05-01 | 2007-11-01 | Li Ge | Real-time click fraud detecting and blocking system |
US8849746B2 (en) * | 2006-12-19 | 2014-09-30 | Teradata Us, Inc. | High-throughput extract-transform-load (ETL) of program events for subsequent analysis |
US8316310B2 (en) * | 2008-08-05 | 2012-11-20 | International Business Machines Corporation | System and method for human identification proof for use in virtual environments |
US20100250397A1 (en) * | 2009-03-24 | 2010-09-30 | Gregory Ippolito | Internet Retail Sales Method and System Using Third Party Web Sites |
-
2010
- 2010-06-10 US US13/376,413 patent/US9300683B2/en active Active
- 2010-06-10 CA CA2764815A patent/CA2764815A1/en not_active Abandoned
- 2010-06-10 DE DE112010002445T patent/DE112010002445T9/en not_active Expired - Fee Related
- 2010-06-10 AU AU2010258278A patent/AU2010258278A1/en not_active Abandoned
- 2010-06-10 WO PCT/IB2010/052588 patent/WO2010143152A2/en active Application Filing
-
2016
- 2016-01-05 US US14/987,823 patent/US9680850B2/en active Active
-
2017
- 2017-05-09 US US15/590,689 patent/US20170243003A1/en not_active Abandoned
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20100031315A1 (en) * | 2003-08-26 | 2010-02-04 | Wu-Chang Feng | Systems and methods for protecting against denial of service attacks |
US20060265493A1 (en) * | 2005-05-20 | 2006-11-23 | Richard Brindley | Fraud prevention and detection for online advertising |
US20070260520A1 (en) * | 2006-01-18 | 2007-11-08 | Teracent Corporation | System, method and computer program product for selecting internet-based advertising |
US20080301808A1 (en) * | 2007-05-31 | 2008-12-04 | International Business Machines Corporation | Internet robot detection for network distributable markup |
US20090241174A1 (en) * | 2008-02-19 | 2009-09-24 | Guru Rajan | Handling Human Detection for Devices Connected Over a Network |
Non-Patent Citations (1)
Title |
---|
Park et al., "Securing web service by automatic robot detection" May 2006 ATEC '06: Proceedings of the annual conference on USENIX '06 Annual Technical Conference; Publisher: USENIX Association, Pages:1-18. * |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10326783B2 (en) * | 2014-03-28 | 2019-06-18 | Amazon Technologies, Inc. | Token based automated agent detection |
US10929878B2 (en) * | 2018-10-19 | 2021-02-23 | International Business Machines Corporation | Targeted content identification and tracing |
US11303695B2 (en) | 2020-04-27 | 2022-04-12 | Imperva, Inc. | Forced identification with automated POST resubmission |
US11716374B2 (en) | 2020-04-27 | 2023-08-01 | Imperva, Inc. | Forced identification with automated post resubmission |
Also Published As
Publication number | Publication date |
---|---|
WO2010143152A3 (en) | 2011-05-05 |
US9680850B2 (en) | 2017-06-13 |
DE112010002445T5 (en) | 2012-10-25 |
US9300683B2 (en) | 2016-03-29 |
US20120090030A1 (en) | 2012-04-12 |
US20160119371A1 (en) | 2016-04-28 |
DE112010002445T9 (en) | 2013-02-07 |
AU2010258278A1 (en) | 2012-01-12 |
WO2010143152A2 (en) | 2010-12-16 |
CA2764815A1 (en) | 2010-12-16 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9680850B2 (en) | Identifying bots | |
US11606374B2 (en) | Analyzing client application behavior to detect anomalies and prevent access | |
US9954841B2 (en) | Distinguish valid users from bots, OCRs and third party solvers when presenting CAPTCHA | |
US11722520B2 (en) | System and method for detecting phishing events | |
CN109274637B (en) | System and method for determining distributed denial of service attacks | |
US8949990B1 (en) | Script-based XSS vulnerability detection | |
US11212305B2 (en) | Web application security methods and systems | |
US8850567B1 (en) | Unauthorized URL requests detection | |
US20190222607A1 (en) | System and method to detect and block bot traffic | |
Shrivastava et al. | XSS vulnerability assessment and prevention in web application | |
US11770385B2 (en) | Systems and methods for malicious client detection through property analysis | |
CN110362992A (en) | Based on the method and apparatus for stopping in the environment of cloud or detecting computer attack | |
Chanti et al. | Classification of anti-phishing solutions | |
US20230291758A1 (en) | Malware Detection Using Document Object Model Inspection | |
CN108028843A (en) | Passive type web application firewalls | |
US10686834B1 (en) | Inert parameters for detection of malicious activity | |
Selvamani et al. | Protection of web applications from cross-site scripting attacks in browser side | |
Jaswal et al. | Detection and Prevention of Phishing Attacks on Banking Website | |
Bashir et al. | The Fuzzy Experiment Approach for Detection and Prevention of Phishing attacks in online Domain | |
Chu et al. | An investigation of hotlinking and its countermeasures | |
CN113542287A (en) | Network request management method and device | |
Veličković et al. | Web Applications Protection from Automated Attacks by the reCAPTCHA API | |
Fairweather et al. | An enhanced approach to preventing the SSLstripping attack | |
Sarika | Browser Security: Attacks and Detection Techniques | |
CN110138719B (en) | Network security detection method and device and electronic equipment |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: FIRE BLADE LTD., ISRAEL Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:RAPAPORT, SHAY;AZARIA, EREZ;REEL/FRAME:042556/0827 Effective date: 20160113 |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |