|
|
|
Web Application Scanners: Technical Challenges
While web application scanners have been around for some time, users have expressed significant dissatisfaction with their performance. Because of the unique nature of Web Applications themselves, crawling and testing them is an extremely difficult technical challenge.
JavaScript
Web Application Scanners must first crawl websites before they can test the links for vulnerabilities. If a scanner cannot find a link, then it cannot test it. This means that a user will likely think that the link has been tested when in fact it has not.
Basic crawling involves going to the first page and then looking for links in the HTML. This is fairly simple. As web applications have increasingly used JavaScript to create more dynamic content, crawling has become increasingly difficult.
Most web scanners claim that they can crawl JavaScript. ; This is an easy claim to make. Some JavaScript is quite easy to crawl because the links are fully embedded in the JavaScript code. Unfortunately, many websites use JavaScript functions to actually create the links by concatenating the component parts of the link.
For example, the link
www.shoppingsite.com/shirts/bluepolo.html
could be expressed
www.shoppingsite.com/ + shirts/ + bluepolo.html
Obviously, creating this link would require the scanner to obtain the JavaScript code and then execute it to create the link. Given the significant effort made by browsers to parse JavaScript, replicating this effort is no easy task.
Additional complexity can be created by websites that have conditional links based on intrinsic events for HTML elements, such as those created by OnMouseOver functions.
NTOSpider has the most advanced Web Application Scanner JavaScript parser on the market. All JavaScript functions are parsed as they would be in a browser to ensure that all links are tested.
Fortunately, this is among the easiest aspects of Web Application Scanner functionality to test. Crawling web applications is perfectly legal and benign (search engines like Google continuously crawl websites). Testing scanners is as simple as pointing them at a few websites with extensive JavaScript and comparing results.
|
|
Authentication
Authentication can be quite difficult for web application scanners. Authentication schemes use multiple redirects that go across domains and ports and use multiple cookies that often change frequently. Additionally, if the site uses form authentication, the scanner must identify the form and often must use JavaScript to submit usernames and passwords. Web applications must keep track of all of this; a single error will result in the authentication failing.
NTOSpider has the most effective automated authentication scheme in the industry. NTOSpider supports Form, Basic, NTLM, dual authentication and single sign-on. It is not necessary to provide “hints” to NTOSpider as to what a login form looks like or what the application’s customized error page looks like.
Session Management
Once a web application scanner is logged in, it must maintain the session. This can be difficult as cookies, ports and domains change. Additionally, scanners must learn to recognize logouts and avoid them. Because of the large number of links that must be crawled and the tests that must be performed, scanners must be multi-threaded (i.e. make multiple server requests at once). Session information must be kept track of and coordinated across multiple, simultaneous threads.
NTOSpider utilizes a sophisticated state management model that has been successfully tested against some of the most complex session management schema.
|
|
Forms
In order to get to many pages, web application scanners must submit data to forms.
The first difficulty here is recognizing the form and the type of data it is expecting. Entering data into forms can also be extremely difficult; often multiple JavaScript functions must be evaluated and executed to enter form data.
NTOSpider uses advanced heuristics to evaluate HTML content to determine variable types in forms prior to entering data. NTOSpider’s industry-leading JavaScript engine is fully integrated into its form population engine, resulting in comprehensive form population and form-based crawling.
|
|
Automation
The combination of JavaScript, complex authentication, session management and form inputs makes automating web application scanners extremely difficult. While it may seem that manual work-arounds (like session takeover and record/playback) are feasible alternatives, they require that users know the links and what they are missing. For example, if a site has 1,000 links that do not require JavaScript and 3,000 behind them that do, a tester may be unaware of the other links. Security departments are frequently responsible for hundreds or thousands of websites, each with hundreds or thousands of links. Moreover, these applications are constantly changing. Being familiar with all of them is an impossible task. The job of the scanner is to automatically discover the links; if it fails, testers may be unaware of them.
Additionally, if testers are responsible for testing a large number of websites, manual walk-throughs are not an option. Hand crawling 100 links on 1,000 sites is simply too time consuming and requires trained personnel to avoid missing links.
NTOSpider is designed to be truly automated, allowing for manual interaction, but not requiring it.
|
|
Reporting Multiple Vulnerabilities in One Root Cause
Most SQL injection, Blind SQL injection and parameter tampering attacks can be fashioned in multiple ways.
The attack
http://www.store.com/login.cgi?u=johnsmith&password’+OR+1=1
can also be fashioned as
http://www.store.com/login.cgi?u=johnsmith&password”+OR+1=1
or
http://www.store.com/login.cgi?u=johnsmith&password%27+OR+1=1
This makes manual testing for these exploits extremely time consuming. Moreover, once results are discovered, a single input validation failure can create hundreds or thousands of observed vulnerabilities. This makes it crucial that vulnerabilities be grouped by root cause so that they can be assigned to developers for remediation.
This problem is made more acute by the fact that a single root cause can create nearly infinite links. The classic example is a calendar where each unique combination of a day, month and year is a link.
Grouping vulnerabilities by root cause allows developers to have a concise review of where they made their mistakes and to plan improvements in security architecture going forward. This allows the improvement of secure coding practices throughout the Software Development Lifecycle.
NTOSpider’s Root Cause analysis groups vulnerabilities by root cause and allows security teams to easily communicate vulnerabilities to developers and retest them. Developers can use these reports to learn where they are making errors and improve their coding going forward.
Minimizing False Positives
Eliminating false positives in web application scanning is extremely difficult. Network vulnerabilities are signature-based, so there is much less of a chance of false positives. Most web application security vulnerabilities are not signature-based because custom web applications all have different responses to attacks.
Web application scanners try to eliminate false positives by comparing page signatures (essentially a mathematical distillation of page content) of known bad pages to responses to attacks. Unfortunately, this process is very difficult because small differences in pages (e.g. a changing ad banner) can create false positives. Given the huge number of attacks made because of the polymorphic nature of web application attacks, even a small false positive ratio can create hundreds or thousands of false positives. False positives can completely destroy the credibility of the security teams reporting them because development teams come to believe that there are no real vulnerabilities.
NTOSpider uses fuzzy logic to evaluate application responses and to differentiate custom error pages from vulnerabilities. NTOSpider users have reported that false positive rates have dropped by an order of magnitude or more when they switched to NTOSpider.
|
|
Inventorying Websites
For security teams to have a comprehensive understanding of the websites they are responsible for reviewing, it is necessary to have an understanding of the site’s architecture and elements. The challenge is that websites have hundreds or thousands of pages with thousands or tens of thousands of elements, including forms, hidden form fields, cookies and external links.
NTOSpider has a series of reports that detail all site elements and consolidate the site architecture into an interactive map that allows users to quickly gain an understanding of the site architecture and what the site does.
For More Information on inventorying web applications, please read "Inventorying Your Site: You Can’t Defend What You Can’t Inventory"
|
|
|
|
