This week a developer from Google released a new vulnerable test app named “Firing Range” which I have been digging into for the last few days. This has been of particular interest because of course I work on the best web scanner on the market, NTOSpider. And also because I have spent many months over this year working on a new test app called Hackazon.
Over the years, I have spent time reviewing all the existing test apps from the great OWASP VWAD list and each certainly has their strengths and weakness, but all are useful. Maybe I will spend time breaking down each test app in the future, but for today I am going to discuss the newest entrant from Google.
Firing Range – What It Is
Firing Range is a “test bed for Web application security scanners that provides coverage for a wide variety of cross-site scripting (XSS)” vulnerabilities and according to Google security engineer Claudio Criscione “they needed a testbed with which [to] analyze current and future scanning capabilities.” In addition, according to Google,
Firing Range “predominantly looks for XSS bugs, but there are other vulnerabilities it can find as well. It differs from previously available tests for XSS scanners in that it doesn’t try to emulate all the possible attack scenarios in a specific application. Instead it relies on automation based on a collection of unique bug patterns drawn from in-the-wild vulnerabilities observed by Google.”
Firing Range – What It Is Not
It is not a “fake site” type of test app or a CBT (computer based training) type of app. A “fake site” is one that is intended to look like a real site, but with vulnerabilities intentionally sprinkled about. Well-known examples of fake site test apps are Hackme Bank/Casino/Books. The CBT test apps are geared toward teaching about web security and usually help walk the user through the process of understanding the various types of attacks and letting them try the attacks, examples of a CBT type of app would be WebGoat.
Firing Range – What It’s Good At
Firing Range is specifically focused on one of Google’s biggest concern, which is Cross-Site Scripting (XSS) attacks and it enables testing for a great number of possible XSS attacks. It is not a complete list, but it is damn close and is the best collection of tests I have seen in one place. Firing Range is nicely organized with a sort of self documenting file structure so its generally clear which attack will work based on the directory/filename.
While there are some benefits and nice features of Firing Range, many of the tests are highly impractical and unrealistic. While Google claims that it wanted to represent vulnerabilities that are seen “in the wild,” the way the vulns are implemented makes it unlike anything seen in the wild. Here are some examples:
Lets looks at /tags/meta?q=a where you attack the ‘q’ parameter on the URL.
Request #1: http://public-firing-range.appspot.com/tags/meta?q=abc
Response #1: Invalid input, no tags
Request #2: http://public-firing-range.appspot.com/tags/meta?q=xyz
Response #2: Invalid input, no tags
Notice that the responses to request 1 & 2 are just “Invalid input, no tags” and the input value does not reflect in the response. Normally when a pen tester is doing a XSS attack, the first thing he/she would do is figure out if the input will “reflect” or show up in the response. If it does, then the pen tester would begin the process of trying different XSS payloads, and if it doesn’t reflect the pen tester might still do a handful of attacks to see if it causes an error response that has the reflection taking place. So you only want to do extensive testing if you’re dealing with a reflection point and avoid having to blast every input with hundreds or thousands of possible attack variations. When using an automated scanning tool, this helps significantly reduce scanning times while still maintaining high quality results.
In the following request #3, we see that if we send a <meta> tag that it gets detected and then reflected after some HTMLTidy work is done to it. Notice the y now has quotes around it and the tag is properly closed. So not only does it only reflect when you send the attack it wants, but it also makes changes to your attack payload which can make it hard to detect if your attack injection is showing up as expected (could trick regex you may be using).
Request #3: http://public-firing-range.appspot.com/tags/meta?q=<meta%20x=y>
Response #3: <meta x=”y” />
This is just one example among many that make it less than ideal as a scanner test site, because its likely that a dumb slow scanner could do well on this site, but scanners using smarter approaches will end up struggling because of the unrealistic situations replicated.
Does This Mean Firing Range Is Bad?
Not at all, and in fact I am really excited about most of Firing Range. I will counter my previous criticism with a big compliment that the structure of the test bed is well organized and generally well conceived for the specific tests it covers. I imagine that there are reasons that it would be hard to make sure that each page only allows the specific attack its supposed to be vulnerable to, such as meta tag but not any other attacks. The way the developer, Mr. Criscione, accomplished this was a logical convenience for him but leaves room for improvement.
Firing Range has proven useful to us already as it has already helped us uncovered some things we needed to do better with our scanner, and we have already started to use to improve some of the routines and tests in our scanner.
I have not seen any other real dissections or analysis of Firing Range and it is early to give broad conclusions, but there is a lot to like about this test bed, but also some things that could use improvement. I spoke with a buddy [Need to ask him if I can use his name here.] that works on an open source web scanner and he too agreed that the unrealistic tests are a problem and is frustrating, but like myself is glad to have another test app available.
I plan to reach out to Mr. Criscione next week to discuss and get one of my developers to see if we can help contribute some code that will improve the tests to be more realistic.
Let us know your thoughts!
More on Google’s Test App, Firing Range
Google’s Blog Post – “Ready, Aim, Fire – an open-source tool to test web security scanners”