Introduction

When the latest report from Larry Suto was set to come out and we had seen previews of the results, our first reaction was “Wow, we did great, but why did we miss those 9 vulns?!” followed by “Whoa – why did the other scanners miss so many vulnerabiities?” and then “Oh no, here we go again. Another round of getting unfairly blasted by the other vendors and their users“.

We certainly were not disappointed by the response from the other vendors and their users, but overall things seem to be different than they were in 2007 when Larry did his first report. In the latest report it is clear that Larry had learned at least two things from his first experience.

The first was that he needed better supporting data which he has certainly done this time by including the full breakdown of the vulns by site and vendor. The second was that he would need to provide for “Trained” scans, because most of the vendors made quite a protest that it was impossible to get proper results without it. My personal feeling on the matter is that “Point-and-shoot” is the most likely way that users will run scans and for that reason it is the responsibility of the scanner to do as much as possible on its own.

Because Larry did the “Trained” scanning this time around, this only leaves the other vendors with the ability to claim that he didn’t do a good enough job with the training. I think Jeremiah Grossman states it the best in his post “Scanner vendors should take into consideration that Larry Suto is certainly more sophisticated than the average user. So if he couldn’t figure out how to run your tool ‘properly’ take that as constructive feedback.”

A friend of mine also had a great line when he said that “from these results it appears that a monkey could get better results from NTOSpider, than an experienced security consultant can with the other scanners“.

General claims about unfair/incorrect vulnerabilities

There is a growing bit of speculation about this topic. I believe that Larry did his best to validate his results and he seems to be very determined to make sure that his results were accurate and wants to get to the bottom of every accusation/concern.

The False Positives

Even with Larry’s best efforts, some issues did slip by. Here’s what I know of so far:

• RFI on: http://testphp.acunetix.com/comment.php (phpaction) Acunetix testphp – According to comments on the blog post at ha.ckers.org, Bogdan Calin explains that one of the vulns would have been impossible to find by other scanners due to the requirement of inside knowledge. This was unfairly included.
• SQL Injection on: http://demo.testfire.net/bank/ws.aspx (SOAP: transferDate) IBM testfire – IBM told Larry that they checked the code and this is not vulnerable
• SQL Injection on: http://demo.testfire.net/bank/ws.aspx (SOAP: transferAmount) IBM testfire – IBM told Larry that they checked the code and this is not vulnerable
• RFI on: http://www.webscantest.com/ (incdir) NTO webscantest – For the same reason as the Acunetix RFI, this vuln is only exploitable from a URL that only we would have known. This was unfairly included.

Thats a total of 4 out of the 154 vulns listed as valid. Keep in mind that 3 of these are included in the 9 we had missed, which brings our missed vulns number down to only 6!

How did NTOSpider find these vulns without cheating?

There have been a number of these questions that have been coming in to me via email, blogs, vendor responses, etc. I will use this opportunity to show how many of these issues are found. Of course I prefer not to give away our secrets, but I will make this exception for the sake of the community.

• XSS on: http://testphp.acunetix.com/secured/newuser.php (uaddress/uphone/uemail/ucc/upass/urname)Our advanced forms analysis engine is able to identify numerous types of forms, as well as hundreds of commonly used inputs to automatically fill out forms in order to get better and deeper coverage of web applications without training. This form was identified as a registration form, so the user/pass values were randomized for each test, which allowed us to avoid being stopped by “Username already taken” responses like the other scanners apparently did.
• Crawling and Vulns on: /forgot1.asp, /forgot2.asp, /join1.asp, /login/login.asp, and /pformresults.asp on the HP site. NTOSpider does limited forced browsing in order to find common resources, and it appears this helped us find things that other scanners missed.
• In order to find login forms we have a list of common login pags we force browse, among them are /login, /login.php, /login.asp, /login.cgi, etc.
• NTOSpider also looks for backup files such as .bak(among other things) for all pages it sees during the crawl. If it find any such files, it looks for sourcecode as well as any forms/links it hadnt seen before.

This is exactly the kind of thing I think users expect automated scanner to be doing, and it was not at all added to be specific to this site.

Here is what I found (grab your favorite local proxy and follow along).

1. SQLi on /forgot1.asp (HP site) From login.asp (force browsed) there is a link in it to forgot.asp which has a form that submits (GET) to forgot1.asp
2. XSS on /forgot2.asp (HP site) When you submit to the previous explained forgot1.asp it 302′s you to forgot2.asp
3. XSS on /join1.asp (HP site) From login.asp (force browsed) there is a link to the join.asp which has a form that submits (GET) to join1.asp
4. XSS on /login/login.asp (HP site) From /login (force browsed) we get 302′d to /login/ which then 302′s to login.asp (there is also a link to login.asp in the body).
   This final 302 is relative, so that works out to /login/login.asp
5. XSS on /pformresults.asp (HP site) From the main page there are links to /pindex.asp and /login.asp When we force browse for .bak files we end up finding /pindex.asp.bak and /login.asp.bak, both of which have forms that POSTs to /pformresults.asp

As you can see, all these are in fact able to be found by very basic forced browsing and then reasonably following the links. The fact that NTO was the only one that seems to be able to find them is simply a credit to our scanner, and further shows the limitations of the other approaches. None of these forced browsed pages were exotic in the least, and it seems unusual that some, much less all, the other tools missed them.

• Normal XSS on: http://www.webscantest.com/bjax/servertime.php (msg) Acunetix was the only other to find this. There are some that do not believe that you can exploit XSS on pages with Content-Type set to “application/xml” and with XML content. This is a very risky assumption to make, because not all browsers behave the same. By simply setting up a redirecting form that submits on body load, and that includes the correct attack payload its possible to trick some browsers (including Firefox) into thinking it is XHTML when you send XML tags that create a new namespace. When you do this the the javascript will execute.
• AJAX XSS on: http://www.webscantest.com/bjax/index.php (calling to servertime.php msg)The /bjax/servertime.php is actually used by the /bjax/ page in XHR calls.
  • On this page you can enter a message and it is included in the XHR request, and then echo’d back along with other info.
  • Our scanner sees this data being pushed back into the DOM like this: Server got the message: “Hello AJAX”
  • So we try XSS attacks. Heres a simple one one would attempt: <a onmouseover=”alert(123)”> Hello AJAX</a>
  • However when you try this you get a parser error on the event handler (because the <a> is basically creating a new child node).
  • NTOSpider understands this and tries it again with CDATA tags <![CDATA[<a onmouseover="alert(123)"> Hello AJAX</a>]]>
  • Now the data is held together in the message content and gets put on the page. Mouse over the “Hello Ajax” and you will see the popup.

  I will admit that this is a bit exotic and that a link cannot be sent to some user that would exploit this, but there are other possible ways of delivering the payload. So this kind of vuln is on the site to see how well we can handle automated AJAX attacking and IMHO is a legit finding that a scanner should be able to handle since more and more of the web is moving the way of AJAX.

• Form Population: On http://scanme.webscantest.com/shutterform/ Our advanced forms analysis engine is able to identify hundreds of commonly used inputs to automatically fill out forms in order to get better and deeper coverage of web applications without training. If you look at the source of this page, the “input name” values are var1 through var7, and if you don’t fill out the form correctly it bounces demanding that the correct data be provided. NTOSpider’s advanced Forms Analysis Engine not only looks at the “input name” attribute, but we also looks at the presentation layer to find the text strings on the page that tells the user what to input. This analysis is able to look at the page like the user does and is able to identify what values to put into each input.

The Blogosphere Response

The reaction has been generally very positive, or simply restated the results. Some took this as a jump off to make other interesting points.

The one major exception worth noting was from a die hard on the fact that web scanners should only be for the elite among us (also an Ex-Spi Dynamics employee) who provide quotes such as “These types of tools are very complicated, require expert customization on a per site basis for the best/most accurate results, and untrained point and shoot is a TERRIBLE comparison methodology.” to which I disagree.
This is simply bad thinking about this space and what can be done with technology. Yes, web app security scanning is hard to automate, but this does not mean that we should not all be doing our best to make this as easy as possible for the 99.9999% of the IT staff out there that wouldnt qualify among the “elite”. Many of our customers have 20-50 apps, and some have hundreds, making it completely impossible to have the time to train the tool to each app.

The Vendors

It seems that because the new Larry Suto’s study ends up showing that NTOSpider has better results than the other scanners that it must be due to some form of cheating or that Larry some how screwed up in his testing. I cannot and would not answer for Larry, but here are my responses to each of the vendor’s claims about this report.

The HP Response – by Matt Wood (HP Web Security Research Group) [link]

HP makes claims of having better results in their own research with claims they “discovered over twice the number of vulnerabilities“, and “also identified several vulnerabilities that we believe to be false positives or not consistent representations with the author’s vulnerability results” and even that they “found that WebInspect identifies several vulnerabilities in other vendors websites not mentioned in the report“. It is hard to evaluate this claim without HP backing it up with specific data which, for some reason, they do not.

According to HP “The percentage comparison is also unfairly weighted against vendors whose sites expose more vulnerabilities than others. For example, NTO’s Webscantest exposes a supposed forty-seven vulnerabilities, while Acunetix’s AspNet and PHP sites only expose a combined thirty-five. This is a poor statistical comparison between vendors, as it is clear that NTO’s applications will likely find all vulnerabilities on their demo site giving them a significantly higher starting point than other vendors.”
This seems like a reasonable claim. However, it is nothing more than a clever attempt to cloud the discussion when you look at the results after removing the NTO test site. The results do change, but only slightly and NTOSpider still remains in the lead by about the same distance.

Results, Excluding NTO test site			Acunetix		Appscan		BurpSuitePro		Hailstorm		NTOSpider		Qualys	WebInspect
			PaS	Trained	PaS	Trained	PaS	Trained	PaS	Trained	PaS	Trained	PaS	PaS	Trained
Grand Totals		Vuln’s Found	41	50	59	57	30	41	43	67	97	100	41	37	45
		% Found	38.32%	46.73%	55.14%	53.27%	28.04%	38.32%	40.19%	62.62%	90.65%	93.46%	38.32%	34.58%	42.06%
Valid Vulns	107	Vuln’s Missed	66	57	48	50	77	66	64	40	10	7	66	70	62
Pages	193	% Missed	61.68%	53.27%	44.86%	46.73%	71.96%	61.68%	59.81%	37.38%	9.35%	6.54%	61.68%	65.42%	57.94%
		FP’s Reported	2	2	0	0	2	4	6	5	1	1	2	1	2

Lets also drink the kool-aid for a moment and assume that they are correct about their ability to find “twice the number of vulnerabilities” in the “point-and-shoot” when properly configured by their in-house experts. This would put them at a fairly impressive 74 findings for a 69% accuracy rating, which is still not nearly as well as we had performed in a simple default “point-and-shoot” scan.

The Unofficial Response by IBM Team Member(s)

In 2007 when Larry did his first report, IBM was one of his harshest critics. At that time IBM stated that “In order to achieve good results, web application scanners should be used in conjunction with manual security assessment, which requires close acquaintance with the web application and its different functionalities.” and that “his experiment methodology was never fully disclosed to the public, therefore could never be confirmed nor rebutted.” and I am sure this is part of the reason that Larry produced a report with manual training and clearly defined methodology. I do agree that it would have been better if Larry had provided those details, but I also think his 2007 findings were fairly accurate.

For this new 2010 report, to date, IBM has not responded in public on their website. This does not mean they have ignored the situation. In fact, they have been spending time investigating the issues and I have received a few emails from a concerned Larry Suto to get my response to a couple concerns coming from someone on their team.
(Note: The quotes from the emailer(s) were asked in an unofficial/offline capacity.)

The first email from Larry concerning a claim that XSS against cookies was invalid.

From Larry: They are concerned that there is "no possible way to trick a user into submitting a request, with a maliciously crafted cookie value."

Some time ago we found an instance where two XSS vulns worked together to help us exploit XSS via a cookie. On the home page (unauthenticated section) I was able to perform a normal XSS attack. Then behind the authenticated portion of the site there was a situation where the value of a cookie was displayed in the page content. So using the first XSS vuln, I performed an attack which set the value of the cookie via javascript (document.cookie =) with its value being an attack payload. So now when the user got to the authenticated section and on the page that displayed the contents of the cookie, presto the payload was executed and now it had the benefit of being on the logged_in/private section of the site.
We have since found more instances of this type of thing and really thought it was a settled issue and had enabled this attack in our tool. I will confess that I was not aware of the “big debate” about this in our industry. It also just so happened that on the same day Jeremiah Grossman posted about this in his Converting unimplementable Cookie-based XSS to a persistent attack post. I will chalk this one up to academic debate, and I understand that he has admitted that he may have to change his view on this attack based on the new evidence/examples.

The second email from Larry was concerning a claim about http://testphp.acunetix.com/newuser.php

"In order to properly test this script, you must have a unique username with each form submission, or else the submission fails (it creates a new user). This can be handled by using "Sequence Variables" in AppScan, ( http://www-01.ibm.com/support/docview.wss?uid=swg21327313 )"

Well, we do. On top of that, we also do it automatically on many types of forms, including registration forms, as described above. Of course, we also have manual training features, but they are not the default/required way of doing things with NTOSpider.

The third email from Larry was about us possibly “rigging” our crawler.

They are concerned that you may have rigged these results:
Some sample URLs, that cannot be reached by following links in the application -
SQLi on /forgot1.asp (HP site)
XSS on /forgot2.asp (HP site)
XSS on /join1.asp (HP site)
XSS on /login/login.asp (HP site)
XSS on /pformresults.asp (HP site)
If the URLs cannot be reached by following the links...only the application owner should know about them, right?

I was curious about this claim, and wondered myself why we were the only one (other than HP) to find these links and vulns, so I investigated the path of the links. The details are described in the section above. None of these forced browsed pages were exotic in the least, and it seems unusual that all other vendors (except our tool and HP’s) tools missed them.

The forth email came directly to me about /bjax/ and /bjax/servertime.php on our site which it seems that the emailer believed is ‘unexploitable’ because the response is returned in XML and then embedded as an element via AJAX.

We had a lengthy debate about the validity of the XSS vulns against these pages. In the end I was able to detail a couple ways it is possible including creating test pages for easy verification. The emailer continues to protests the legitimacy of the AJAX version of the finding, since there is no
way to send a link to someone to exploit it; to which they are
correct. However, if the data were to be stored persistently on the
server and then other people end up getting the content via XHR
requests, then it would execute on their machines.

Note: While I do pick on the IBM team member(s) a little, I appreciate the willingness to engage in debate and that they are making an effort to research the details before flying off with an "Official IBM Response" that would have otherwise included these inaccuracies.

Update: I removed a direct quote along with changed the section header to reflect that these were offline discussions and not official responses from IBM.

The Cenzic Response by Lars Ewe (CTO) [link]

The only response from Cenzic was a blog post on Feb 10th that explained what the report was about and that they were studying the results. They say that they are performing their own test and "Interestingly enough, however, our own results were somewhat different
than Larry's findings. We're in current discussions with Larry to
better understand how he configured the product and confirm his
assumptions versus our own."

This seems reasonable, and gives them wiggle room to make their disputes later on if they so choose. I heard from Larry that they were not happy with the results, but they do appear to be interested in learning from the results and improving their tool. I will be curious what their further public response will be, if anything at all. Their tool actually did well in comparison to the other scanners coming in second place at finding vulns after training, so maybe they are happy to leave it at that.

The Acunetix Response by Bogdan Calin (Acunetix CTO) [link]

Bogden starts by trying to make a big deal of the fact that Larry meant "Point and Shoot" and that even covers creating login macros (this is some level of training) which is the only way that their scanner is able to handle form based authentication. I can understand their point, and maybe next time Larry will allow for some minimal training for scanners like Acunetix. Because of this they point out 4 vulnerabilities on two sites (8 total) that they should have otherwise been able to find. This still only brings them up closer to their Trained results at 46%. Does this really justify their claim that "Therefore, Acunetix WVS was clearly disadvantaged in this comparison report."? I hardly think so.

Back to the whole reason that their scanner didn't have the benefit of being logged in. This is due to their own technology choice, which according to Bogdan "We don't have automated form based login support because I don't see the point in that. The application can log you out anytime so I don't see much gain from implementing that. Our Login Sequence Recorder is much more flexible because it automatically detects when the session is invalidated and reruns the login sequence."

The flaw in his thinking is that invalid session detection is somehow a reason you cannot handle automated form based login. It is possible to have both automated form based login and automated invalid session detection, because we do. We have automated both, but also allow our users to manually define either in the rare event that our automation isn't able to correctly deal with either or both. I will accept that this is a difficult bit of technology and that it has taken years of development to get it to work as well as our does, so I can understand Bogdan's preference to take the easy way out and just force the user to do the job. But, at NTO we believe that we should be doing as much as possible to save our users time and effort by tackling these hard issues.

Finally, Acunetix closes by accusing NTO of cheating. They do this by showing a log entry from their web server which includes evidence that someone at NTO with "access to the NTOSpider source code" had been running tests. The question is "Did NTOSpider find these vulnerabilities outside of the debugger?" to which the answer is YES. It is not like these test sites are exactly secret, and we did not "rig"/(hard-code) our scanner to find these vulns. We constantly review our methodologies to make sure we are always improving so that we do not miss vulnerabilities the next time.

To address their specific claim; After finding out about the report Larry was about to release, which was no secret to the various vendors, I told my team about this and you bet that some of our developers decided to run scans to see what the results were going to look like. To clear up any doubt, look at the time stamp on that log, which is [20/Jan/2010:08:44:58 +0100], the day after the email below. Take into account that this was already past the time that Larry had already finished most of his testing, because I had heard from him in the days prior that he was done running the scans and was putting together his results. I had asked him if WhiteHat and Qualys had been included, and found out that they had not been.
I suggested to Larry that they should be and so I sent the following email to Jeremiah Grossman:

Date: Tue, 19 Jan 2010 16:28:39 -0800
From: Dan Kuykendall <dk@nto****ves.com>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091204 Thunderbird/3.0
MIME-Version: 1.0
To: Jeremiah Grossman <jeremiah@wh****ec.com>,
Larry Suto <larry.suto@*****.com>
Subject: Larry Suto's next report
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi Jeremiah,

I dont know if you have ever been introduced to Larry Suto, but he is
finalizing his next report about the scanners. This one is pretty simple
in that hes taking each of the scanners and running them against each of
the vendors test sites.
He forgot to include you guys as he was getting this done and I
mentioned it to him
<snip>
I hope all is well and that your having a great 2010.
<snip>

My email should help show that I was actually interested in what was best for the community and helping look out for others in this small community we are in. I have great respect for Jeremiah and think his response as to why WhiteHatSec declined participation was well explained. This email was from the day before my developer had decided to look at this site (see the log) to check things out, and it was clear that Larry was already about done collecting the data, much less running his scans with NTOSpider. From what Larry has told me, most (if not all) the vendors knew by this point and certainly Acunetix knew well before this point since they had provided a license to Larry to perform these scans.

Again, I think that a focus on the data and study will be more useful to the community than trying to shoot the messenger.

The White Hat Security Response by Jeremiah Grossman (CTO) [link]

Since Jeremiah and team declined, their response was of a different nature. Their reasons were well stated and included a brief review of the results without really criticizing the results. I certainly think it would have been better to have seen their results, but I understand their reasons.

The Mavituna Response by Ferruh Mavituna (Founder) [link]

Mavituna was not included in the report, so they took it upon themselves to perform the tests to see how they would have done. In "Point-and-shoot" they did about as well as the other scanners. When "Trained" by one of their internal guys, they out-performed Cenzic Hailstorm to take second place in that category.
They do make claims of having "False-Positive Free Scanning" which I question very much. I know that we have done a terrific job of avoiding False-Positives, but sometimes they can slip in. Seems like a claim that's unlikely backed up by reality.
All in all, I was impressed with how quickly they were able to get the tests done and posted, how well they performed and how open they have been during the process.

The Qualys Response

No response has been posted by Qualys that I have seen. I heard from Larry that they tended to be fairly happy considering the fact that their web scanning service is still very new.

Conclusion

Overall I think there have been a few sour grapes from the other vendors, and I can understand why, but I hope that Larry (and others willing to take the fire) will continue to devote the time and energy to perform these kinds of reviews. They are really important to the entire security industry and it is a loss to us all that the press hasnt been doing this service like they used to.

NT OBJECTives has worked hard for many years to create the best web application security scanner on the market, and we were not surprised that our tool was more accurate. We were surprised by how much better we performed and some of the gaps that existed in the other scanners, but we won't rest until we fix NTOSpider's logic so that we find the last 6 vulnerabilities!

- Dan Kuykendall

Last updated by Dan Kuykendall at September 2, 2011.