When the latest report from Larry Suto was set to come out and we had seen previews of the results, our first reaction was “Wow, we did great, but why did we miss those 9 vulns?!” followed by “Whoa – why did the other scanners miss so many vulnerabiities?” and then “Oh no, here we go again. Another round of getting unfairly blasted by the other vendors and their users“.
We certainly were not disappointed by the response from the other vendors and their users, but overall things seem to be different than they were in 2007 when Larry did his first report. In the latest report it is clear that Larry had learned at least two things from his first experience.
The first was that he needed better supporting data which he has certainly done this time by including the full breakdown of the vulns by site and vendor. The second was that he would need to provide for “Trained” scans, because most of the vendors made quite a protest that it was impossible to get proper results without it. My personal feeling on the matter is that “Point-and-shoot” is the most likely way that users will run scans and for that reason it is the responsibility of the scanner to do as much as possible on its own.
Because Larry did the “Trained” scanning this time around, this only leaves the other vendors with the ability to claim that he didn’t do a good enough job with the training. I think Jeremiah Grossman states it the best in his post “Scanner vendors should take into consideration that Larry Suto is certainly more sophisticated than the average user. So if he couldn’t figure out how to run your tool ‘properly’ take that as constructive feedback.”
A friend of mine also had a great line when he said that “from these results it appears that a monkey could get better results from NTOSpider, than an experienced security consultant can with the other scanners“. Continue reading