For all of the “growing up” that the security industry has done in recent years it is amazing that proper distinctions are not being made when it comes to testing. No, not that testing, security testing. See, it demonstrates the point. When I say security testing all sorts of concepts are cognitively connected in that moment. Unfortunately, these connections are different for each of us. When some hear security testing they may think of using a new and improved fantastic commercial tool that produces beautiful output. Others think of an uber hacker sitting in a dimly lit room reading the network-byte-ordered data from some super secret application protocol. Yet others have never heard of the term security testing. Instead, they think you’re talking about pen-testing which, for the purposes of this discussion, is a fine synonym. The challenge is not with the definitions per se, but with the perceived effectiveness.

To be honest, it isn’t really the perceived effectiveness. Well it is, but let me elaborate. The challenge lies in the way in which we swap the effectiveness. Somehow we’ve managed to equate the ability of the uber hacker to find implementation bugs with that of a tool. We’ve given the tool the title of uber hacker. A title that, despite all of its merits, it cannot bear because it cannot deliver the goods. This does not mean that tools are not an expedient way to identify and respond to low-hanging fruits. They are great for that. But, sometimes it is thought that either low-hanging fruit represents the greatest risks or that this low-hanging fruit is really all the fruit there is on the tree. I don’t think this is the case and this is the problem. We cannot presume that what a tool can identify is all that there is and that what it can identify is all the low-hanging fruit there is. In the context of the tool it may be all there is, but in the context of an uber hacker or even a not-so-uber hacker it may not.

An example may help clarify what I am talking about. In a previous life we were testing an online application that was responsible for delivering sensitive documents to users with accounts. This was a manual, exploratory type of test because there were no real design documents that described the behavior. We immediately created an account and after some time visited the ‘I forgot my password screen’. The email we received looked odd. To make an exceedingly long story short, we discovered that the application was generating a pseudo-hash as the means of authenticating the user. This hash, if you can call it that, was really a simplistic Vignere cipher using the username as the key material. Of course you can imagine what happens next.

Running the variety of commercial tools out there could not find the flaw mentioned above, but we knew this. This was why we performed this ‘other’ kind of pen/security testing. Was this flaw easy to find? It was and tools were not equipped to find it. The fact is if we continue to label automated tools as uber-hackers-in-a-box, then we may be in for quite a surprise. This is why distinctions, especially with regards to pen/security testing are important. We have to be aware of how we are testing not simply that we are testing. And even with all that we have to remember that in the end test results without any identified vulnerabilities only tell us that we were unable to find any and not that they do not exist.