Today I had an interesting discussion with a colleague who will remain nameless. The subject was one of my favorites: attacks and countermeasures. One of the methods that we employ to evaluate new, existing or  emerging technologies is to run it through the attack tree gauntlet.  When the gauntlet is run we are left with a variety of attacks that range from the real to the completely theoretical.  Following this and other bits to be discussed later we try to discover countermeasures for each of the attacks. The countermeasures range from factoring out the exposures or adding compensators to reduce the as yet unquantified risk.

Here is where the fun started. I am of the opinion that there has to be some method for quantifying the effectiveness of the countermeasures. In other words, if I deploy one of the discovered countermeasures how much does it really help? We have a bad habit of listing countermeasures and taking the all-or-nothing approach. I happen to think we can use an informal qualitative method of communicating effectiveness. What does such a method look like? Well, a rough percentage works for me. Some complain that it is too “gut-check” oriented. They’d be right. However, I’d have to argue that the very method for discovering countermeasures themselves is a gut-check. “Say it ain’t so!”, you claim. Sorry, it is. If we can’t say a thing about how effective a countermeasure is then how can we claim it can do anything at all? The fact is that we use intuition, reflection or what some would call our experience to make a universal claim. Induction anyone? If we can use experience to discover the countermeasures themselves then why can’t we also use the same set of collective experiences to estimate the effectiveness (Induction again..) “Yes, but what numbers are you going to use?”, you may ask. You can use any range, scheme, etc that works for you. I use percentage in increments of 10. I like this because it is easy to reason with it. “80 times out of 100 AV is effective against non-0day malware delivered via E-mail.”, seems to work for me. Is it too detailed? I don’t think so. I think a broader range is better for the types of estimates we’re dealing with. If we use high, medium or low we run the risk of grossly over/under estimating. The bottom line is that we have to provide information to make informed decisions. We don’t get this right very often. By exposing our assumptions and methods we can move away from the “black magic” of the security practice. Being fearful that our methods are not “scientific” is a lame excuse for not trying in my opinion.