Anti-virus testing

How hard should a security test be?

Thank you for opening this report. We hope you’ll be able to use it to get a better idea about which anti-malware products you might want to buy (or get rid of!)

What do the awards mean?

The report starts off with a list of products, each of which win impressive-looking awards. But have you considered what those awards mean? How come there aren’t any massive losers in the list? How hard is this security test anyway?

Baseline Testing

There are lots of ways you can test products. You could prod a teddy bear and say, “well, that looks good enough,” or you could take it to pieces and analyse every component forensically for build and functional quality. “This toy looks safe, its parts are large, soft and non-toxic, and we can’t burn it easily. Plus, it’s got big, cute eyes.”

This could be a baseline for cuddly toys: SAFE, with cuteness as an extra bonus.

For anti-malware products we have to consider a few different things, including the following:

• Is it really an anti-malware product? Is it at least basically functional?
• Can it determine a good quantity of common malware, without blocking lots of useful software?
• Can it stop the malware, as well as simply detecting it?
  
  

How hard do you want your security testing to be? We could take a product, scan a real virus and record that it detected a threat. Is that good enough? It’s good enough to answer the first question above, and to reassure you that you’ve installed the software correctly. But you can’t tell if it’s better than other anti-malware products, because most will all react the same.

You also can’t tell if the product has extra functionality capable of detecting and stopping other types of threats, of which there are many. Scanning files is a basic way to test anti-malware. In most cases, it’s too basic.

Attacks For Everyone

Let’s turn up the dial and throw a wider range of attacks at the products. We can use malware that bad guys use to attack everyone, every day. We call these threats ‘commodity attacks’ because they are all over the place, indiscriminately damaging computers all over the world. Our hope is that all main anti-malware products will detect each of these threats. To us, that’s the baseline. If a product can’t do that, there’s something very wrong going on.

We go further than that, though. By using forensics, we can tell not only how many threats a product can recognise, but how well it can protect against them too. It might say, “I see a virus and I’ve blocked it!” but we don’t trust that claim. We check that it’s really done what it said it did.

Attacks Just For You

We could go harder, and use targeted attacks designed to evade anti-malware. So we do. That’s the ‘targeted attacks’ part of the test. You can see how the products handled them in Appendix D, buried at the end of this report. That’s possibly the most interesting part!

You don’t generally see products that perform less well in our tests because we think you probably don’t even want to consider them. Don’t waste your time. But sometimes they won’t let us test them. If your favourite anti-malware vendor isn’t in our report, please tell them to play!

Any A, AA or AAA awards that products achieve in our reports show that they go well beyond basic functionality. It shows that they can handle both common and customised threats, without blocking the software you need to run on your computer.