Can general security tests be useful?

Real-world security reports don’t always reflect your real world.

What makes a real-world security test useful? Does it need to provide a full assessment of a product or service? An assessment that is directly relevant for all potential customers? Or does it need to give just a taste of how effective a product can be?

The perfect security test

Tests can vary in how they are run and the level of information that they provide. Not all tests are equally reliable or even useful. But one thing they all have in common is that they aren’t perfect. Let’s look at how tests are limited, how you can interpret them and what the future holds.

Every large business uses security products, usually in combination with each other. Each is configured according to that organisation’s needs, almost certainly not using the default settings. This makes for a very specific setup.

A perfect real-world security test would exactly replicate this setup and expose it to every conceivable threat that the business faces, in exactly the way that an attacker would. The easiest way to achieve this is monitor everything in an organisation, invent a time machine and travel back to the previous year with the knowledge of what will happen. Given that we don’t have a TARDIS we need to produce something akin to a reasonable health check.

Shopping around

When you buy a product, whether it’s anti-virus or a carpet for your hall, you probably base your decision on your existing opinion, the opinion of others and in the context of your other possessions. In other words, the colour of your carpet will probably be related to the colour of your walls, furniture and your own taste (and that of your partner, if you have any sense!)

Your choice of anti-virus will be based on your previous experience, reviews you’ve read and the other security measures you’ve already taken. With both anti-virus and carpets, cost also plays a part in the decision-making process.

If your computer is not connected to the internet your endpoint security focus ought to be more on full disk encryption than anti-phishing. You might deploy intrusion detection and prevention on a network to protect servers, and use a basic anti-virus to give your users’ files a first-look. You might not use anti-virus at all if you are using a mobile phone.

When you shop around reviews play only one part of your decision making. But they are important, because the alternative is to simply believe the marketing claims. Such claims are often misleading or at least misunderstood.

One review to rule them all

There is never going to be one real-world security report that tells you everything you need to know. Testers use a finite number of threats to see if the product works. They will always miss something that might affect your organisation. It is possible to deploy enterprise security products in many different ways, with different options and configurations. A tester cannot assess a product in every possible use-case. It’s simply not practical.

Your business probably uses more than one security product. Let’s say you use Anti-Virus-X and AllowList-Y on all your endpoints. A single product test of Anti-Virus-X by a third-party tester, even using a different combination of configurations and products, doesn’t give you a perfect replication of your environment. So does that mean the public test report is useless?

If the report is transparent about how it was conducted, its strengths and weakness will be apparent and you can use it to inform your opinion and your buying decision. Use it as one piece of information, along with vendor claims and price. Use reports from multiple testers if you can. We don’t all test the same way and we don’t get the same results, but that’s actually a good thing. If SE Labs says a product is good and so do three other labs then it’s probably good. If we all disagree with each other then you might consider another option.

Complexity in enterprise product testing

In an ideal world you could download a report from a testing organisation and see exactly how ‘good’ or ‘bad’ a product is, and choose accordingly. As we’ve previously discussed, things are much more nuanced that good and bad, even when comparing quite basic products.

In a comparative test between two home user anti-virus products, such as those bought off the shelf in a store, you can make some reasonable assumptions. These assumptions could include:

1. The user is likely to use the default settings.
2. The user’s threat model is largely the same for most other home users, being untargeted attacks that potentially affect everyone indiscriminately.
3. The user is naive enough to click on malicious links but will also likely follow the advice of the security product.

Armed with these assumptions a tester can find some current malware, test the products with it and see what happens. Don’t forget to test for false positives too! This sounds like a simple proposition but for various technical reasons it’s actually very hard to do fairly.

When testing enterprise products things get even more complicated! Businesses will almost certainly need to tweak configurations. Their networks will be set up differently and their threat model will vary too. They will have security teams with different skills and there will be politics too.

We’ve seen large organisations in which regional security officers have passionate, personal preferences for certain products. (Top-level discussions about global security can sometimes sound a little like a pub conversation.)

Four ways to treat ‘real-world security reports’

Given all this, there are a few options an enterprise can take when choosing products:

1. Trust the tests. These testers must know what they are doing? How hard can it be to test a firewall or anti-virus? (Spoiler alert: very hard!)
2. Accept that public results are limited but look into the detail of the report and balance the methodology with the results. A very transparent report will allow you to see any relevant gaps in how the test was conducted. You will then get an idea about how useful and relevant the results are to your own organisation. For example, the test might avoid using phishing attacks but you already have a solution for that problem, in which case the report’s results are still useful. (We recommend this option.)
3. Discount all public tests as being useless and run internal tests only. (A brave and inefficient position to take…)
4. Employ a third-party lab to test the exact same products, using the same configuration and possibly even the same hardware. SE Labs runs tests like this for enterprises when they prepare for a major change in their security. (We also recommend this option!)

Ultimately a security test produces useful information that is otherwise very hard to obtain. How you use that information to make a robust buying decision will vary according to your own experience, how clearly the report describes the testing process and the number of similar reports available from other testers.