How We Test Archives - Page 7 of 8

Are you buying solid protection or snake oil?

Written By

SE Labs Team

Published on

May 9, 2018

Sometimes testers need to be tested too. We’re always up for a challenge!

How do you know which security products to buy? Are you buying solid protection or snake oil? Many rely on independent tests to help in the decision-making process. But how do you know if a test is any good or not?

Latest reports now online.

The Anti-Malware Testing Standards Organization (AMTSO) has been working to create a Standard that will give you, the customer, some assurance that the test was conducted fairly.

Benefits of following a Standard

Earlier this year AMTSO has been trying out its Standard, which it has been working on for many months. SE Labs is proud to be involved in this initiative and the testing for this report has been assessed for compliance with the Standard.

If that sounds a bit dry, what it means is that there are experimental rules about how a tester should behave and we have put ourselves up for judgment by AMTSO.

Did participating in this process change the way we worked? Yes, but not in the technical ways that we test. Instead we turned the testing world’s business model on its head.

Are you buying solid protection or snake oil?

Many testers charge vendors money to be tested. Some will test regardless, but charge money if the vendors want to see their results before publication (and have the opportunity to make requests for corrections).

We think that the dispute process should be free for all. SE Labs has not charged any vendor for its participation in this test and we provided a free dispute process to any vendor that requested it. In this way every vendor is treated as equally as possible, for the fairest possible test.

UPDATE (10th May 2018): We are extremely proud to announce that our 2018 Q1 reports have been judged compliant (PDF) with the AMTSO Draft Standard v6.1 – 2018-05-10.

If you spot a detail in this report that you don’t understand, or would like to discuss, please contact us via our Twitter or Facebook accounts.

SE Labs uses current threat intelligence to make our tests as realistic as possible. To learn more about how we test, how we define ‘threat intelligence’ and how we use it to improve our tests please visit our website and follow us on Twitter.

These reports, for enterprise, small business and home users are now available for free from our website. Please download them and follow us on Twitter and/or Facebook to receive updates and future reports.

All posts

Tough test for email security services

Written By

SE Labs Team

Published on

May 3, 2018

Our latest email cloud security test really challenged the services under evaluation.

Latest report now online.

Last summer we launched our first email cloud security test and, while it was very well received by our readers and the security industry as a whole, we felt that there was still work to do on the methodology.

This report shows the results of six months of further development, and a much clearer variation in the capabilities of the services under test.

The most significant change to the way we conducted this test lies in the selection of threats we used to challenge the security services: we increased the number and broadened the sophistication.

Whereas we might have used one fake FBI blackmail email previously, in this test we sent 10, each created using a different level of sophistication. Maybe a service will detect the easier versions but allow more convincing examples through to the inbox?

We wanted to test the breaking point.

We also used a much larger number of targeted attacks. There was one group of public ‘commodity’ attacks, such as anyone on the internet might receive at random, but also three categories of crafted, targeted attacks including phishing, social engineering (e.g. fraud) and targeted malware (e.g. malicious PDFs).

Each individual attack was recreated 10 times in subtly different but important ways.

Attackers have a range of capabilities, from poor to extremely advanced. We used our “zero to Neo” approach to include basic, medium, advanced and very advanced threats to see what would be detected, stopped or allowed through.

The result was an incredibly tough test.

We believe that a security product that misses a threat should face significant penalties, while blocking legitimate activity is even more serious.

If you’re paying for protection threats should be stopped and your computing experience shouldn’t be hindered. As such, services that allowed threats through, and blocked legitimate messages, faced severe reductions to their accuracy ratings and, subsequently, their chances of winning an award.

Intelligence-Led Testing

We pay close attention to how criminals attempt to attack victims over email. The video below shows a typically convincing attack that starts with a text message and ends stealing enough information to clean out a bank account.

All posts

Predictably Evil

Written By

SE Labs Team

Published on

April 11, 2018

A common criticism of computer security products is that they can only protect against known threats. When new attacks are detected and analysed security companies produce updates based on this new knowledge. It’s a reactive approach that can provide attackers with a significant window of opportunity. Some use special technology to predict the future, but does AI really work?

AV is dead (again)

It’s why anti-virus has been declared dead on more than one occasion.

Latest report now online.

Security companies have, for some years, developed advanced detection systems, often labelled as using ‘AI’, ‘machine learning’ or some other technical-sounding term. The basic idea is that past threats are analysed in deep ways to identify what future threats might look like. Ideally the result will be a product that can detect potentially bad files or behaviour before the attack is successful.

(We wrote a basic primer to understanding machine learning a couple of years ago.)

Does AI really work?

So does this AI stuff really work? Is it possible to predict new types of evil software? Certainly investors in tech companies believe so, piling hundreds of millions of funding dollars into new start-ups in the cyber defence field.

We prefer lab work to Silicon Valley speculation, though, and built a test designed to challenge the often magical claims made by ‘next-gen’ anti-malware companies.

With support from Cylance, we took four of its AI models and exposed them to threats that were seen in well-publicised attacks (e.g. WannaCry; Petya) months and even years later than the training that created the models.

It’s the equivalent of sending an old product forward in time and seeing how well it works with future threats. To find out how the Cylance AI models fared, and to discover more about how we tested, please download our report for free from our website.

All posts

Hacked! Will your anti-malware protect you from targeted attacks?

Written By

SE Labs Team

Published on

February 5, 2018

Is anti-malware protection always strong? The news isn’t good. Discover your best options in our latest reports.

Latest reports now online.

Criminals routinely create ingenious scams and indiscriminate attacks designed to compromise the unlucky and, occasionally, foolish. But sometimes they focus on a specific target rather than casting a net wide in the hope of landing something interesting.

Targeted attacks can range from basic, like an email simply asking you to send some money to an account, through to extremely devious and technical. If you received an email from your accountant with an attached PDF or Excel spreadsheet would you open it?

Most would and all that then stands between them and a successful hack (because the email was a trick and contained a dodgy document that gives remote control to the attacker) is the security software running on their PC. How good is your anti-malware protection?

Anti-malware protection from different threats

In this test we’ve included indiscriminate, public attacks that come at victims from the web and via email, but we’ve also included some devious targeted attacks to see how well-protected potential victims would be.

We’ve not created any new types of threat and we’ve not discovered and used ‘zero day’ attacks. Instead we took tools that are freely distributed online and are well-known to penetration testers and criminals alike. We used these to generate threats that are realistic representations of what someone could quite easily put together to attack you or your business.

The results are extremely worrying. While a few products were excellent at detecting and protecting against these threats many more were less useful. We will continue this work and report any progress that these companies make in improving their products.

See if your anti-malware protection is up to scratch!

Our latest reports, for enterprise, small business and home users are now available for free from our website. Please download them and follow us on Twitter and/or Facebook to receive updates and future reports. Our blog is always here with extra help.

All posts

Network appliances vs. targeted attacks

Written By

SE Labs Team

Published on

December 12, 2017

There have been so many publicised data breaches in 2017 that we didn’t even have enough space in our latest report to provide a basic summary. In many cases a business network was breached. Business networks comprise endpoints (usually Windows PCs), servers, Point of Sale computers and a range of other devices.

Latest reports now online.

In this report (PDF) we explore the effectiveness of network appliances designed to detect and block attacks against endpoint systems. Who wins in the battle between network appliances vs. targeted attacks?

One approach to compromising a business is to hack an endpoint (PC) and then to use it as a platform from which to launch further attacks into the network. For example, rather than going straight for a company’s main servers why not trick a user into infecting his/ her computer with malware? We can then scan and infect the entire network, stealing information, causing damage and generally behaving in ways contrary to the business’ best interests.

There is some really good endpoint software available, as we see in our regular Endpoint Protection tests, but nothing is perfect and any extra layers of security are welcome. If one layer fails, others exist to mitigate the threat. In this report we explore the effectiveness of network appliances designed to detect and protect against attacks against endpoint systems.

Network appliances vs. targeted attacks

The systems we have tested here are popular appliances designed to sit between your endpoints and the internet router. They are designed to detect, and often protect against, threats coming in from the internet or passing through the local network. Their role is to stop threats before they reach the endpoints. If they fail to stop a threat, they might learn that an attack has happened and generate an alert, while subsequently blocking future, similar attacks.

There are no guarantees that technology will always protect you from attackers, but our results show that adding layers of security is an effective way to improve your prospects when facing general and more targeted attacks.

Featured podcast:

All posts

100% Certifiable – security product certifications

Written By

SE Labs Team

Published on

October 17, 2017

Whether you’re in the market for a car, hamburger or computer security product, certifications are useful. They don’t tell you how smooth the car drives, how tasty the sandwich is or how completely accurate the anti-virus software will be, but certifications indicate a general level of competence.

Latest reports now online.

In the UK new cars must be certified by the Vehicle Certification Agency (VCA), restaurants are checked for hygiene by the Food Standards Agency (FSA) and various independent testing organisations, including SE Labs, test IT security products for basic functionality.

A certification emphatically does not indicate the overall quality of a product, though. The FSA specifically states that, “The food hygiene rating is not a guide to food quality.” In other words, the food won’t make you ill, but you might not like it! Similarly, the VCA cares more about cars being made according to specification rather than how nice they look.

Security product certifications

SE Labs has a range of available testing services. We consider certification to be the most basic type of testing. If a product claims to be able to detect malware then we can test that. But we don’t claim it can detect all types. For a higher level of understanding about a product’s capabilities so-called ‘real-world’ testing is necessary.

The report you are reading now is based on our more advanced testing. This exposes real products to live threats in a realistic environment, running on real computers on an internet-connected network.

But how can you be sure that we’re really doing that? Are we just making up the figures or giving some products an unfair advantage? After all, some companies contribute financially to supporting the tests, while others do not.

To go some way to addressing this concern, as well as to improve generally and continue to evolve the business, SE Labs has achieved ISO 9001:2015 certification for “The Provision of IT Security Product Testing”. We think it’s fair to test the testers and we’re very proud to have passed!

If you spot a detail in this report that you don’t understand, or would like to discuss, please contact us via Twitter.

All posts

Email hosted protection tested

Written By

SE Labs Team

Published on

September 19, 2017

Our first cloud-based email protection report is now available.

Email provides a route right into the heart of our computers, phones and other devices. As such, it is frequently abused to perform a variety of attacks against potential victims of cybercrime.

Latest report now online.

The sophistication of attacks vary but many rely on our almost unbreakable instinct to open, read and interact with messages sent to work and personal email accounts. Businesses rely on email security services to filter out large numbers of such attacks.

Types of attack

The range of attack types in the real world is wide, but in general we consider there to be two main categories: targeted attacks, in which the attacker attempts to target a specific individual; and public attacks, which spread wide and far in an attempt to compromise as many people as possible.

Targeted attackers and general criminals use many of the same techniques. The least technically sophisticated include requests for a money transfer or banking login credentials. More credible attempts include professionally-formatted emails and links to fake websites designed to trick users into entering their valuable details.

Attackers with more resources may use malware to achieve their goals, either in the form of attached files or by linking to websites that exploit visiting computers.

How does email protection compare?

SE Labs monitors email threats in real-time, analysing large numbers of messages and extracting samples that represent large groups of those threats. Human testers then manually verify that any malware included works properly. They then re-send these threats to our own accounts through the tested services.

We also generate targeted attacks using the same tools and techniques used by advanced attackers. In gathering threats this way we achieve a realistic and relevant coverage of existing threats in a small set of test samples.

All posts

Testing anti-malware’s protection layers

Written By

SE Labs Team

Published on

April 11, 2017

Endpoint security is an important component of computer security, whether you are a home user, a small business or running a massive company. But it’s just one layer. Our first set of anti-malware test results for 2017 are now available.

Latest reports now online

Using multiple layers of security, including a firewall, anti-exploit technologies built into the operating system and virtual private networks (VPNs) when using third-party WiFi is very important too.

What many people don’t realise is that anti-malware software often actually contains its own different layers of protection. Threats can come at you from many different angles, which is why security vendors try to block and stop them using a whole chain of approaches.

A fun video we created to show how anti-malware tries to stop threats in different ways

How layered protection works

For example, let’s consider a malicious website that will infect victims automatically when they visit the site. Such ‘drive-by’ threats are common and make up about one third of this test’s set of attacks. You visit the site with your web browser and it exploits some vulnerable software on your computer, before installing malware – possibly ransomware, a type of malware that also features prominently in this test.

Here’s how the layers of endpoint security can work. The URL (web link) filter might block you from visiting the dangerous website. If that works you are safe and nothing else need be done.

But let’s say this layer of security crumbles, and the system is exposed to the exploit.

Maybe the product’s anti-exploit technology prevents the exploit from running or, at least, running fully? If so, great. If not, the threat will likely download the ransomware and try to run it.

At this stage file signatures may come into play. Additionally, the malware’s behaviour can be analysed. Maybe it is tested in a virtual sandbox first. Different vendors use different approaches.

Ultimately the threat has to move down through a series of layers of protection in all but the most basic of ‘anti-virus’ products.

Testing all layers

The way we test endpoint security is realistic and allows all layers of its protection to be tested.

See all blog posts relating to test results.

All posts

How well does your anti-virus really protect you?

Written By

SE Labs Team

Published on

January 10, 2017

Not equally well, is the short answer. Find out which anti-virus products are consistently the best. And how we help make anti-virus improvements.

Latest reports now online

Welcome to the final set of endpoint security tests for 2016. We’ve spent the entire year scanning the internet for prevalent threats that affect real people and exposing popular security products to those same threats in real-time.

All posts

Does your anti-malware stop hacking attacks?

Written By

SE Labs Team

Published on

November 1, 2016

An attack rarely ends when the malware runs. That’s just the beginning…

Latest reports now online.

Testing security software is a challenging task and it’s tempting to take clever shortcuts. However, while doing so might save the tester time and other resources, it doesn’t always produce useful results. And if the results aren’t accurate then the test becomes less valuable to you when you’re choosing which product to use.

Can anti-malware stop hacking?

We are big supporters of the idea of full product testing. This means installing the security product the way it was intended to be used, on systems commonly used in the real world and ensuring that every component of that product has a chance to defend the system.

In practice this means that we installed the anti-malware products tested in this report on regular PCs that are connected to a simple network that has unfiltered internet access. We visit malicious websites directly, where possible, and use a special replay system when the bad guys start to interfere with our activities.

Since the beginning of this year we started including targeted attacks in our testing. These types of attacks try to compromise the target using infected documents and browser exploits. Once an exploit has succeeded we then continue ‘hacking’ the target. This step is crucial because in many cases it is these post-exploitation hacking activities that can trigger an alert.

Full product testing doesn’t just mean turning on (or leaving enabled) all of a product’s features. It also means running a full attack as realistically as possible. Testers should not make assumptions about how a product works. You need to act like a real bad guy to understand how these products protect the system. Can anti-malware stop hacking? Test like a hacker and find out. And read our results!

These reports, for enterprises, small businesses and home users are now available for free from our website.

Category: How We Test

Are you buying solid protection or snake oil?

Sometimes testers need to be tested too. We’re always up for a challenge!

Benefits of following a Standard

Are you buying solid protection or snake oil?

Predictably Evil

AV is dead (again)

Does AI really work?

Hacked! Will your anti-malware protect you from targeted attacks?

Anti-malware protection from different threats

Network appliances vs. targeted attacks

Network appliances vs. targeted attacks

Featured podcast:

100% Certifiable – security product certifications

Security product certifications

Testing anti-malware’s protection layers

How layered protection works

Testing all layers

How well does your anti-virus really protect you?

Does your anti-malware stop hacking attacks?

Can anti-malware stop hacking?

Contact us

Get in touch

Feel free to reach out to us with any questions or inquiries