World’s first in-depth, public test of security services vs. targeted attacks.
This email security test report is the product of two years of advanced threat research. We have worked with the security companies themselves and with their customers. We have monitored what the bad guys have been doing and identified and replicated real-world email threats that affect everyone generally, and also specific types of businesses.
There is no report like this anywhere in the public domain. We are extremely proud to present the results here.
This new email protection test shows a wide variation in the abilities of the services that we have assessed.
You might see the figures as being disappointing. Surely Microsoft Office 365 can’t be that bad? An eight per cent accuracy rating seems incredible.
Literally not credible. If it misses most threats then organisations relying on it for email security would be hacked to death (not literally).
But our results are subtler than just reflecting detection rates and it’s worth understanding exactly what we’re testing here to get the most value from the data. We’re not testing these services with live streams of real emails, in which massive percentages of messages are legitimate or basic spam. Depending on who you talk to, around 50 per cent of all email is spam. We don’t test anti-spam at all, in fact, but just the small percentage of email that comprises targeted attacks.
In other words, these results show what can happen when attackers apply themselves to specific targets. They do not reflect a “day in the life” of an average user’s email inbox.
We have also included some ‘commodity’ email threats, though – the kind of generic phishing and social engineering attacks that affect everyone. All services ought to stop every one of these. Similarly, we included some clean emails to ensure that the services were not too aggressively configured. All services ought to allow all these through to the inbox.
So when you see results that appear to be surprising, remember that we’re testing some very specific types of attacks that happen in real life, but not in vast numbers comparable to spam or more general threats.
The way that services handle threats are varied and effective to greater or lesser degrees. To best reflect how useful their responses are, we have a rating system that accounts for their different approaches. Essentially, services that keep threats as far as possible from users will win more points than those who let the message appear in or near the inbox. Conversely, those that allow the most legitimate messages through to the inbox rate higher than those which block them without the possibility of recovery from a junk folder or quarantine.
If you spot a detail in this report that you don’t understand, or would like to discuss, please contact us via our Twitter or Facebook accounts.
SE Labs uses current threat intelligence to make our tests as realistic as possible. To learn more about how we test, how we define ‘threat intelligence’ and how we use it to improve our tests please visit our website and follow us on Twitter.
Our latest reports, for enterprise, small business and home users are now available for free from our website. Please download them and follow us onTwitter and/orFacebookto receive updates and future reports.
Last summer we launched our first email cloud security test and, while it was very well received by our readers and the security industry as a whole, we felt that there was still work to do on the methodology.
This report shows the results of six months of further development, and a much clearer variation in the capabilities of the services under test.
The most significant change to the way we conducted this test lies in the selection of threats we used to challenge the security services: we increased the number and broadened the sophistication.
Whereas we might have used one fake FBI blackmail email previously, in this test we sent 10, each created using a different level of sophistication. Maybe a service will detect the easier versions but allow more convincing examples through to the inbox?
We wanted to test the breaking point.
We also used a much larger number of targeted attacks. There was one group of public ‘commodity’ attacks, such as anyone on the internet might receive at random, but also three categories of crafted, targeted attacks including phishing, social engineering (e.g. fraud) and targeted malware (e.g. malicious PDFs).
Each individual attack was recreated 10 times in subtly different but important ways.
Attackers have a range of capabilities, from poor to extremely advanced. We used our “zero to Neo” approach to include basic, medium, advanced and very advanced threats to see what would be detected, stopped or allowed through.
The result was an incredibly tough test.
We believe that a security product that misses a threat should face significant penalties, while blocking legitimate activity is even more serious.
If you’re paying for protection threats should be stopped and your computing experience shouldn’t be hindered. As such, services that allowed threats through, and blocked legitimate messages, faced severe reductions to their accuracy ratings and, subsequently, their chances of winning an award.
Intelligence-Led Testing
We pay close attention to how criminals attempt to attack victims over email. The video below shows a typically convincing attack that starts with a text message and ends stealing enough information to clean out a bank account.
SE Labs uses current threat intelligence to make our tests as realistic as possible. To learn more about how we test, how we define ‘threat intelligence’ and how we use it to improve our tests please visit our website and follow us on Twitter.
Our first cloud-based email protection report is now available.
Email provides a route right into the heart of our computers, phones and other devices. As such, it is frequently abused to perform a variety of attacks against potential victims of cybercrime.
The sophistication of attacks vary but many rely on our almost unbreakable instinct to open, read and interact with messages sent to work and personal email accounts. Businesses rely on email security services to filter out large numbers of such attacks.
The range of attack types in the real world is wide, but in general we consider there to be two main categories: targeted attacks, in which the attacker attempts to target a specific individual; and public attacks, which spread wide and far in an attempt to compromise as many people as possible. Many of the same techniques are used in public and targeted attacks. The least technically sophisticated include requests for a money transfer or banking login credentials. More credible attempts include professionally-formatted emails and links to fake websites designed to trick users into entering their valuable details.
Attackers with more resources may use malware to achieve their goals, either in the form of attached files or by linking to websites that exploit visiting computers. SE Labs monitors email threats in real-time, analysing large numbers of messages and extracting samples that represent large groups of those threats. Human testers then manually verify that any malware included works properly before re-sending these threats to our own accounts through the tested services.
We also generate targeted attacks using the same tools and techniques used by advanced attackers. In gathering threats this way we achieve a realistic and relevant coverage of existing threats in a small set of test samples.
Our latest reports, for enterprise, small business and home users are now available for free from our website. Please download them and follow us on Twitter and/or Facebook to receive updates and future reports.
SE Labs Ltd is a private, independently-owned and run testing company that assesses security products and services. The main laboratory is located in Wimbledon, South London. It has excellent local and international travel connections. The lab is open for prearranged client visits.
