Public and private testing
How security vendors work with SE Labs.
SE Labs works with security companies to help develop and validate their products. When a security company works with SE Labs it gains two main benefits. If the product performs well it gains a much sought-after award. If it encountered problems the testing team will provide valuable information to help fix the issues.
Most of the testing SE Labs conducts is private, but we publish reports too. In this article we explain the different types of testing we perform, and how some reports end up online while others stay confidential.
Five simple rules
Testing and publishing results can get complicated so we have some simple rules to maintain the integrity of the reports, while ensuring our readers can access as many results as possible.
- No sponsored comparative reports
- Review process after each test
- Publication is decided before the test starts
- Standalone reports can be public or private
- No public reports for development projects
No sponsored comparative reports
A report comparing different vendors can be very useful, but if it was commissioned and paid for by just one of the vendors in the test it looks unfair. Even though the tester might have performed the most technically capable and ethically sound test, readers will always suspect some level of bias.
They might ask themselves, “did the tester consciously give the sponsor better results?” They might also wonder how many times the tester ran similar, unpublished tests before the ‘right’ result was obtained and then published. Another possibility is that the tester inadvertently chose a testing method that favoured the sponsor.
A sponsored comparative test, where the paying vendor wins over its competitors, always lacks credibility.
It’s why we don’t publish sponsored comparatives.
Review process after each test
No test is perfect and both people and computers make mistakes. The review process is where security companies enjoy arguably the most value from SE Labs. During this period, we share detailed data to validate the results and to help improve the product. If genuine errors are identified then we correct them at this stage.
Our error levels are the lowest in the industry.
Publication is decided before the test starts
This rule applies to tests that compare competing products. We fund comparative testing, which produces work like our regular Endpoint Security Protection reports, through providing consultancy services to the security vendors. The vendors that participate in our comparative ‘group’ tests must decide on publication before the test starts. There is a simple reason for this. Only one vendor can come top of a score chart.
We require a commitment to the test in advance so that readers of the report can access a wide set of results.
Standalone reports can be public or private
Vendors in standalone reports can decide to publish (or not) after they learn of their results. All of the tests on our website are funded in some way. Standalone reports, featuring one product, are paid for by the vendor that owns that product. If the product performs poorly the vendor uses the data we collect to improve the technology. If it performs well it can celebrate by publishing the report.
Many standalone reports are comparable. For example, the Enterprise Advanced Security tests include Endpoint Detection and Response (EDR) reports covering different products.
By choosing carefully you could create your own customised comparative report.
No public reports for development projects
A development project involves the SE Labs team working directly with your developers to improve a product. Often this involves us running attacks while the developers tweak the product to perform optimally. Given that we are helping the product achieve good results, we don’t provide public reports for this type of work.
It would be misleading and biased.
Sign up to our security vendor newsletter.
Your route through SE Labs testing
FAQs
Can I make a copy of a report?
SE Labs has a licensing programme that permits organisations to redistribute reports. Licenses for one year are available, as is a perpetual license. Please email or call your main point of contact at SE Labs if you want to license a report.
Why don’t you review my product on your site?
We try to maintain coverage of the most important security vendors but inevitably there will be gaps. If you are a security vendor simply contact us and we’ll try to help.
Why doesn’t my product perform the best in your tests?
There will be reasons, and we have discussed in the review stage of the testing process. Usually the product has not completely succeeded in detecting or preventing threats, or it may have misclassified legitimate objects such as good applications. Or a combination of these factors.
Will you work with any security vendor?
Within existing UK legislation, we will work with any security vendor. If you don’t see a major vendor represented in our public reporting we may be testing it privately. Or it may avoid testing completely.
Can I find out how my product compares to others privately?
SE Labs constantly tests a range of security products and services. We have a unique insight into how these products work and how they compare, in terms of features and effectiveness. As such, we produce private reports that compare different products.
We sometimes share part of this information with the security companies that engage with our private testing programmes. In many cases we anonymise some of the results, so each vendor knows how it stands compared to the wider industry. This level of competitive intelligence gives a good overview without getting into too much specific detail.
Find out more
Our latest reports, for enterprise, small business and home users are now available for free. Please download them and follow us on Twitter and/or LinkedIn to receive news, comment, updates and future reports.
Sign up to our monthly business and personal security newsletters.
See all blog posts relating to test results.
