such as AV-Comparatives and AV-Test.
Antivirus tests from reputable agencies (where the result is not conditional upon payment) generally have two primary approaches when they create their testing methodologies: A bucket or some pincers. And yes, both approaches are legitimate and very useful.
AV-Comparatives and AV-Test take more of a bucket approach. Essentially, they direct a large quantity of known malware at the device and see what goes in the cracks of the antivirus app. They then run a large number of files and apps on the device to see if it triggers a false positive alarm.
SE Labs takes more of the pincer approach. Instead of using a huge sample set of malware or suspect files, they take a penetration test approach. Their researchers’ tools analyze how deep select malware can get into the tested device and also the extent and importance of a false positive alert. The deeper the malware penetrates, the more important a false positive file, and the bigger the impact on the test results.
The SE Labs protection rating assesses the antivirus’ ability to handle malicious files as well as indirectly malicious portions of an attack such as URLs or droppers. Here is their scoring breakdown:
SE Labs takes a similar approach with their Legitimate Accuracy test by quantifying the “nag” factor for false positives. Their Non-Optimal Classification/Action (NOCA) criteria gives a much more nuanced picture than the traditional yes/no false positive alert. Products that classify most applications correctly and which do not bother users score more points than those that ask questions and flag legitimate applications.
It is also a question of usability. As SE Labs describes it, there is a huge difference between blocking the latest version of Microsoft Word and condemning a rare Iranian dating toolbar for Internet Explorer 6. As a result, their formula is defined as: Interaction rating x Prevalence rating. The more hassle for users – and the more important the file – the bigger the penalty.
Avira scored well in both aspects of the test. The Avira Free Security Suite did not detect any of the clean files (708/708 points) which gives us an accuracy rating of 100%. In the Protection test, Avira achieved 309 points or a protection rating of 77%. Altogether, Avira achieved a rating of 92%, well above the industry average, earning us the AA award.
The tests confirm that Avira Antivirus is doing what we want it to do – protect users. As Avira Antivirus goes through various tests, we believe that having independent reviewers, with an approved and tested methodology, produces good results all round. They are good for security developers such as Avira – and for users who can be confident that their Antivirus has been tested under a variety of real-life conditions.