In an article entitled “How we test antivirus? The making of CheckLab.pl, a website dedicated to security tests” we have explained in detail what is a new project called CheckLab. Recently founded organization with a website at www.CheckLab.pl is completely focused on security tests. We have decided that CheckLab will be functioning as part of well known company AVLab.pl that has been operating since 2012. Tests carried out by AVLab.pl are recognized and valued by developers themselves and commented on international boards so that is why creating the new project has got business, branding, and technical foundations. There is so much additional information related to tests so we have decided to prepare a supporting website.
All instructions regarding the methodology and tools used in tests were published on the website https://checklab.pl/en.
Now to the point, let us explain what is the „Advanced In The Wild Malware Test”.
Introduction to the „Advanced In The Wild Malware Test”
The name of the „Advanced In The Wild Malware Test” perfectly reflects its character. The source of malicious software are honeypots located on all continents of the world. We collect malware, among other, for the Windows system. Samples captured in attacks are checked on the basis of over 100 patterns before they are qualified for testing. These patterns allow us to determine whether a potentially dangerous file is actually a threat to the Windows 10 operating system.
In July 2019, in the first edition of security tests we tested eight solutions to protect computers. In the results we included seven products because we enable developers to informally join our tests. If a certain company thinks that its solution hasn’t been fully developed or is worried about a bad result, it has the opportunity to join the tests on a trial basis. At that time, the test results for a given product are not made publicly available. In addition, we provide to such company with all necessary details that will help improve the effectiveness of their security solution.
The tests lasted the whole July 2019 and were based on testing security of the following software:
- AVIRA Antivirus Pro
- EMSISOFT Business Security
- F-SECURE Total
- G DATA Endpoint Protection Business
- G DATA Total Security
- KASPERSKY Endpoint Security Cloud Protection
- SECUREAPLUS Premium
- The eighth product was not included in the official results.
We have used almost thousand malware samples for the test. In the contrast to the well-known testing institutions, such as AV-Comparatives or AV-Test, our tests are much more transparent because we share the full list of malware samples. It is available in the form of tables on the new project website at www.CheckLab.pl/en. During testing, all solutions have access to the Internet. We use real working environments in a graphics mode that is why the results of individual samples may differ from those presented by the VirusTotal service. We take note of that because we guess that curious users will compare our tests with the scanning results of VirusTotal. Differences between real products installed on Windows 10 and scanning engines on VirusTotal are significant. We explained everything in the article “How we test antivirus? The making of CheckLab, a website dedicated to security tests”.
The products and Windows 10 settings: daily test cycle
Tests are carried out in Windows 10 Pro x64. The user account control (UAC) is disabled because the purpose of the tests is to check the protection effectiveness of a product against malware and not a reaction of the testing system to Windows messages.
Additionally, the Windows 10 system contains software installed: an office suite, document browser, email client, and other tools and files, pretending to be a normal working environment.
Automatic updates of the Windows 10 system are disabled in a given month of tests. Due to the complicated process and the possibility of a malfunction, Windows 10 is updated every few weeks under close supervision.
Security products are updated one time within a day. Before tests are run, virus databases and protection product files are updated. This means that the latest versions of protection products are tested every day.
Levels of blocking malware samples
We are probably pioneers in this respect — we show our readers more detailed diagnostic data than any other testing institution so far (including the largest like AV-Comparatives and AV-Test). Levels of blocking each malware sample by tested protection solution are published on the website of the CheckLab organization.
We have divided the blocking viruses into few levels:
- Level 1 (L1): The browser level, i.e. a virus has been stopped before it has been written on a hard drive.
- Level 2 (L2): The system level, i.e. a virus has been downloaded, but it has not been allowed to run.
- Level 3 (L3): The analysis level, i.e. a virus has been run and blocked by a tested product.
- Failure (F): The failure, i.e. a virus has not been blocked and it has infected a system.
The result of blocking each sample can be traced on the website https://checklab.pl/en/last-results in the following table:
A detailed methodology can be found in the section “Methodology — Methods adopted in tests”.
The results of July 2019
As the first organization in the word we show detailed information from tests for our readers. We share checksums of malicious software by dividing them into protection technologies which have contributed to stop a threat. We hope that this type of innovative approach to testing will be very interesting experience for recipients of our work.
We have granted the best certificate BEST+++ to:
- Avira Antivirus Pro
- G Data Endpoint Protection Business
- Emsisoft Business Security
- Kaspersky Endpoint Security Cloud Plus
- SecureAPlus Premium
We haven't granted the certificate BEST++.
We have granted the certificate GOOD+ to:
- G DATA Total Security
The F-Secure TOTAL software has not reached the required threshold of 90% which is why it has not been granted any award.
Download certificates in high resolution: http://checklab.pl/en/about-us
We encourage our readers to get familiar with the website www.checklab.pl/en. We particularly recommend links in the menu:
All these materials explain the procedural approaches in testing. In one of the articles we have also included two videos in order to further illustrate how we automate the tests.
It is our first post regarding the first edition of the “Advanced In The Wild Malware Test” which were carried out in July 2019. We can already tell that we are still working on improving our algorithms that are fine-tuned on a regular basis. At the moment we look into upgrading tools for monitoring changes in the Windows system and tracking actions performed by tested products. This way, our cooperation with developers allows us to provide more detailed data in order to verify each sample and the reaction of tested product. We also hope that more and more companies interested in cooperation will participate in our tests. Our motto is transparency to which we encourage large institutions involved in testing.
Information about CheckLab
The CheckLab organization was founded in July 2019 by the AVLab.pl company operating since 2012 in the industry of informatics security. The primary objective of the CheckLab organization is to test security usefulness, and issuing certificates confirming the protection effectiveness against malware, and also provide results to public information while ensuring the maximum transparency of the tests. In the studies, the CheckLab employees use malicious software, tools, and techniques of bypassing security that are used in real cyberattacks. Even though the project called CheckLab has existed only for a few weeks, the organization already cooperates with the largest companies in the security industry.