Fighting spam on hosting. Setup EFA Project Free Spam / antivirus filter

In this article, as promised , we want to share our experience in dealing with spam. It is known that any cause has a consequence. This phrase expresses one of the philosophical forms of communication phenomena.

Our reason was repeated complaints about spam emanating from our hosting and VPS clients. It is not always possible to say with certainty whether it was the deliberate actions of customers or they themselves did not suspect that they were the victim of spam bots. Whatever it was, the problem had to be solved.

Spam do not like. Spam leaves a “black spot” on the provider’s face when his IP addresses are added to blacklists, which all clients suffer from. Removing IP from blacklist is a special conversation. But this is one side of the coin. If it is possible to return the reputation to an IP address, then it is much more difficult to return the company's reputation and trust.

We decided to find a solution and implement the Unihost complex to protect and prevent unwanted mailings. After brainstorming and discussions, they began to check and compare what the SPAM / AV community can offer.

There are many options on the market. However, the majority of quality solutions are paid with a tariff of 1 license for 1 server or even for the number of outgoing / incoming emails, which would increase the cost of tariffs. Therefore, choose only among opensource.

Popular opensource anti-spam solutions

Rspamd

It is suitable for systems of various sizes. It can integrate into various MTAs (Exim, Postfix, Sendmail and Haraka are described in the documentation) or work in the SMTP proxy mode.

The message rating system is the same as in SpamAssassin, in particular, based on various factors: regular expressions, DNS block lists, white, gray, blacklists, SPF, DKIM, statistics, hashes (fuzzy hashes) and other things - they are used only other algorithms.

Rspamd supports plugin extensions.

Apache SpamAssassin

Fame SA gained through the use of Bayesian filtering technology. Each message when passing tests receives a certain score and is placed in spam when the threshold is reached.

Easily integrates with almost any email service. In SA, popular technologies are available that connect as plugins: DNSBL, SPF, DKIM, URIBL, SURBL, PSBL, Razor, RelayCountry, automatic white list (AWL) and others.

Installation is generally not complicated. After installing SpamAssassin requires fine-tuning parameters and training on spam letters.

ASSP

A platform-dependent SMTP proxy server that accepts messages prior to the MTA and analyzes it for spam.

All popular technologies are supported: white and gray lists, Bayesian filter, DNSBL, DNSWL, URIBL, SPF, DKIM, SRS, virus scan (with ClamAV), blocking or replacing attachments and much more. Detected encoded MIME spam and images (using Tesseract). The possibilities are expanded with the help of modules.

Project documentation is not always intelligible, and the instructions are often outdated, but with some experience you can figure it out.

Mailscanner

MailScanner is an all-inclusive solution for dealing with phishing emails and checking mail for viruses and spam. It analyzes the content of the letter, blocking attacks aimed at email clients and HTML tags, checks attachments (forbidden extensions, double extensions, encrypted archives, etc.), controls the substitution of the address in the letter and much more.

MailScanner easily integrates with any MTA, there are ready-made configuration files in the delivery. In addition to his own work, he can use third-party solutions. SpamAssassin can be used to check for spam.

EFA-project

There is another open source project - “eFa-project” - Email Filter Appliance. EFA was originally designed as a virtual device to work on VMware or HyperV. The program uses ready-made MailScanner, Postfix, SpamAssasin packages (the entire list below) to stop spam and viruses, and they are already installed and configured to work properly in vm. This means that crutches are not needed - everything works out of the box.

EFA includes the following components:

Postfix is a MTA (mail transfer agent) - reliable, fast, proven over the years;
Kernel spam filter - MailScanner - shoulder to shoulder with antivirus take the whole blow;
Spam filter - SpamAssassin - identifies spam emails. The basis includes a variety of evaluation systems, MTA and sets of regular expressions;
ClamAV - antivirus that works with MailScanner;
MailWatch - a convenient web interface for working with MailScanner and other applications;
Content Filtering - DCC - determines the mass mailing by sending the hash sums of the body of letters to a special server, which in turn provides the answer in the form of the number of hashes received. If the number exceeds the threshold score = 6, the letter is considered spam;
Pyzor and Razor - help SpamAssassin more accurately recognize spam using spam detection networks;
SQLgrey is used for gray-listing - postfix policy service, which allows reducing the amount of spam that can be received by recipients;
For image recognition, the ImageCeberus module is used — identifies porn images, etc.
We chose EFA, because the project includes all the best features of the above. In addition, our administrators have already had some experience with it, so the choice was stopped on the EFA. We proceed to the description of the installation.

Install and configure EFA

They decided to install on a VPS with pure CentOS 6.8 x64, which acts as a relay-server. First of all, you need to update all system utilities and components to the latest versions that are available in the repositories. To do this, use the command:

yum -y update 

Then install the wget and screen utilities if they have not been installed:

 yum -y install wget screen 

After that, download the script that installs the EFA:

 wget https://raw.githubusercontent.com/EFA/v3/master/build/prepare-build-without-ks.bash 

We give the script the right to execute:

 chmod +x ./prepare-build-without-ks.bash 

Run the screen:

 screen 

And run the script:

 ./prepare-build-without-ks.bash 

Now you can minimize our screen using the Ctrl + A + D combination.

After installation, you need to re-enter the server via ssh, using the data for the first login. This is needed to run the initialization script and initial configuration of the EFA.

After logging in, the system prompts you to answer a few questions in order to configure the EFA.

The list of questions is as follows:

After such a questionnaire, the entire list of answers is displayed. If you need to change something, dial the question number and enter new data. When we are ready to move on, type OK and press Enter. The system will start the autotune process.

Upon completion of the configuration, the system will reboot and be fully operational.

Next time, logging in via ssh, the EFA configuration menu is displayed immediately. There are many useful actions in this menu:

• System reboot / shutdown;
• Change network settings;
• MailScanner setup;
• Turn on / off gray-listing;
• Enable / disable auto-update;
• Setting up the system as an outgoing relay-server;
• Change Adminemail box;
• Add / remove mail domains;
• Change spam filter settings;
• Recover mysql database in case of damage due to emergency shutdown.

This is a list of the main EFA options that cannot be edited through the MailWatch web interface. Therefore, it is good to know where to find them.

Manual EFA Setup

We went a difficult way, but more flexible. Customization of EFA for themselves was done not through an interactive menu, but the configuration files ruled. We wanted not just to set everything up, but also to understand all the components and understand what worked and how.

First of all, in the main.cf file of postfix settings, added mynetworks, from which SMTP connections were received. Then they wrote down restrictions on helo requests, senders, recipients, and indicated the paths to maps with ACCEPT or REJECT policies, subject to certain conditions. Also, inet_protocols was changed to ipv4 to eliminate ipv6 connections.

Then changed the Spam Actions policy to Store in the configuration file /etc/MailScanner/MailScanner.conf. This means that if the letter is identified as spam, it will go to quarantine. This helps to further train SpamAssassin.

After these settings, we encountered the first problem. Thousands of emails from you@example.com, fail2ban@example.com, root@localhost.localdomain, etc. have fallen upon us. Recipients were similar. We also received letters sent by MAILER-DAEMON, that is, in fact, without a sender.

As a result, we received a clogged queue without the possibility of finding normal, non-spam letters among the “red canvas”. We decided to make a REJECT of such letters using standard Postfix card functionality: helo_access, recipient_access, sender_access. Now the harmful recipients and the like have become successfully REJECT'itsya. And those letters that were sent to MAILER-DAEMON are filtered by helo requests.

When the queue was cleared, and our nerves calmed down, we began to set up SpamAssassin.

SpamAssassin Training

Learning SpamAssassin is done on emails that have already fallen into spam. There are two ways to do this.

Via web interface

The first method is via the MailWatch web interface. In each letter, you can see the headlines, the body, as well as evaluation by the Bayes algorithm and other indicators. It looks like this:

Having opened the letter, you can put a check in the SA Learn checkbox and choose one of several actions:

• As Ham - mark the letter as clean (training the Bayes algorithm);
• As Spam - mark the message as spam (Bayes algorithm training);
• Forget - skip the letter;
• As Spam + Report - mark the message as spam and send information about it in the network for spam detection (razor + pyzor);
• As Ham + Revoke - mark a message as clean and send information about it online to detect spam (razor + pyzor).

Via console

This is done simply. The command is as follows:

 sa-learn --ham /20170224/spam/0DC5B48D4.A739D 

In this command, a letter with ID: 0DC5B48D4.A739D, which is in the archive of spam emails for a specific date / 20170224 / spam /, is marked as clean (not spam) bash--ham .

There is an opinion that it is enough to train SpamAssassin only for effective mail filtering. We decided to train SpamAssassin, feeding him absolutely all letters, both clean and spam. In addition, we found a base of spam letters and gave SA to be torn apart.

Such training helped to more accurately calibrate the Bayesian algorithm. As a result, filtering is much more efficient. We carry out such trainings when the mail traffic is not very high in order to have time to analyze and capture the maximum number of letters.

In order for SpamAssassin to start working at full capacity, it needs to feed about 1000 different letters at the start. So have patience and start training.

It is too early to talk about a complete victory over spam. However, now the number of complaints about spam from our servers is zero. Now we will not talk in more detail about the learning process itself — I don’t want to disclose all the chips. Although, if you dig deeper in the settings, it is not difficult to understand.

PS: This article is not a panacea. We just decided to share with you one of our methods of dealing with spam.

All good and may the ham be with you! :)