Requirements for sending e-mail to major mail providers. Linux Mailsystem 2/4

Those who have only recently started operating their own mail server may have already noticed that their own e-mails are not always accepted by other servers and quickly end up in the spam suspicion folder. In fact, there are a few things to consider if you want to be accepted into the league of reputable mail providers. In order to achieve a good reputation with foreign systems, the following characteristics should be met:

• Static IP address – preferably not from a residential network.
• Host name in DNS (e.g. mail.testsys.de)
• PTR (Reverse DNS) record from IP address to host name
• SPF record in DNS
• DKIM signing for outgoing e-mails

Not all major mail providers require all features – however, most providers require at least a static IP, a valid hostname and a PTR record. SPF and DKIM increase the chances that one’s own mail server is classified as a reputable sender. Sent mails are then less likely to end up in the spam folder or are no longer completely rejected. But why all the effort at all? The reason is as follows:

Every mail server can send mails in the name of other mail servers. It is easily possible to send an email with my own mail server in the name of bestellungen@amazon.de, and thus give the impression that my email actually comes from Amazon (this is exactly what is exploited for phishing emails!). How is the recipient server supposed to know that the email actually comes from the specified sender and is not a fake? The techniques mentioned above revolve around this question. With their help it can be determined (sometimes more, sometimes less reliably) whether a server is authorized to send under a certain name / domain at all. To answer the question, the DNS is primarily consulted.

If we want to send emails to a popular, large provider, we should make sure that we know or support techniques like PTR, SPF and DKIM. Otherwise, our server will often be classified as a spam server and emails will be blocked.

PTR Record

A PTR record can be described as the opposite of a conventional DNS record. While a normal DNS record assigns one or more IP addresses to a name (a domain / subdomain), a PTR record assigns a name to an IP. Here’s how it works:

When receiving an email from a foreign server, the receiving server knows only one safe attribute: the IP address of the sending server that connected to it. Everything else (and that is the mail data itself) could be faked. If the sending server says „I am host mail.testsys.de“ and sends on behalf of „username@testsys.de“, then this is not necessarily true. And how to look up if the mentioned hostname is true? You try to find out the hostname to the IP and compare if it is correct! – And this is exactly what a server does when it checks for a PTR record via reverse DNS. If the host name of the mail server specified in the e-mail data matches the host name determined, the mail is accepted – otherwise it is probably a forged e-mail.

You can request a PTR record from your server hoster. You only have to tell them on which hostname the PTR record should be created, e.g. 123.155.144.256 <=> mail.testsys.de. Don’t forget to add a possibly existing IPv6 address. The hostname must be linked to the server IP in your DNS zone file.

SPF (Sender Policy Framework)

The SPF is a technique that can be used to determine whether a server is authorized to send an e-mail under the specified sender domain. After the checked PTR record has already determined that the host name of the sending server is correct, it is now checked whether this server is also authorized to send e-mails for the respective sender domain. A mail server running under the name „mail.testsys.de“ can not only send emails for „yourdomain.de“ users, but also for other domains like „yourhomepage.de“. This way you can serve multiple domains with only one mail server (multi-domain operation). Let’s assume that the receiving server receives an email from a sender „user@yourhomepage.de“ – how can it determine if host mail.testsys.de is even available to send from this mail address?

At this point the SPF record helps: The recipient server extracts the domain name from the sender address (here „myhomepage.de“) and downloads a possibly existing SPF record from the DNS zone of „myhomepage.de“. The downloaded SPF record lists which mail servers with which hostname or IP are allowed to send for this domain. If the hostname of the sending server is mentioned in this record, the e-mail is considered legitimate. If the hostname is not named, the sending mail server is apparently not authorized to send mail for this domain. The mail will be rejected.

You can create an SPF entry in your DNS zone file by yourself. At http://www.spf-record.de there is a very handy and easy to understand SPF generator. Select which servers under which hostnames and IP addresses are allowed to send mail for a domain and let it generate the code. The code is then inserted into a TXT record, e.g. like this:

Testsys.de.    86400    IN    TXT    v=spf1 a mx a:mail.testsys.de a:mail.testsystwo.de ip4:123.123.123.42 ?all

In this case, the following is specified:

a: Hosts that have already been mentioned in an A or AAAA record for the domain are allowed to send (e.g. user1@testsys.de).
mx: Hosts that have already been mentioned in an MX record for the domain are allowed to send (means: Receiving hosts are also allowed to send).
a: In addition, hosts that have the mentioned A-/AAAA record are allowed to send.
ipv4: The host with IP address 123.123.123.42 is also allowed to send.
?all: Presumably other mail servers are not allowed to send mail under this domain.

Thats all……

DKIM or DomainKeys Identified Mail 

Unfortunately, SPF has proven to be particularly ineffective when used in conjunction with mail relays and mailing lists.

For this reason DKIM was created, which is based on the verification of cryptographic signatures. Every outgoing email is automatically signed by the mail server with a secret key before it leaves its own mail server. The recipient server uses the sender domain to retrieve the DKIM record with the public key from the DNS and can thus check whether an e-mail actually originates from the server whose public key was published in the DNS. A server that does not have the matching private key cannot sign e-mails correctly, and the scam is busted. The method fixes some weaknesses of SPF, but is a bit more complicated to set up. The email filter framework Amavis, for example, supports DKIM signing. Alternatively, OpenDKIM can be used.

A DKIM entry can look like this, for example:

dkim._domainkey.testsys.de. 3600 IN TXT v=DKIM1; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCobe5aaqYeaUxLG3lTV3rcUP8OnB1KfGAOUU4/yYj/Fy0uOJNWajyGeAS9s3IEOhb5Zl8t0q4CB20ani3WfxhWQUGLCd3tkmira4nD8k5HMeCzUD5XbB1pmd3xTfdD03ahAm11Ii6skHYvZxJ+HOLdDaogDK8nkQw9cnydCZOTJwIDAQA

Let’s explain it with the example of AmavisD:

A DKIM signature establishes the digital identity of a sender domain. It is the basis for reputation-based email policies and helps effectively detect spam such as phishing and keep it away from users.

amavisd-new has long been able to detect and verify DKIM signatures in incoming messages and apply policies depending on the result of the verification. Less well known is that amavisd-new can also very easily apply DKIM signatures to outgoing messages. In this blog, I show how to enable and use all three of the above features.

Verify DKIM signatures:

amavisd-new relies on Jason Lang’s Perl library Mail-DKIM to recognize a DKIM-signed message and verify the validity of its signature. It is easy to install under Debian:

$ sudo apt-get install libmail-dkim-perl

To explicitly enable DKIM verification, the following parameter must be set in the configuration file – /etc/amavisd.conf or /etc/amavis/conf.d/50-user on Debian:

$enable_dkim_verification = 1;

After restarting, amavisd-new checks the signatures of DKIM-signed e-mails for their validity. The check result is noted in the header provided for this purpose:

Authentication-Results: mail.sys4.de (amavisd-new); dkim=pass (1024-bit key)
        header.d=example.com
...
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=example.com; s=mail;
        t=1378023161; bh=cDhLKw7xbii5H41D4ljCDX3wo8z1PPdSY5LI8gNj0Lg=;
        h=References:In-Reply-To:Mime-Version:Content-Type:
        Content-Transfer-Encoding:Message-Id:Cc:From:Subject:Date:To;
        b=KYyZ6V7A2zOoxTk29KVKvLBby4kKMLC9QQ3GVT0pVUUJHmp99rbHU3siZY5U29u6W
        rT+g+UNr3AiLr2rpJ7M7WX4SMVZyUe6DFEzYQJaQFrNPgMkImp2Yv3i4up3zzeMdOe
        7UIQYiASDHBAQt1NThBaIDE1UDyrZXn8Pxvf1Phk=

If amavisd-new does not insert Authentication-Results: headers for DKIM-signed messages, it may be because this feature has been explicitly disabled:

Control amavisd-new Policies with DKIM

Policies may be applied to valid signatures, but not to invalid ones. The reason for the invalid signature could be a processing error and not a phishing attempt on the sender’s side.

Note: This is similar to SPF: If the routing announced in SPF is correct, you can use it for the sender. If it is not correct, one should not use the deviation against the sender.

If amavisd-new finds a valid DKIM signature, it checks @author_to_policy_bank_maps to see if the sender or sender domain is listed there with a policy:

@author_to_policy_bank_maps = ( {
    'python.org'              => 'WHITELIST,NOBANNEDCHECK,NOVIRUSCHECK',
    'amavis.org'              => 'WHITELIST,NOBANNEDCHECK,NOVIRUSCHECK',
    'p@testsys.de'               => 'WHITELIST,NOBANNEDCHECK,NOVIRUSCHECK',
    'postfix.org'             => 'NOBANNEDCHECK',
} );

Sender or Sender domain refer to $policy_banks that enable or disable certain functions:

$policy_bank{'WHITELIST'} = {
  bypass_spam_checks_maps => [1],
  spam_lovers_maps => [1],
};

$policy_bank{'NOVIRUSCHECK'} = {
  bypass_decode_parts => 1,
  bypass_virus_checks_maps => [1],
  virus_lovers_maps => [1],
};

$policy_bank{'NOBANNEDCHECK'} = {
  bypass_banned_checks_maps => [1],
  banned_files_lovers_maps  => [1],
};

Note: The $policy_banks mentioned in the example must be created manually. They are not part of a standard installation.

Apply DKIM signatures to outgoing messages

DKIM signing of a message requires an x509 key pair. The private key is passed to amavisd-new for signing the DKIM signatures. The public key is published via DNS – so content filters on the receiving end can easily download it and use it to verify the signature.

amavisd-new brings its own commands to generate x509 key pairs, to output them and to check if they can be used in principle. Let’s start by generating a key pair. The following command generates a key pair for the sender domain example.com with a key length of 2048 bits:

% amavisd-new genrsa /var/lib/amavis/db/testsys.de 2048
Private RSA key successfully written to file "/var/lib/amavis/db/example.com" \
  (2048 bits, PEM format)

Note: Keys should always have a length of at least 1024 bits. Anything less can be abused in finite time

Once the key pair is generated, the private key must be declared in the amavisd-news configuration file for the sender domain:

$enable_dkim_signing = 0;
dkim_key('example.com', 'mail201309', '/var/lib/amavis/db/example.com');
@dkim_signature_options_bysender_maps = (
    { '.' =>
        {
                ttl => 21*24*3600,
                c => 'relaxed/simple'
        }
    }
);

Important

Do not enable $enable_dkim_signing until the public key has been published and tested. Otherwise amavisd-new will already sign outgoing messages while recipients cannot yet verify the signature. This can lead to rejects and bounces!

Now the public key can also be printed. amavisd-new generates the appropriate ZONE entry for a bind nameserver. The TXT record also lists the optional DKIM entries made in @dkim_signature_options_bysender_maps:

% amavisd-new showkeys example.com
; key#1, domain testsys.de, /var/lib/amavis/db/testsys.de
mail201309._domainkey.testsys.de.      3600 TXT (
  "v=DKIM1; p="
  "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvrqQssP4rC6bo22YzzpX"
  "zd0OCBRrcN6uPpRrvPq69Nn/i/B3dSzu639OWt622QXdUBS8e5xFZnZslPMMIRDl"
  "QIJx/HFoYY6wgUm4B8Nb83SJK0aLGtlMwxhlGEkSX8gNlmMsA28LWeEIQhJyA9+D"
  "7j7DcA0aGJU3JzFgG/YHBYL8+6RWAKf+vgGQNmGYBPCb3Mm5LQqAzqE55yW+iL06"
  "cYHs+7niLWyAbosRhfCea0i+WqBIrclH32mXwNt/FO43lDlnPdy+gaHaE4fnqNqN"
  "3jK5YnslEd3mHJd/a20eaHjwjXSP6F48KKsQuL+rezohlG5QKQF9qIZbHHEHp4Yn"
  "AwIDAQAB")

This TXT entry must now be entered in the ZONE file of the sender domain (here: example.com). After a reload of the zone, amavisd-new can check whether the public key is accessible and the keys match:

% amavisd testkeys testsys.de
TESTING#1: mail201309._domainkey.testsys.de     => pass

If the test returns a pass, everything is in the green.

In order for amavisd-new to start signing outgoing messages, not only the already entered $enable_dkim_signing must be activated. amavisd-new must also know which messages are outgoing.

Outgoing messages are marked in amavisd-new by the parameter ORIGINATING. This is not set by default. It must be controlled, i.e. applied only to trusted networks and submission senders. In a $policy_bank for submission senders it looks something like this:

$policy_bank{'SUBMISSION'} = {
    originating => 1,
    bypass_spam_checks_maps   => [1],
    bypass_banned_checks_maps => [1],
    final_virus_destiny => D_REJECT,
    final_bad_header_destiny => D_PASS,
    terminate_dsn_on_notify_success => 0,
    warnbadhsender => 1,
};

After a reload, amavisd-new will add DKIM signatures to outgoing messages.

DMARC (Domain-based Message Authentication, Reporting and Conformance)

DMARC stands for Domain-based Message Authentication, Reporting and Conformance. It was developed to reduce e-mail abuse. This technology uses the DKIM (Domain Keys Identified Mail) and SPF (Sender Policy Framework) specifications.

The DMARC policy allows you to specify that your email is protected using SPF (Sender Policy Framework) and DKIM (DomainKeys Identified Mail) procedures. In addition, you can use this policy to define how the recipient should handle email messages depending on the results of the DKIM and SPF checks, and to notify you as the domain owner of any misuse.

DMARC policies are added in the form of a TXT record. This record is called a DMARC record. This is created with the subdomain name _dmarc (for example, _dmarc.example.com). The content of the TXT record consists of tags separated by semicolons.

Example:

v=DMARC1;p=reject;pct=100;rua=mailto:postmaster@testsys.de

Structure of a DMARC record
The following are the common tags used in DMARC (TXT) records:

Example configurations

This configuration rejects all emails that do not match DKIM and SPF check results. In addition, an aggregated status report is sent to the email address postmaster@example.com.

v=DMARC1;p=reject;pct=100;rua=mailto:postmaster@testsys.de


The configuration in the following example does not reject any emails and sends an aggregated status report of all emails that do not match the SPF and DKIM check results for the domain and all subdomains to the postmaster@example.com email address. 

v=DMARC1;p=none;pct=100;rua=mailto:postmaster@testsys.de

This configuration does not affect email delivery and is particularly useful for testing.


The configuration below marks all emails that do not match DKIM and SPF check results as spam or suspicious. In this case, 10% of the emails are checked. In addition, a report is sent to postmaster@testsys.de.com for each error.

v=DMARC1;p=quarantine;pct=10;ruf=mailto:postmaster@testsys.de

Configuring the DMARC record
You can configure the DMARC (TXT) record in the Control Center.

Click the gear icon under Actions for the desired domain, and then click DNS.

Click ADD RECORD and select TXT under Type.

In the Hostname field, enter the subdomain name _dmarc.
The subdomain _dmarc.your-domain.com is automatically created for the TXT record.

In the Value field, enter the desired tags, which you can separate with a semicolon (;).
Example: v=DMARC1;p=reject;pct=100

Optional: Select the desired TTL (Time-To-Live).

Click on Save.

Your changes will take effect immediately on IONOS. However, it may take up to 1 hour for the change to take effect everywhere due to the decentralized structure of the Domain Name System.

Final word..

If you have configured all these mechanisms correctly, your mails should no longer end up in spam, even with large mail providers. If this is still the case, we recommend using a mail tester to see which blacklist you are on and then asking the provider separately to release your IP address.