Exim

From HerzbubeWiki
Jump to navigation Jump to search

This page contains information about how to configure Exim, the default MTA (mail transfer agent) in Debian.


Debian packages

Install the Debian meta package 

exim4

and the daemon package 

exim4-daemon-heavy

The "heavy" variant of the daemon is required because it contains the so-called exiscan patch, a content-scanning extension to Exim which is a pre-requisite for the integration of external tools such as SpamAssassin and virus-scanners.


Also install this package for greylisting support: 

greylistd


References

Weblinks:


Some shortcuts into the Exim 4 documentation

  • List with all expansion variables: Chapter 11.9
  • General syntax for expansion (e.g. ${if <condition> ...}): Chapter 11.5 (Expansion items)
  • Exim filter files: Chapter 3 ("Exim filter files") in the document called "Exim filter specification" (/usr/share/doc/exim4-doc-html/html/filter_3.html)


Basic understanding of how Exim works

If you have no idea about how Exim works, you should read chapter 3 of the Exim documentation ("How Exim receives and delivers mail").


The following itemized list is only a quick reminder for the basic concepts:

  • Routers and transports are all drivers
  • A specific named router or transport is an instance of a driver, the type of which is given by the driver=... option. I like to compare this with classes (types) and objects (instances) in an object-oriented programming language.
  • A router operates on an address: It routes to a transport, generates a new address (e.g. alias), or simply declines
  • A transport transmits a message copy to some destination (e.g. a file)
  • ACLs (Access Control Lists) define how Exim responds to SMTP commands (e.g. the EHLO command)


Basic daemon configuration

Exim runs in the background as a daemon process when the command line option -bd is specified. A number of basic daemon attributes can be changed by editing the file 

/etc/default/exim4


After the file has changed, the daemon needs to be restarted with 

/etc/init.d/exim4 restart


One important option is the so-called "queue interval", i.e. how often the daemon should process its queue. By default this is set to 30 minutes (-q30m). This is much too long: If a client queues its messages instead of sending them immediately via a direct SMTP session to the destination MTA, the messages are stuck for 30 minutes on the system until Exim finally sends them out. I have set the queue interval to 5 minutes on my system.


Network configuration

The FQDN

The file 

/etc/mailname

contains a single line with the FQDN (fully qualified domain name) of the mail system.

In my case this is 

pelargir:~# cat /etc/mailname
herzbube.ch


Surprisingly it is not pelargir.herzbube.ch. Reason: The domain is herzbube.ch, the host is pelargir.


Hostname in /etc/hosts

The file /etc/hosts should contain a line that resolves the system's hostname, i.e. pelargir. I used to have this configuration 

127.0.0.1      localhost pelargir

but nowadays, since the machine is a full-blown Dedicated Server that has a network interface with a public IP, I prefer this configuration: 

82.195.228.21	pelargir.herzbube.ch pelargir [... other names ...]


If the hostname is not resolved like this, Exim may have difficulties in certain situations. For instance, on one occasion I had the following problem:

  • Client connects
  • Client gives an invalid name during HELO
  • Exim tries to lookup the name via IPv6 and fails (gethostbyname2(af=inet6) returns with HOST_NOT_FOUND)
  • Instead of continuing with the IPv4 lookup, Exim now blocks forever


It is not entirely clear why this happened. The issue was definitely connected to my running a local DNS server, because instead of adding the hostname to /etc/hosts I was also able to fix the problem by disabling the local DNS server (which was not configured for IPv6). Exim started to use an external DNS server, and even though that server also did not provide IPv6 support this did not cause any trouble. I assume I had mis-configured something in the local DNS server, but to this day I do not know what it was.


Exim configuration overview

Filesystem locations

Exim requires a single file that contains the entire configuration. On a Debian system, this file is located here: 

/var/lib/exim4/config.autogenerated


You should never edit this monolithic file. As is usual for Debian, configuration has been split into several files, which are all located in this folder: 

/etc/exim4/conf.d


In addition, answers to Debconf questions and other options are stored in this file: 

/etc/exim4/update-exim4.conf.conf

Note: You can manually edit this file, but you should only change those options that are not handled by Debconf. As usual, Debconf-handled options should be changed by running dpkg-reconfigure (see next section).


Activating configuration changes

After changing any of the files in /etc/exim4, the monolithic config.autogenerated must be re-generated. The following Debian-specific command does this: 

update-exim4.conf

Refer to man update-exim4.conf for more information.


Before you activate the changes, you should first test them. Running the following command simulates a connection from IP address 212.101.18.224. This is extremely useful if you want to (relatively) quickly test out ACL changes. To simulate a connection from a specific port, add the port to the IP address using a full stop as separator (e.g. add ".587" to simulate a connection from port 587). 

exim4 -bh 212.101.18.224


To activate the changes you have to restart the Exim daemon 

/etc/init.d/exim4 restart


Debconf questions and answers

The Exim configuration process starts with a number of high-level questions asked by Debconf. Surprisingly it is not the exim4 package that needs configuration, instead it's exim4-config. So if you ever need to change your answers to the Debconf questions, you have to run this: 

dpkg-reconfigure exim4-config


Here are the questions Debconf asked the last time I ran it, and my answers. The code in parenthesis is the technical ID of the Debconf question.

  • (dc_eximconfig_configtype) General type of mail configuration = internet site; mail is sent and received directly using SMTP
  • (mailname) System mail name = herzbube.ch
    • The default is pelargir.herzbube.ch
  • (dc_local_interfaces) IP-addresses to listen on for incoming SMTP connections = 127.0.0.1 ; ::1 ; 82.195.228.21
    • This could also be left empty, in which case Exim would listen on all available network interfaces
  • (dc_other_hostnames) Other destinations for which mail is accepted = herzbube.ch ; moser-naef.ch ; francescamoser.ch ; grunzwanzling.ch
    • The system mail name must be repeated here
    • The local hostname pelargir.herzbube.ch and localhost are always added to the list specified here
  • (dc_relay_domains) Domains to relay mail for = <empty>
  • (dc_relay_nets) Machines to relay mail for = <empty>
    • In a home network many computers and mobile devices will submit mail, so this might have to be set to something such as 192.168.0.0/16
    • But if the server is only accepting mails from local processes, this must be left empty
  • (dc_minimaldns) Keep number of DNS-queries minimal (Dial-on-Demand)? = No
  • (dc_localdelivery) Delivery method for local mail = Maildir format in home directory
  • (use_split_config) Split configuration into small files? = Yes (default = No)
    • I prefer a split configuration, this entire Wiki page is based on the assumption that the configuration is split and not monolithic


A few notes:

  • As mentioned in the previous section, Debconf answers are not only stored in the Debconf database, but also in the file /etc/exim4/update-exim4.conf.conf
  • Various settings in the configuration files below /etc/exim4/conf.d contain place holders that are replaced by the answers given to Debconf at the time update-exim4.conf is run


File /etc/exim4/update-exim4.conf.conf

Besides the answers to Debconf questions, the file 

/etc/exim4/update-exim4.conf.conf

stores a couple of other configuration options. Refer to man update-exim4.conf for details.


Directory /etc/exim4/conf.d/main

This directory contains files that define configuration options that go into the main configuration file section. The section Solutions lists files that I use to achieve a specific effect.


This file 

/etc/exim4/conf.d/main/01_exim4-config_listmacrosdefs

defines some important Exim configuration options. Although I have not edited this file, I am discussing its content here in order to understand where those Exim configuration options take their values from.

local_domains
This option is a list. It is generated from
  • /etc/mailname (= herzbube.ch)
  • The Debconf answer dc_other_hostnames (herzbube.ch ; moser-naef.ch ; francescamoser.ch ; grunzwanzling.ch)
  • localhost
relay_to_domains
This option is a list. It is generated from the Debconf answer dc_relay_domains. In my case it is empty because I do not allow relaying to other domains.
relay_from_hosts
This option is a list. It is generated from
  • 127.0.0.1
  • The Debconf answer to dc_relay_nets (currently none, but in a home network this could be something like 192.168.0.0/16)
qualify_domain
The value of this option is set with the content of the file /etc/mailname (= herzbube.ch)
local_interfaces
This option is a list. It is generated from the Debconf answer dc_local_interfaces (127.0.0.1 ; ::1 ; 82.195.228.21)


Directory /etc/exim4/conf.d/acl

This directory contains files that define ACLs. The order in which files appear is not important because ACLs become active only when they are referenced by other sections of the configuration.

For more information on ACLs

  • The section ACLs discusses how ACLs work in general
  • The section Solutions explains the concrete ACLs I use to achieve a specific effect


Directory /etc/exim4/conf.d/router

This directory contains files that define routers. Because the order in which routers appear in the Exim configuration is important, files in the router directory must have a numbered prefix that makes sure in which order these files are concatenated into the final, monolithic configuration file.

For more information on routers

  • The section Routers discusses how routers work in general
  • The section Solutions explains the concrete routers I use to achieve a specific effect


Directory /etc/exim4/conf.d/transport

This directory contains files that define transports. The order in which files appear is not important because transports operate only when they are referenced by a router.

For more information on transports

  • The section Transports discusses how transports work in general
  • The section Solutions explains the concrete transports I use to achieve a specific effect


Mail submission

Overview

Mail submission is what happens when a mail client (aka Mail User Agent, or MUA) connects to a mail server (aka Mail Transfer Agent, or MTA) and submits a message via SMTP to be sent to its destination address. After the MTA accepts the message it then relays the message to the next MTA via SMTP, which in turn may relay the message again to another MTA, and so on and so forth, until the final MTA is reached which then delivers the message to the recipient's mailbox.

Originally only a single port was used for both SMTP submisson and relay: The canonical port 25.

These days port 25 is mostly used for SMTP relay, but a different port is used for SMTP submission: Port 587. Port 25 in theory can still be used for SMTP submission, but in practice it is often blocked by public networks.

To avoid problems with blocked ports, especially when traveling, we therefore should configure Exim to listen not only on port 25, but also on port 587.


/etc/default/exim4

One way to achieve this is to set the variable SMTPLISTENEROPTIONS in file /etc/default/exim4. The command line option to use is -oX. When this option is used Exim no longer writes its process ID into a file, therefore PID file writing must also be enforced with the -oP option. The final result would look like this: 

SMTPLISTENEROPTIONS='-oX 25:587 -oP /var/run/exim4/exim.pid'


daemon_smtp_ports

Using the -oX option causes the configuration option daemon_smtp_ports to be set. Since I use the split configuration model, it is therefore also my preferred option to enable port 587 by creating a config file in /etc/exim4/conf.d/main and setting daemon_smtp_ports in that file. This also provides the opportunity to improve logging so that the port used for connecting is logged. 

# 00_exim4-config_pelargir_ports
  
# The ports we define here are used for listening on all network
# interfaces 1) that are listed in option "local_interfaces",
# and 2) for which no port is specified in "local_interfaces".
# - Port 25 is the canonical SMTP port. It is used nowadays mostly
#   for SMTP relay, i.e. moving messages between servers. It can
#   be used for SMTP submissions, but is frequently blocked by
#   public networks.
# - Port 587 these days is used by MUAs for SMTP submissons.
#
# Note: Instead of specifying port numbers we could also use
# service names, i.e. names from /etc/services. However, using
# port numbers seems more straightforward.
#
# If this setting is left undefined it has the default value "smtp",
# which is equivalent to port 25.
daemon_smtp_ports = 25:587

.ifdef MAIN_LOG_SELECTOR
MAIN_LOG_SELECTOR == MAIN_LOG_SELECTOR +incoming_interface
.else
MAIN_LOG_SELECTOR = +incoming_interface
.endif


ACLs

Definition and Usage of ACLs

ACLs are used to control Exim's behaviour when it receives certain SMTP commands (e.g. the EHLO command).

ACLs need to have a name (the name is arbitrary). They are defined in the configuration file section that starts with begin acl. The order in which ACLs are defined is not important. ACL definitions should be placed in files thare are located in this folder 

/etc/exim4/conf.d/acl

In order to actually use an ACL, it must be referenced in the main configuration file section, e.g. like this: 

acl_smtp_rcpt = foobar_acl

In the example foobar_acl is called when Exim receives the RCPT command.


SMTP Dialog

Chapter 43 ("Access control lists") subsection 2 ("Specifying when ACLs are used") in the Exim documentation lists where/when during the SMTP dialog it is possible to call an ACL.

If no ACL is defined for a specific part of the SMTP dialog, exim assumes either accept or deny as the default. Most important is that deny is the default for acl_smtp_rcpt, which means that no messages at all can be received via SMTP if no ACL is defined for the RCPT command.


Format of an ACL

For details see chapter 43 "Access control lists" in the Exim documentation.

A very short description of the ACL format:

  • An ACL contains one or more verbs (e.g. accept, deny, defer, discard, warn)
  • Each verb is followed by conditions and modifiers
  • A verb is "executed" only if all of its conditions are met
  • Not all conditions use the keyword "condition" - that keyword exists to allow generic checks, but most conditions use keywords that test for something very specific. For instance, the "hosts = ..." condition tests if the connecting host is in the specified host list. A list of all conditions can be found in chapter 43, subsection 26 "ACL conditions" of the Exim documentation.

Example for a verb: 

drop   message   = I don't take more than 20 RCPTs
       condition = ${if > {$rcpt_count}{20}}


Variables (chapter 39.15)

  • $acl_c0 - $acl_c9 retain their value during the entire SMTP connection
  • $acl_m0 - $acl_m9 retain their value only while a message is being received

The values of both sets of variables are later accessible when the message is delivered (unless the message was denied, of course).


Testing an ACL

In order to test a new ACL the monolithic configuration file needs to be generated with 

update-exim4.conf

DO NOT restart the Exim daemon yet (otherwise it would immediately start to use the ACL you want to test). Instead run the following command to simulate an SMTP connection: 

exim4 -bh 212.101.18.224

Notes:

  • The example simulates a connection from IP address 212.101.18.224.
  • Depending on how far into the SMTP dialogue you want to go, you need to temporarily disable some things
    • Go beyond EHLO: Disable TLS by setting MAIN_TLS_ENABLE=0 in main/00_exim4-config_pelargir_tlsoptions
    • Go beyond MAIL FROM: Disable authentication by commenting out the accept option that contains authenticated = * in acl/62_exim4-config_pelargir_mail_from, acl/63_exim4-config_pelargir_rcpt_to and/or acl/64_exim4-config_pelargir_data
  • If an ACL defines a delay, the simulation displays the delay but does not execute it.
  • Warnings/errors related to tainted data do show up in the simulation.


For reference, this is an example SMTP dialog: 

EHLO morannon.herzbube.ch
MAIL FROM:<herzbube@herzbube.ch>
RCPT TO:<herzbube@herzbube.ch>
DATA
Subject: Test subject
From: test-from <herzbube@herzbube.ch>
To: test-to <herzbube@herzbube.ch>
Test body
.
QUIT


Routers

Overview

The most important points about routers are:

  • A router operates on an address
  • Routers can do different things:
    • Route to a transport
    • Generate a new address (e.g. alias)
    • Or, simply, decline further processing
  • Routers are defined using a name and a set of options. The name must not contain "." or "-" characters, but can otherwise be freely chosen
  • Routers are drivers. A specifically named router is an instance of a driver, the type of which is given by the driver=... option.


Example of a router definition: 

 router_foo:
   debug_print = "router_foo"
   driver = accept
   domains = +local_domains
   transport = transport_foo
   [...]

Discussion:

  • The example router is an instance of the accept driver
  • The router is "activated" for all addresses that match the domains option, i.e. for all addresses in the local domains
  • The message to which the address currently being processed is attached is transmitted using the transport transport_foo.


Storage

Router definitions should be placed in files thare are located in this folder 

/etc/exim4/conf.d/router


Transports

Overview

The most important points about routers are:

  • Transports are used by routers. When a router operates on an address it will select a transport, thereby "activating" that transport
  • A transport transmits a message copy it receives from the router to some destination (e.g. a file)
  • Transports are defined using a name and a set of options. The name must not contain "." or "-" characters, but can otherwise be freely chosen
  • Transports are drivers. A specifically named transport is an instance of a driver, the type of which is given by the driver=... option


Example of a transport definition: 

transport_foo:
  debug_print = "transport_herzbube for $local_part@$domain"
  driver = appendfile
  file = $home/inbox
  [...]

Discussion:

  • The example transport is an instance of the appendfile driver
  • The message processed by the transport is appended to the file $home/inbox
  • The value of $home is determined by the router
  • The format of the file is the traditional Unix mailbox format


Storage

Transport definitions should be placed in files thare are located in this folder 

/etc/exim4/conf.d/transport


Transports of type "appendfile"

Local delivery is performed by a transport that is an instance of the appendfile driver. The transport can deliver messages either

  • By appending them all to a single "mbox" file
  • Or by creating a new file for each message inside a "Maildir" directory

The two cases are distinguished by specifying either the file or the directory option to the transport. These options are mutually exclusive, but one of them must be set.

The single "mbox" file approach is the traditional way to store mail on Unix based mail servers. The one-file-per-message approach is used by modern IMAP servers to provide access to mail reader clients. Note: Maildirs for new users should be created using the maildirmake.dovecot from the Dovecot package (or the maildirmake program from the Courier package if you're using Courier IMAP): 

maildirmake.dovecot /home/<user>/Maildir


Concrete examples of an appendfile transport that delivers to an IMAP Maildir can be found further down in the section titled Mail delivery on pelargir.


.forward file

Overview

The previous section stated that local delivery is performed by transports of type appendfile. However, the delivery process does not end with Exim having determined a local user that should receive a message. Often users want to provide individual rules that determine the final location where a message should be placed.

Traditionally, a user's .forward file is used to forward the message to one or more users, either on the same or on a different machine. Exim, however, also supports that the file contains a special kind of command syntax. A file with such content is called an "Exim filter file". For details about this, read chapter 3 ("Exim filter files") in the document called "Exim filter specification". This is a distinct document in (i.e. not just a chapter of) the Exim documentation.

To distinguish an Exim filter file from the conventional .forward file format, the first line must look exactly like this: 

# Exim filter


Exim filter file example

A simple example for the entire content of a .forward file that is an Exim filter file might look like this: 

# Exim filter

# A sequence of if-elif-endif
if $h_X-Spam-Status: contains "Yes" or
   "${if def:h_X-Spam-Flag {def}{undef}}" is "def" then
   save Maildir/.Junk/
elif $h_to: contains "debian-news@lists.debian.org" then
   save Maildir/.Lists.debian-news/
elif $message_body: contains "foobar" then
   save Maildir/.Foobar/
endif
 
# This is an essential catch-all for all messages that have fallen through the previous checks
if not delivered then
   save Maildir/
endif

Discussion:

  • Messages are saved into the Maildir directory because I use Dovecot IMAP on my system, and the Maildir directory is where Dovecot expects messages to reside
  • If the location given to the save command does not start with a "/" character, the contents of the $home variable are prepended
  • The directory given to a save command must end with a "/" character because the transport performing the delivery is of the "create-a-new-file-for-each-message" kind (i.e. the transport is configured for delivery suitable to Dovecot)
  • The check $message_body: contains is limited to what the variable $message_body contains; by default the variable contains only the first 500 characters of the message; if more content is required, the option message_body_visible must be set to a higher value in the main part of the Exim configuration file


Requirements for routers

It is the router's job to enable the use of a .forward file. Here is a partial router definition that shows the relevant options: 

router_herzbube:
  driver = redirect
  [...]
  file = $home/.forward
  allow_filter
  user = patrick
  modemask = 022
  router_home_directory = /home/patrick
  [...]

Discussion:

  • The option file specifies the location of the .forward file.
  • The option allow_filter specifies that the file is an Exim filter file. Without this option, Exim expects the file to be in the conventional .forward file format.
  • The option user is mandatory if allow_filter is set. It instructs the router to execute the .forward file as the specified user.
  • The option modemask is set with those mode bits that must not be set on the forward file. In the example, the router requires that the file is not group-writable and not world-writable.
  • The option router_home_directory is set because the router performs no recipient verification (see no_verify), therefore is not associated with a user, and thus has not set the $home variable. This variable is required, though, if commands inside the .forward file do not use absolute paths.


Mail delivery on pelargir

Summary

The following sections discuss two different approaches for mail delivery:

  • The first approach uses domain-specific routers and transports. This is not how the current solution on pelargir looks like - this section merely illustrates how a simple solution for a single domain could look like, and what the drawbacks of this solutions are as soon as you start to expand from a single domain to multiple domains.
  • The second approach is called "Virtual domains". It supports an unlimited amount of domains with only minimal configuration overhead. This is the current solution that is in place on pelargir.


Whatever approach is finally used on pelargir, it must meet the following requirements:

  • All addresses in a domain must be valid and delivered to a local user, even if there is no direct correspondence between an address and a local user, and even if there is no explicit alias definition for an address. This I require so that I am free to use arbitrary addresses when I register an account at some website, either because I truly want a service-specific address, or because I want to use a spam trap address.
  • Different addresses of one domain must be mappable to different local users. This I require because I want to share a domain with my wife.
  • It must be possible to deliver a copy of the same message to several local users. This I require so that notifications of an event can be sent to both myself and my wife simultaneously.


Domain-specific routers and transports

Transports

As explained further up, message delivery to a local user is made by transports that are an instance of the appendfile driver. This section has examples both for "mbox" and for "Maildir" delivery.


Example 1: The following transport appends all messages to the "mbox" file /var/mail/patrick: 

transport_patrick:
  debug_print = "T: transport_patrick for $local_part@$domain"
  driver = appendfile
  file = /var/mail/patrick
  delivery_date_add
  envelope_to_add
  return_path_add
  user = patrick 
  group = mail
  mode = 0660
  mode_fail_narrower = false

Discussion:

  • The various _add options add headers to the message that contain useful/interesting information. These options are purely optional, though.
  • The options user and group determine the user/group under whose uid/gid the delivery process is to be run, overriding any uid/gid that may have been set by the router. These options are especially important because the delivery process needs to create a lock file in the directory /var/mail (where the mbox file is located).
  • The option mode tells the transport to create the mbox file using the given permissions. If the mbox file already exists and has wider permissions, the permissions are reduced to the given ones. If the mbox file already exists and has narrower permissions, the option mode_fail_narrower decides whether delivery fails or succeeds. In the above example, delivery succeeds (and the narrower permissions are left unchanged).
  • The format of the mbox file is the traditional Unix mailbox format. Other formats can be specified using additional options (e.g. mbx_format), but Exim must have been compiled explicitly to use these formats.


Example 2: The following transport creates a new file for each message inside the directory /home/patrick/Maildir: 

transport_patrick:
  debug_print = "T: transport_patrick for $local_part@$domain"
  driver = appendfile
#  directory = /home/patrick/Maildir
  maildir_format
  delivery_date_add
  envelope_to_add
  return_path_add
  user = patrick 
  group = mail
  mode = 0660
  mode_fail_narrower = false

Discussion:

  • Most of the options are explained in the previous chapter.
  • The directory option is commented out because it would interfere with the commands (e.g. save Maildir/.foo/) inside the .forward file used by the router that calls this transport. See the following chapters for details about .forward files.
  • The maildir_format option specifies that the files created by this transport should have the so-called "Maildir" format. This format is required by the Dovecot server that I am using.


Routing to a transport

The following router processes all addresses in the herzbube.ch domain. It uses the transport_patrick transport from the previous section to deliver the messages. 

router_herzbube:
  driver = redirect
  domains = herzbube.ch
  directory_transport = transport_patrick
  check_local_user = false
  file = $home/.forward
  allow_filter
  user = patrick
  no_verify
  check_ancestor
  modemask = 022
  router_home_directory = /home/patrick

Discussion:

  • The option directory_transport must be used because the messages should be delivered by a transport using the directory option. If the transport were delivering to an mbox file, this router would have to use the file_transport option.
  • When the option check_local_user is set to false, there is no check if the local part of the address is a valid local user listed in /etc/passwd.
  • The options file, allow_filter, user and modemask are required to enable .forward file support. For details see the section about .forward files further up on this page.
  • The option no_verify is required because a redirect router cannot be effectively used for recipient verification (this is usually done while the SMTP dialog is running). See the Exim documentation, chapter 22.2 ("Forward files and address verification") for more information. Also see the next section, which presents a router that performs recipient verification.
  • The option check_ancestor is specified to prevent the router from getting into an infinite loop.
  • The option router_home_directory is set because the router performs no recipient verification (see no_verify) and therefore is not associated with a user, and has not set the $home variable. This variable is required, though, if commands inside the .forward file do not use absolute paths. Note: the $home variable remains set for the transport, unless the transport has its own home_directory or transport_home_directory set.


Routing for recipient verification

The ACL acl_check_rcpt requires that recipient verification be performed during the SMTP dialog. The router presented in the previous section is not suitable for this task and therefore has set the option no_verify.

The following router is therefore presented as a kind of "dummy" router that has only one task: performing recipient verification: 

router_herzbube_verify:
  driver = accept
  domains = herzbube.ch
  verify_only

Discussion:

  • Because of the option verify_only the router does not need a transport.


Virtual domains

Comparison

The solution using the routers router_herzbube and router_herzbube_verify, and the transport transport_patrick, has two important drawbacks:

  • For every new domain we need to write 2 new routers (3 when spam filtering comes into play, see the Spam filtering section further down) and 1 new transport
  • Each domain is mapped to a single user. Without modifications to the presented solution it is not possible to have 2 or more users share addresses within the same domain.

The "virtual domains" approach solves these problems.


Virtual domains router

The following router is the key to a universal and flexible solution: 

router_virtualdomains:
  debug_print = "R: router_virtualdomains"
  driver = redirect
  # The "domains" option places the untainted result of the lookup into the $domain_data variable
  domains = dsearch;/etc/mail/virtual
  data=${lookup{$local_part}lsearch*{/etc/mail/virtual/$domain_data}}
  no_more

Discussion:

  • The dsearch part in the domains option tells the router to look for a file inside /etc/mail/virtual that is named the same as the current domain. If such a file exists, the router processes the address. Here's a list of virtual domain files:
root@pelargir:~# ls -l /etc/mail/virtual
total 16
-rw-r--r-- 1 root root  23 Jul 14 00:03 francescamoser.ch
-rw-r--r-- 1 root root  21 Jul 14 00:06 grunzwanzling.ch
-rw-r--r-- 1 root root  21 Jul 14 00:03 herzbube.ch
-rw-r--r-- 1 root root 368 Jul 14 00:03 moser-naef.ch
  • In the data option a lookup is made in a virtual domain file. It is important to NOT use the variable $domain to form the filesystem path, because that variable contains tainted data, i.e. data from an untrusted source. Using $domain results in an error in newer Exim versions. Instead use $domain_data, which contains the untainted result of the lookup made in the domains option. The variable $domain_data is associated with the domains option, not the dsearch lookup.
  • The data option treats the virtual domain files as a kind of alias file. For instance:
pelargir:~# cat /etc/mail/virtual/moser-naef.ch
patrick: patrick@localhost
francesca: francesca@localhost
foo: patrick@localhost,francesca@localhost
*: patrick@localhost
  • When the router finds a matching entry in the virtual domain file, it spawns a new cycle of address processing, again beginning with the first configured router. The address being processed is now the "alias" that was found in the virtual domain file. Note: it is possible to verify the new address processing cycle by looking at the message headers. The original To: message header is left unchanged, though.
  • The option no_more makes sure that no more routers are tried after this one. In other words, all routers behind the virtual domains router become irrelevant.
  • Note: Do not (!) set the option no_verify as this would prevent recipient verification by ACLs during the original SMTP dialog.


Routers and transports for local delivery

The following routers and transports are used as the back-ends for the virtual domains router presented above. The idea of the virtual domains router is to "convert" all domain addresses to local addresses that can then be processed by 2 routers (a verifying and a non-verifying variant) and a single transport. 

# This router performs recipient verification during SMTP
router_localuser_verify:
  verify_only
  driver = accept
  address_data = $local_part
  domains = localhost
  check_local_user

# This router "executes" the transport for actual delivery
router_localuser:
  driver = redirect
  domains = localhost
  directory_transport = transport_localuser
  check_local_user
  file = $home/.forward
  allow_filter
  no_verify
  check_ancestor
  modemask = 022

# This transport performs actual delivery
transport_localuser:
  driver = appendfile
  delivery_date_add
  envelope_to_add
  return_path_add
  maildir_format
  group = mail
  mode = 0660
  mode_fail_narrower = false

Discussion:

  • Most options have been discussed above.
  • Both routers set the option domains to localhost because they only operate on local users. As already explained, it is the job of the virtual domains router to convert domain addresses to local users.
  • The option address_data in router_localuser_verify is especially important so that the ACL that performs recipient verification has access to the local part of the address.
  • As opposed to the router discussed above in Routing to a transport, it makes sense to use the option check_local_user for router_localuser. A side effect of this option is that the variables user and home are set to values appropriate for the local user.


Spam filtering

References

The following sections cover the handling of spam and - even more importantly - collateral spam. I use the following excellent documents as the base for my endeavours to eliminate spam from my mailbox:


Policy

The policy I adopt is this:

  • The first front line in battling spam is the SMTP dialog
    • A mail client that does not follow conventions and rules during the SMTP dialog can be regarded with suspicion: It possibly is a Ratware client that wants to foist spam on us
    • If a client lightly violates conventions and rules, it is punished with transaction delays. Legitimate clients should be able and willing to handle these delays, while ratware clients usually do not take the time to handle delays because their objective is to deliver large amounts of email in a short span of time.
    • If the client grossly violates conventions and rules, the message(s) it tries to send are rejected outright. The client is still punished with an initial delay to slow it down.
    • Also, if the connecting host is on any of the DNS blacklists I am consulting, messages it submits are rejected
  • The second front line is greylisting. Statistics have shown that greylisting is highly effective and prevents roughly 90% of all spam messages.
  • The third front line is classification of accepted email by SpamAssassin


Rules used during SMTP dialog

Basically I use the ACLs presented in the paper on tldp.org (see above), with some modifications. The rules that my ACLs apply, listed by SMTP dialog stage, are:

  • When mail client connects
    • If connecting host does not have a verifiable host name = Delay
      • A reverse DNS lookup is made with the IP address of the host that connects. If this fails = Delay
      • A forward DNS lookup is made with the host name from the reverse DNS lookup. If this yields the original IP address, the host name is verifiable and no delay is imposed.
      • If IP addresses differ = Delay
  • HELO/EHLO
    • Host name presented is not a FQDN = Delay
    • Host name presented is plain IP (e.g. 1.2.3.4) = Delay, then deny. All further attempts during the same SMTP connection are rejected even if client tries another HELO with a different host name.
    • Host name presented is IP literal (e.g. [1.2.3.4]) = Delay
    • Host name presented is one of my own domains = Delay, then deny. This works only for domain names, i.e. if a host name presented (e.g. www.herzbube.ch) the result is only a delay.
    • DNS lookups
      • A forward lookup of the host name presented is made: If the resulting IP address matches the IP from where the connection is coming = HELO is ok
      • A reverse lookup with the IP from the forward lookup is made: If the resulting host name matches the host name presented by HELO = HELO is ok
      • If host names differ = Delay
  • MAIL FROM
    • If sender did not provide a HELO/EHLO greeting = Delay, then deny
    • A delay from any of the previous ACLs is repeated
  • RCPT TO
    • If the recipient is not for one of our own domains = Delay, then deny.
    • If the recipient name contains illegal characters or starts with a dot = Deny
    • Sender is <> and recipient is not a valid bounce recipient = Deny. See the section Collateral spam below for details what this is about.
    • Greylist check results in greylisting = Defer
      • Only messages with a non-null sender (MAIL FROM) are checked here, in order to prevent sender callout verification from breaking
    • Greylist check results in blacklisting = Deny
      • This check is performed only if the previous greylist check was also performed
    • A delay from any of the previous ACLs is repeated
  • DATA
    • Header syntax is incorrect = Deny
    • Greylist check results in greylisting = Defer
      • Only messages with null sender (MAIL FROM) are checked here. This complements the greylisting check during the RCPT TO phase.
    • Greylist check results in blacklisting = Deny
      • This check is performed only if the previous greylist check was also performed
    • Invoke SpamAssassin to perform spam classification


I explicitly do not use the following rules/checks/features of the tldp.org ACLs:

  • No sender/callout verification: This seems too expensive for my small site
  • Ditto for recipient/callout verification
  • No check for file $home/.forwarders: Whatever it is, this file is not in use on my system
  • No check for file /etc/mail/whitelist-hosts: I do not (yet) have any hosts that I want to whitelist
  • No SPF checks: See the section Why not SPF? below for more details.
  • No logging of Subject: header
  • No limit for message size
  • No MIME tests, in particular no checks for file extensions
  • No exceptions from spam check just because of a certain message size


Collateral spam

As the paper on tldp.org rightly points out, collateral spam is the most annoying form of spam because it cannot be properly filtered out by SpamAssassin because it is, basically, legitimate mail.


How to identify collateral spam?

  • 90% of the time a collateral spam message is a bounce message: A remote mail server rejects a spam message, e.g. because the message failed a virus check, then notifies my server by sending a bounce message
  • Regular bounce messages have <> as their envelope sender (= MAIL FROM)


An empty envelope sender therefore is our criteria for identifying bounces. The envelope recipient, on the other hand, is useful to distinguish collateral spam from rightfully generated bounces:

  • If the recipient is an address that could have sent a message from my system, the bounce message is valid
  • If the recipient is any other address, the bounce message is considered collateral spam


How to identify recipients that could have sent a message from my system?

  • Create directory /etc/mail/bounce-rcpts
  • In that directory, create a file for every domain hosted on my server
  • In every domain file, list all names that could have sent a message from my system. These are considered "legal bounce recipients".
  • Very important: Every domain file should contain the postmaster entry, because at least sourceforge.net, and possibly others, perform sender/callout verification when they receive a message from your domain, using an empty envelope sender (which qualifies the verification as a possible bounce) and the postmaster account on the domain you are sending from. Why sourceforge.net uses postmaster instead of the user name that you are sending from is not clear to me - they justify their behaviour with this reason: "Several RFCs state that you are required to have a postmaster mailbox for each mail domain." Well, so be it...


Example domain file: 

pelargir:~# cat /etc/mail/bounce-rcpts/moser-naef.ch
postmaster
patrick
francesca
fm


To activate the check for bogus bounces, add the following snippet to whatever ACL is configured for the option acl_smtp_rcpt: 

deny
  message     = This address never sends outgoing mail. \
                You are responding to a forged sender address.
  log_message = bogus bounce for user <$local_part@$domain>
  senders     = : postmaster@*
  domains     = +local_domains
  set acl_m9  = ${extract{1}{=}{${lc:$local_part}}}
  # $domain contains tainted data, so it cannot be used directly to construct
  # the filesystem path of the lookup file in the "condition" statement.
  # The result of the dsearch lookup is untainted, though. The "ret=full"
  # option gives us the full path instead of just the untainted domain.
  set acl_m_legal_bounce_recipients_file = ${lookup {$domain} dsearch,ret=full {/etc/mail/bounce-rcpts}}
  !condition = ${lookup{$acl_m9}lsearch*{$acl_m_legal_bounce_recipients_file}{true}{false}}

Discussion:

  • The senders condition restricts the verb to empty envelope senders, or to postmasters.
  • The $acl_m9 variable is used to store and lookup the local part in lower-case.
  • As explained in the comment, $domain contains tainted data, i.e. data from an untrusted source, which requires an intermediate lookup to "untaint" it.


DNS blacklisting

How does DNS blacklisting work in general?

  • The mail server (Exim in my case) uses the IP address of the connecting host to perform a reverse DNS lookup in the zone of the DNS blacklisting service. For instance, if the connecting host has IP address 1.2.3.4 and the DNS blacklist is dnsbl.sorbs.net, a reverse DNS lookup is made for 4.3.2.1.dnsbl.sorbs.net.
  • The reverse DNS lookup returns 0-n results:
    • No result = The IP address is not listed. Messages from the host can be accepted.
    • 1-n results = The IP address is listed. The mail server either rejects all messages from the host, or marks accepted messages with a mail header for later filtering. If you want to actually evaluate the result values, you have to check the respective DNS blacklist's website to understand the meaning of each value.
  • The reverse DNS lookup may yield an optional TXT record with information about the listing. Usually the information is the URL of an IP address-specific page on the DNS blacklist's website. That page then has an explanation why the IP address got on the blacklist, and/or how the IP address can be de-listed.


DNS blacklists that I am using

A directory of DNS blacklists can be found here.


These are the blacklists that I am using:

  • Barracuda (BRBL)
    • This blacklist is based entirely on mails sent to spamtrap addresses. Its operation is entirely automated. See their methodology.
    • Barracuda trusts the whitelist at emailreg.org. This whitelist costs $20 / per year.
    • Barracuda promises to process removal requests within 12 hours
    • A slight inconvenience is that Barracuda requires registration to use, but on the other hand they provide a valuable service, so this seems acceptable. When you register you must specify which name servers will query the blacklist - in my case I am giving the names of the DNS servers in my /etc/resolv.conf. At the time of writing these are: 146.228.101.20, 146.228.101.28.
    • From their mode of operation, Barracuda looks very trustworthy in my opinion. Because only senders to spamtraps are blacklisted, no innocent mail server will be caught. Innocent users that use the same mail server as a spammer will be affected, but if such a server is well administrated this will only be a temporary condition.
    • Also see the listing on dnsbl.info
  • Spamhaus Zen Blacklist
    • This is an aggregrate of several blacklists maintained by the Spamhaus organization
    • Spamhaus appears to be serious and conscientious. I have seen only good reports about them, so I trust them to a very high degree.
    • For non-commercial use the Spamhaus lists are free to use, under the condition that the site traffic is below certain limits and the DNS blacklist query volume isn't too high. See this page for details.
    • Also see the listing on dnsbl.info


DNS blacklists that I no longer use

  • SORBS
    • SORBS was shut down on 05-Jun-2024
    • The Spam and Open Relay Blocking System (SORBS) project operates a rather large number of blacklists for various purposes
    • SORBS is a classic among the DNS blacklists, so using them shouldn't be too controversial. The reviews I have seen are somewhat mixed, so my trust in SORBS is medium.
    • The FAQ at abuseat.org suggests that only some of the SORBS lists should be used - especially the "dynamic" list should not be used. I follow this recommendation and have activated only the open relay / open proxy lists
    • This page has details about the blacklists maintained by SORBS and which return codes they use
    • Also see the listing on dnsbl.info
  • SpamCop (SCBL)
    • Spamcop is currently disabled because it resulted in too many false positives
    • This blacklist is "powered" by people who actively report spam that they have received on their real-life accounts
    • From the SpamCop website: "The SCBL is aggressive and often errs on the side of blocking mail."
    • Although its cause is honorable, I don't put my full trust in this blacklist
    • Also see the listing on dnsbl.info


ACL configuration

Finally, here is the DNS blacklist configuration that I am using. It appears in the ACL that is configured for the option acl_smtp_rcpt, because we don't want to perform blacklist checks for authenticated SMTP connections. 

  ######################################################################
  # DNS blacklist checks
  ######################################################################
  #
  # If a sending host appears on one DNS blacklist, we stall the sender
  # but do nothing else.
  #
  # If a sending host appears on two DNS blacklists, we deny service.
  # Even though Barracuda and Spamhaus appear very trustworthy, we don't want
  # to give any single DNS blacklist the power to decide what we accept.
  #
  # Notes
  # - Each DNS blacklist is processed in its separate "warn" block
  # - Blocks only differ in the dnslists condition. Blocks can therefore be
  #   freely re-arranged to change the order in which blacklists are consulted.
  # - The first condition in a block checks the threshold. This prevents
  #   unnecessary lookups in case the threshold has already been reached.
  # - If we deny service we are not interested in further communication with
  #   the host. Because here we are in the RCPT TO phase of the SMTP dialog,
  #   we have to use the verb "drop" to close the connection. If we were using
  #   the verb "deny" then the host would be able to send further commands.
  #
  # Why don't we perform DNS blacklist checks in the SMTP connect phase?
  # The underlying reason is that we don't want to perform DNS blacklist
  # checks for authenticated SMTP connections. This means that we have to
  # wait with the checks until we can be sure that authentication has
  # occurred. Authentication should happen right after EHLO, because that's
  # when Exim advertises its "AUTH ..." capabilities. But since we don't know
  # exactly how various SMTP clients are implemented, we have to wait a little
  # bit longer: A legitimate client MUST authenticate before RCPT TO, because
  # RCPT TO is the phase where the client reveals that it wants to submit a
  # message that requires relaying. Because the server will deny a relaying
  # request for unauthenticated clients, authentication is guaranteed to
  # happen before RCPT TO. Because an ACL is executed only AFTER the
  # corresponding phase in the SMTP dialog, we must place DNS blacklist
  # checks in the ACL that is executed after RCPT TO (the one that is
  # configured for acl_smtp_rcpt). In addition we must place the checks
  # after the "accept" verb that checks for authenticated connections.

  warn
    set acl_m_dnslist_matchcount = 0
    set acl_m_dnslist_threshold = 2

  warn
    condition = ${if < {$acl_m_dnslist_matchcount}{$acl_m_dnslist_threshold}}
    dnslists = b.barracudacentral.org
    set acl_m_dnslist_matchcount = ${eval:$acl_m_dnslist_matchcount + 1}
    set acl_m_dnslist_domain1 = ${if def:acl_m_dnslist_domain1 {$acl_m_dnslist_domain1} {$dnslist_domain}}
    set acl_m_dnslist_domain2 = $dnslist_domain
    set acl_m_dnslist_text1 = ${if def:acl_m_dnslist_text1 {$acl_m_dnslist_text1} {$dnslist_text}}
    set acl_m_dnslist_text2 = $dnslist_text

  warn
    condition = ${if < {$acl_m_dnslist_matchcount}{$acl_m_dnslist_threshold}}
    dnslists = zen.spamhaus.org
    set acl_m_dnslist_matchcount = ${eval:$acl_m_dnslist_matchcount + 1}
    set acl_m_dnslist_domain1 = ${if def:acl_m_dnslist_domain1 {$acl_m_dnslist_domain1} {$dnslist_domain}}
    set acl_m_dnslist_domain2 = $dnslist_domain
    set acl_m_dnslist_text1 = ${if def:acl_m_dnslist_text1 {$acl_m_dnslist_text1} {$dnslist_text}}
    set acl_m_dnslist_text2 = $dnslist_text

# SORBS was shut down on 05-Jun-2024
#  warn
#    condition = ${if < {$acl_m_dnslist_matchcount}{$acl_m_dnslist_threshold}}
#    dnslists = smtp.dnsbl.sorbs.net : http.dnsbl.sorbs.net : socks.dnsbl.sorbs.net : misc.dnsbl.sorbs.net
#    set acl_m_dnslist_matchcount = ${eval:$acl_m_dnslist_matchcount + 1}
#    set acl_m_dnslist_domain1 = ${if def:acl_m_dnslist_domain1 {$acl_m_dnslist_domain1} {$dnslist_domain}}
#    set acl_m_dnslist_domain2 = $dnslist_domain
#    set acl_m_dnslist_text1 = ${if def:acl_m_dnslist_text1 {$acl_m_dnslist_text1} {$dnslist_text}}
#    set acl_m_dnslist_text2 = $dnslist_text

# Spamcop is currently disabled because it resulted in too many false positives
#  warn
#    condition = ${if < {$acl_m_dnslist_matchcount}{$acl_m_dnslist_threshold}}
#    dnslists = bl.spamcop.net
#    set acl_m_dnslist_matchcount = ${eval:$acl_m_dnslist_matchcount + 1}
#    set acl_m_dnslist_domain1 = ${if def:acl_m_dnslist_domain1 {$acl_m_dnslist_domain1} {$dnslist_domain}}
#    set acl_m_dnslist_domain2 = $dnslist_domain
#    set acl_m_dnslist_text1 = ${if def:acl_m_dnslist_text1 {$acl_m_dnslist_text1} {$dnslist_text}}
#    set acl_m_dnslist_text2 = $dnslist_text

  drop
    condition = ${if >= {$acl_m_dnslist_matchcount}{$acl_m_dnslist_threshold}}
    message = Access denied - $sender_host_address is listed by two DNS blacklists\n1) $acl_m_dnslist_domain1\n$acl_m_dnslist_text1\n2) $acl_m_dnslist_domain2\n$acl_m_dnslist_text2
    log_message = $sender_host_address is blacklisted by $acl_m_dnslist_domain1 and $acl_m_dnslist_domain2
    delay = 20s

  warn
    condition = ${if > {$acl_m_dnslist_matchcount}{0}}
    add_header = X-DNSbl-Warning: $sender_host_address is blacklisted by $acl_m_dnslist_domain1
    delay = 20s


.forward file configuration

In the user's forward file I am adding the following check to react to the X-DNSbl-Warning header that is added when only a single DNS blacklist has the sender listed. The check appears after the usual spam filtering checks, which means that the "DNSbl-Warning" mailbox receives only mails that are not classified as spam. It will be interesting to see whether such mails turn out to be actual ham, or mostly false negatives. 

elif "${if def:h_X-DNSbl-Warning {def}{undef}}" is "def" then
  save Maildir/.Junk.DNSbl-Warning/


Testing

Apparently an IP address that is widely used to test DNS blacklists is 127.0.0.2 - I found out about this on the Barracuda website, and it is confirmed by this Spamhaus page. To test a DNS blacklist on the command line you can use the host utility like this 

host 2.0.0.127.some.blacklist.net

Obviously you have to replace "some.blacklist.net" with the actual name of the DNS blacklist you want to test. For instance, with Barracuda the test looks like this: 

host 2.0.0.127.b.barracudacentral.org

To test whether your Exim ACLs correctly query a blacklist, run this command to simulate an SMTP connection. You specify the IP address of the connecting host. 

exim4 -bh 127.0.0.2


Greylisting

How it works

The basic concept of greylisting is this:

  • The greylisting mechanism identifies a message delivery by the following information triplet:
    • The IP address of the host attempting the delivery
    • The envelope sender address
    • The envelope recipient address
  • If our server has already whitelisted a triplet, it will not perform greylisting but accept the message immediately. The triplet's whitelist lifetime is extended.
  • If our server has not whitelisted a triplet, it will temporarily reject the message (with a 451 SMTP response), and put the triplet on a "greylist", together with a "greylist delay" time and a "greylist timeout" time
  • The sender, if it is legitimate, is expected to retry message delivery within a certain time frame bounded by the greylist delay and greylist timeout
  • If the sender retries message delivery before the greylist delay has passed, the message is temporarily rejected again
  • If the sender does not retry message delivery before the greylist timeout occurs, the triplet is removed from the greylist
  • If the sender retries message delivery after the greylist delay has passed, but before the greylist timeout occurs, the message will be accepted. The willingness to spend time and retry message delivery marks the sender/message as legitimate. In addition to the message being accepted, the triplet is also moved from the greylist to the whitelist
  • This means that next time when the same sender attempts message delivery with the same triplet, the sender will not be punished again by the greylisting mechanism
  • On the other hand, the sender must undergo greylisting again if it retries too late, or if it tries an altogether new message delivery with a different triplet
  • It is therefore crucial to choose a greylist timeout that is large enough for all legitimate senders. Because senders decide after how much time they will retry message delivery, choosing a suitable greylist timeout may be tricky. In general, choosing a higher timeout makes it less likely that a message will be permanently rejected - but we increase the chance for successful delivery both for legitimate and spam messages. The main effect of greylisting, though, is expected to come from the initial temporary reject and subsequent delay, so choosing a safe-and-high timeout is vastly preferrable to choosing a risky-and-low timeout. Another effect of having a high timeout is that triplets remain longer on the greylist, therefore the greylist in total grows larger. On busy sites this might be a factor that helps deciding for a lower timeout, but on my system I prefer to play it safe
  • Last but not least: Triplets that are on the whitelist also have a lifetime. That lifetime is set when the triplet is moved to the whitelist for the first time, and extended whenever another message matching the triplet is received. If a triplet's lifetime has expired, it is removed from the whitelist and messages from that sender again become subject to greylisting


Installation

Install the Debian package 

greylistd

This installs a Python script that acts as a simple greylisting daemon that can be integrated into Exim.


Configuration

Configuration files are located in 

/etc/greylistd


The default configuration (in /etc/greylistd/config) is less strict than what is suggested in the Greylisting whitepaper, but is still quite acceptable. It looks like this:

  • greylist delay = 10 minutes
  • greylist timeout = 8 hours
  • whitelist lifetime = 60 days


For a few years I have lived with the default 10 minutes delay time, but recently I have relaxed the delay to only 1 minute. The reason is simple: Delays are always inconvenient, and people (myself included) simply expect email communication to be quasi-instantaneous. Well-behaved MTAs often retry within a couple of minutes, so a one minute delay usually doesn't hurt much and is much simpler to explain away :-) than 10 minutes. So this is the single change I am making to the default configuration: 

[timeouts]
retryMin     = 60


Whitelisting

A list of sending hosts that should be whitelisted by default can be defined in 

/etc/greylistd/whitelist-hosts

Important: The file is not read by greylistd, but referenced from the configuration of the invoking MTA, i.e. Exim. However, you do not have to restart the Exim daemon if you make changes to this file.


Whitelisting is useful - one might even say mandatory - for cases where the email sender is using Amazon's SES, or a similarly configured MTA infrastructure. Amazon's SES consists of a large number of sending MTAs, and once a message has been delayed by our greylisting service it is not at all guaranteed that the next delivery attempt will be made by the same MTA. Without a whitelisting solution, it is very conceivable that a message might never be delivered because it will be delayed again and again until our greylist timeout kicks in. Even if the message is eventually delivered, the delay usually is several hours instead of just a couple of minutes. This is especially grave when the message content is time sensitive - for instance a password reset URL or a login verification code or something similar.


Here is the whitelist I am using: 

*.amazonses.com


Integration with Exim

Note: In theory there is a Python script greylistd-setup-exim4 that can be invoked as root to add greylistd integration to the Exim configuration. Since I have already significantly tweaked Exim's configuration, I don't really trust this script to work properly. Also, when installing the greylistd package Debconf tells me that I may need to re-run the script if the Exim package is upgraded - this is a pretty sure sign that the intended automated integration is bound to be very fragile. Instead of relying on automation, I therefore prefer to understand how integration is supposed to work, and then manually add integration.


Documentation:

  • /usr/share/doc/greylistd/README.Debian
  • File that contains the proposed ACLs: /usr/sbin/greylistd-setup-exim4
  • Paper on tldp.org


RCPT TO

  • Greylisting happens during the RCPT TO phase of the SMTP dialog, i.e. the Exim configuration must be added to whatever ACL is configured for the option acl_smtp_rcpt
  • I add two verbs to the ACL, after all the other spam-related checks have happend, and just before a message is finally accepted
  • The first verb is defer and checks whether a triplet is greylisted
  • The second verb is deny and checks whether a triplet is blacklisted. Note: This refers to blacklisting performed by greylistd, it has nothing to do with DNS blacklists.
  • The blacklist check occurs only if a greylist check has occurred before
  • The greylist check is skipped if certain conditions apply, e.g. the sending host is permanently whitelisted, the SMTP session is authenticated, or the message is a bounce message (in the latter case, the greylist check is only deferred until later, but not skipped altogether - see DATA phase below)


Here is the verbatim configuration snippet that I add at the bottom of the ACL: 

  # --------------------------------------------------------------------
  # Check greylisting status for this particular peer/sender/recipient.
  # Before uncommenting this statement, you need to install "greylistd".
  # See:  http://packages.debian.org/unstable/main/greylistd
  #
  # Note that we do not greylist messages with NULL sender, because
  # sender callout verification would break (and we might not be able
  # to send mail to a host that performs callouts).
  defer
    message        = $sender_host_address is not yet authorized to deliver mail \
                     from <$sender_address> to <$local_part@$domain>. \
                     Please try later.
    log_message    = greylisted.
    !senders       = :
    !hosts         = : +relay_from_hosts : \
                     ${if exists {/etc/greylistd/whitelist-hosts} \
                                 {/etc/greylistd/whitelist-hosts}{}} : \
                     ${if exists {/var/lib/greylistd/whitelist-hosts} \
                                 {/var/lib/greylistd/whitelist-hosts}{}}
    !authenticated = *
    domains        = +local_domains : +relay_to_domains
    # Request string, i.e. the string written to the socket. greylistd will
    # respond with one of the following, depending on how the lookup
    # triple is currently classified: "white", "grey", "black". See greylistd
    # man page for alternative requests/responses.
    set acl_m9     = $sender_host_address $sender_address $local_part@$domain
    # Write to/read from socket; timeout = 5 seconds. Store the result of the
    # lookup in a variable that can be queried by the next verb (will check
    # for blacklisting)
    set acl_m6     = ${readsocket{/var/run/greylistd/socket}{$acl_m9}{5s}{}{}}
    # Check for greylisting. The blacklist check happens in the next verb.
    condition      = ${if eq {$acl_m6}{grey}{true}{false}}
  # --------------------------------------------------------------------

  # --------------------------------------------------------------------
  # Deny if blacklisted by greylist.
  #
  # We rely on acl_m6 having been filled by the previous verb with the result
  # of the greylistd lookup. At this point, acl_m6 may contain either "black",
  # or "white", or it may be undefined. The latter is the case if the previous
  # verb has not performed a lookup, e.g. because the host has been whitelisted,
  # in which case we also do not have to check for blacklisting.
  deny
    message     = $sender_host_address is blacklisted from delivering \
                  mail from <$sender_address> to <$local_part@$domain>.
    log_message = blacklisted.
    condition   = ${if and {{def:acl_m6}{eq{$acl_m6}{black}}} {true}{false}}
  # --------------------------------------------------------------------


DATA

  • If a message is a bounce message, the RCPT TO ACL defers the greylist and blacklist checks. If the RCPT TO ACL did not defer, we might not be able to send mail to hosts that perform sender callout verification, because that verification would fail when our RCPT TO ACL would apply greylisting.
  • For this reason, greylist and blacklist checks for bounce messages are performed during the DATA phase of the SMTP dialog. Only real bounces will finish the DATA phase, while sender callout verification (which appears as a bounce because it uses an empty envelope sender) does not.
  • The greylist and blacklist checks occur in the same fashion as during RCPT TO
  • The checks occur before SpamAssassin is invoked
  # --------------------------------------------------------------------
  # Perform greylisting on messages with no envelope sender here.
  # We did not subject these to greylisting after RCPT TO: because
  # that would interfere with remote hosts doing sender callouts.
  # Note that the sender address is empty, so we don't bother using it.
  #
  # Please refer to 63_exim4-config_pelargir_rcpt_to for detailed comments.
  #
  # Please note that in contrast to the RCPT ACL we cannot use the "domains"
  # condition here. Reason: Exim allows that condition only in an RCPT ACL,
  # but not in a DATA ACL. We therefore rely on the RCPT ACL having made
  # sufficient checks so that the SMTP dialog only progresses to here for
  # valid recipient domains.
  defer
    message        = $sender_host_address is not yet authorized to send \
                     delivery status reports to <$recipients>. \
                     Please try later.
    log_message    = greylisted.
    senders        = :
    !hosts         = : +relay_from_hosts : \
                     ${if exists {/etc/greylistd/whitelist-hosts} \
                                 {/etc/greylistd/whitelist-hosts}{}} : \
                     ${if exists {/var/lib/greylistd/whitelist-hosts} \
                                 {/var/lib/greylistd/whitelist-hosts}{}}
    !authenticated = *
    set acl_m9     = $sender_host_address $recipients
    set acl_m6     = ${readsocket{/var/run/greylistd/socket}{$acl_m9}{5s}{}{}}
    condition      = ${if eq {$acl_m6}{grey}{true}{false}}


  deny
    message     = $sender_host_address is blacklisted from sending \
                  delivery status reports to <$recipients>.
    log_message = blacklisted.
    condition   = ${if and {{def:acl_m6}{eq{$acl_m6}{black}}} {true}{false}}
  # --------------------------------------------------------------------


Manually feeding greylistd

It is possible to manually feed the greylist daemon using the command 

greylist

Read the command's man page for details about how to add, remove or manipulate entries on the greylist or whitelist.


SpamAssassin

See SpamAssassin


SPF & Co

Why not SPF?

Some people are quite radical in their negative opinion about SPF (Sender Policy Framework). Although I do not want to follow these people by saying SPF is completely useless, I must admit that I do not see how SPF can ever be successful.

In order for SPF to work 100%, all email participants in the entire Internet must support SPF:

  1. all domain holders must publish SPF record
  2. mail servers must check SPF records when they receive messages
  3. mail servers must implement SRS (Sender Rewriting Scheme) when they are forwarding messages

What happens if I support SPF, but others do not?

  • If someone does not follow 1 and/or 2, this is not too grave - it is their problem, not mine, if they receive a lot of spam.
  • If someone does not follow 3, however, this has consequences for me: if someone forwards a message that I have sent, without implementing SRS, and the next receiver performs an SPF check, the check will fail and my message will be rejected

Since I have no control about who does, or does not, forward my messages, I must assume the worst: that potentially every other message of mine will run into the SRS trap! In other words, if I support SPF I damage myself!!!


SPF overview

Links:

Tests:


An SPF record contains n mechanisms:

  • mechanisms define which hosts are allowed to send mail for a domain
  • if a mechanism matches, its prefix is evaluated in order to determine the result of the SPF query
  • a dash ("-") represents "fail" and is therefore the most restrictive prefix
    • for instance, if an SPF record ends with -all this means: if a check did not succeed with any of the previous mechanisms, it now fails


Suggested SPF record for herzbube.ch and other domains: 

v=spf1 a include:solnet.ch -all

Discussion:

  • the record is a TXT record
  • a = take the domain from the "MAIL FROM" address and lookup all A records of that domain in DNS; the SPF check succeeds if the IP address of the host that initiated the SMTP session equals one of the IP addresses returned by the A records
  • include:solnet.ch = also perform an SPF check with the SPF record for solnet.ch
  • -all = the SPF check fails for all remaining IP addresses


For comparison, the SPF record for solnet.ch: 

v=spf1 ip4:212.101.0.0/22 ip4:212.101.4.0/24 ?all

Discussion:

  • the SPF check succeeds if the IP address of the host that initiated the SMTP session is within the named networks
  • ?all = the SPF check result is neutral for all remaining IP addresses


Roles in the SPF scheme

Sender of email messages:

  • I must publish an SPF record
  • the same record can be used for all domains that I manage
  • legitimate MTAs for my domains are located on one of the following hosts:
    • pelargir.herzbube.ch
    • mail.solnet.ch

Forwarder of email messages:

  • if recipient is a local user there is no problem
  • if recipient is a mailing list: contemporary mailing lists (e.g. mailman) are supposed to already rewrite the envelope sender (MAIL FROM); I have not verified this information yet
  • everything coming in for ludenti.ch needs to be passed through SRS (unless it results in a local address)

Receiver of email messages:

  • I must check the SPF record


SRS overview

The forwarding party (e.g. ludenti.ch) modifies the envelope sender address (MAIL FROM) by adding its own domain to the original sender address. For instance: 

SRS0=HHH=TT=source.com=sourceuser@ludenti.ch

If the new receiver produces a bounce, the bounce is sent to the forwarder (ludenti.ch), who in turn extracts the original envelope sender and re-bounces back to the original sender.

If the new receiver, or any future receiver, forwards again, the envelope sender address is rewritten so that it points back to the original forwarder (ludenti.ch). A bounce is therefore not bounced back the entire forwarder chain, instead the bouncing party bounces to the original forwarder, which in turn bounces to the original sender. For instance: 

SRS1=HHH=ludenti.ch==HHH=TT=source.com=sourceuser@foobar.com


SPF and Exim

SPF checks must be done during the "MAIL FROM" phase of the SMTP dialog. Unfortunately, the exim4-daemon-heavy package does not contain the native ACL condition spf. Because of this it is necessary to use the perl module Mail::SPF::Query. The package provides the daemon spfd, although without an init script.


DomainKeys

Links:


The DomainKeys scheme is based on a public/private key pair. The sending server has the private key, which it uses to sign the entire message, including all its headers.

The receiving server fetches the public key via DNS. It fetches the key for the domain that appears in the header From: and/or Sender:. Because DomainKeys does not work with envelope addresses, message forwarding is no problem (unless the forwarding server would rewrite the mail headers From: and/or Sender:).

Unfortunately, no changes may be made to any headers after the sending server has signed the message. This at least causes problems with mailing lists.

If somebody still wants to make changes to headers, the modifying agent must sign the entire message again, thus appearing as the new "author" of the message.


Smarthost

It may become necessary to temporarily delegate relaying for all outgoing mail to a smart host. In my case this was the case when http://www.sorbl.com/ decided to blacklist an entire range of IP addresses that belong to my ISP, among which was my own fixed IP. sorbl.com was incredibly slow in removing the IP addresses, which forced my ISP to provide a fix that required relaying through its own mail server mail.solnet.ch.

The immediate approach was to update /etc/exim4/update-exim4.conf.conf and to set the following line: 

dc_smarthost='mail.solnet.ch'

This defines the smart host, but the router that actually uses the smart host is still de-activated by an ifdef: 

.ifdef DCconfig_smarthost DCconfig_satellite
smarthost:
  driver = manualroute
  [...]
.endif

In order to activate the router it would be necessary to update update-exim4.conf.conf and to change 

dc_eximconfig_configtype='internet'

to 

dc_eximconfig_configtype='smarthost'

I did not want to do this because this setting has far-reaching consequences. I therefore decided to copy the provided smarthost router to a separate file whose content can be commented/uncommented as I need to disable/enable the smart host solution.

The original router is named smarthost and can be found in /etc/exim4/conf.d/router/200_exim4-config_primary. I copied it to the file /etc/exim4/conf.d/router/099_exim4-config_pelargir_smarthost. 

# Send all non-local mail to a single other machine (smarthost).
smarthost:
  debug_print = "R: smarthost for $local_part@$domain"
  driver = manualroute
  domains = ! +local_domains
  transport = remote_smtp_smarthost
  route_list = * DCsmarthost byname
  host_find_failed = defer
  same_domain_copy_routing = yes
  no_more


SMTP Authentication

Overview

By default the mail server does not permit relaying, which means that the server only accepts messages that will be delivered locally. The mail server can be configured, though, to allow relaying if a connecting host meets certain criteria. For instance, in a local network certain clients with a known IP address might be allowed to submit messages for relaying. The typical case, though, is that clients must authenticate via username/password. This is what I am going to use in my configuration.


Use cases:

  • Locally submitted messages must be relayed without SMTP authentication. The typical client is the webmail client.
  • I want to be able to submit messages both from the inside and from the outside of my protected Intranet environment (e.g. from Apple Mail that is installed on a MacBook, or from a mobile device while I am on the road). These connections must use SMTP authentication.
  • My Fritz!Box ADSL router must be able to send voicemail recordings by email. These connections must use SMTP authentication.


The next few sections cover

  1. How to configure the SASL plaintext authentication mechanism
  2. How to encrypt the SMTP session with TLS so that the transmitted credentials are not visible in the clear
  3. How to enforce the use of both TLS and authentication for relaying


References

/usr/share/doc/exim4-base/README.Debian.gz
Explains the Debian way how to configure TLS and plain authentication in Exim
/etc/exim4/conf.d/main/03_exim4-config_tlsoptions
This file should not be changed, but it is a good reference for checking how Debian's Exim configuration is supposed to work
Chapter 41 of the Exim specs
Has all the details about TLS
Chapter 33 of the Exim specs
Has the basics of SMTP authentication
Chapter 34 of the Exim specs
Has the details for the plaintext authenticator


Certificate for TLS encryption

The plan is for Exim to be accessible under smtp.herzbube.ch. Creating a new key and certificate for smtp.herzbube.ch is not necessary, I already have a wildcard certificate for *.herzbube.ch which I am going to reuse.


Special attention is needed to provide the Exim daemon with access to the private key file. Unlike the Apache daemon who starts as user root and then changes its user ID to something else, the Exim daemon immediately starts as user Debian-exim. This means that normally Exim does not have access to the private key file in /etc/letsencrypt/live/herzbube.ch. The following modifications therefore need to be made to give Exim access: 

chgrp ssl-cert /etc/letsencrypt/live
chgrp ssl-cert /etc/letsencrypt/archive
chmod 710 /etc/letsencrypt/live
chmod 710 /etc/letsencrypt/archive
adduser Debian-exim ssl-cert

Discussion:

  • Other daemons than Exim may also have similar requirements, so we don't want to modify the "Let's Encrypt" folders with an Exim-specific user or group. Instead we rely on the scheme that Debian developers have figured out for restricting access to /etc/ssl/private, which is by granting access to a special group ssl-cert (see the SSLCertificatesOnDebian page for details). We therefore modify the "Let's Encrypt" folders to be owned by group "ssl-cert" and to give that group permission to access files within the folders. The live folder must be modified because that's where the Exim configuration points to. The archive folder must be modified because the live folder contains symlinks that point back to the archive folder.
  • The Debian-exim user must be added to the ssl-cert group so that it has the permissions mentioned above


Configuring TLS

The main file that contains TLS configuration options is 

/etc/exim4/conf.d/main/03_exim4-config_tlsoptions

However, this file should not be modified directly. Instead its behaviour should be modified by macros (Debian-specific) placed in a file that is guaranteed to be processed before the TLS configuration file. On my server, I place everything into 

/etc/exim4/conf.d/main/00_exim4-config_pelargir_tlsoptions

This is the content of the file: 

# 00_exim4-config_pelargir_tlsoptions

# Generally enable TLS. This causes Exim to advertise STARTTLS
# when connections are made on the normal SMTP port.
MAIN_TLS_ENABLE = 1

# MAIN_TLS_ADVERTISE_HOSTS could be used to restrict advertising
# of STARTTLS to certain hosts only. Leaving this macro undefined
# causes it to be set to its default value "*", meaning all hosts
# will get the advertisement.
# Note: For unknown reasons, it is not possible to use here the
# variable relay_from_hosts, nor the macro MAIN_RELAY_NETS, even if
# this file is processed after the variable/macro have been defined.
#MAIN_TLS_ADVERTISE_HOSTS = ! 192.168.0.0/16

# Path to the certificate, or certificate chain. If this is not set,
# the default location is /etc/exim4/exim.crt. It is possible to use
# the string "tls_sni" as part of this macro (or better: as part of
# the Exim option tls_certificate). Exim will expand the string to
# whatever hostname it received from the client via SNI, with the
# purpose to present different certificates when it is contacted under
# different host names. In this configuration this is not necessary,
# since clients are expected to always connect this server under
# smtp.herzbube.ch.
MAIN_TLS_CERTIFICATE = /etc/letsencrypt/live/herzbube.ch/fullchain.pem

# Path to the unencrypted private key. If this is not set, the
# default location is /etc/exim4/exim.key. The key file must be kept
# 'secret' and should be owned by root.ssl-cert and have mode 640.
# User Debian-exim must be added to group ssl-cert to be able to
# 1) look into /etc/ssl/private and 2) read the key file.
MAIN_TLS_PRIVATEKEY = /etc/letsencrypt/live/herzbube.ch/privkey.pem

# MAIN_TLS_TRY_VERIFY_HOSTS can be set to a list of hosts from whom
# Exim should attempt to obtain a certificate for authentication.
# If the host does not present a certificate Exim will proceed as
# normal. MAIN_TLS_VERIFY_HOSTS can be set to a list of hosts who
# MUST present a certificate.
#
# If a client certificate is presented, it is checked against the
# certificates in the file defined by the macro MAIN_TLS_VERIFY_CERTIFICATES.
# If the macro is not set, the default certificate file is used
# (/etc/ssl/certs/ca-certificates.crt). This, however, is a large file
# with many certs which is reported to cause problems with Outlook and
# other popular mail clients.

# Log the cipher that Exim and the client negotiated. Also log the
# distinguished name of any certificates used (either when Exim
# connects to another server, or when a client connects to Exim).
# Finally, log the hostname received from the client via SNI.
.ifdef MAIN_LOG_SELECTOR
MAIN_LOG_SELECTOR = MAIN_LOG_SELECTOR +tls_cipher +tls_peerdn +tls_sni
.else
MAIN_LOG_SELECTOR = +tls_cipher +tls_peerdn +tls_sni
.endif

Notes:

  • Reportedly some clients (apparently many versions of Outlook) do not support STARTTLS (i.e. connect unencrypted to port 25, then upgrade the session to encrypted), instead they want to connect to port 465 where they expect to immediately start an encrypted session (so-called SSMTP, or SMTPS). The README.Debian.gz file (see the references section above) explains how to configure Exim to properly support this scenario. Currently I am not supporting such broken clients.


Configuring authentication

The SASL plaintext authentication mechanism is configured in this file: 

/etc/exim4/conf.d/auth/50_exim4-config_pelargir


Change the file's permissions because it contains a password 

root@pelargir:~# ls -l /etc/exim4/conf.d/auth/50_exim4-config_pelargir 
-rw------- 1 root root 1445 Jul 14 02:13 /etc/exim4/conf.d/auth/50_exim4-config_pelargir


This is the content of the file: 

# 50_exim4-config_pelargir

plain_server:
  driver = plaintext
  # The PLAIN authentication mechanism is defined in RFC 2595.
  # The driver also supports the LOGIN authentication mechanism,
  # for which there is no specification. At the moment this
  # configuration does not support LOGIN.
  public_name = PLAIN
  # This condition is used to restrict the authentication mechanism to TLS
  # connections only. Because we use the "server" prefix, Exim knows that
  # this should be used only if it acts as a server.
  .ifndef AUTH_SERVER_ALLOW_NOTLS_PASSWORDS
  server_advertise_condition = ${if eq{$tls_cipher}{}{no}{yes}}
  .endif
  # An empty prompt that is sent to the client if it does not supply
  # credentials with the AUTH command. The client is expected to supply
  # credentials in response to the prompt. The prompt is not sent if the
  # client already sends credentials with the AUTH command.
  server_prompts = :
  # $auth2 contains the user name, $auth3 contains the password, $auth1 is
  # always empty. Read the docs for the plaintext authenticator for information
  # why this is so. Using the credentials thus obtained, we can perform a
  # lookup to see if the credentials were correct. Note that the expression
  # must guard against non-existing user names!
  server_condition = ${if and {{eq{$auth2}{username}}{eq{$auth3}{secret}}}}
  # This sets $authenticated_id for further use.
  server_set_id = $auth2

Notes:

  • Obviously username and password have been omitted from the configuration snippet above.


In addition to the above, we need a little tweak in another global configuration file 

/etc/exim4/conf.d/main/00_exim4-config_pelargir_auth

This is the content of the file: 

# 00_exim4-config_pelargir_auth

# This option can be used to restrict advertising of auth mechanisms
# to certain clients only. As an include-mechanism it is useless in this
# configuration since we need to support mobile devices that can connect
# from anywhere in the world. However, it can be cleverly used as an
# exclude-mechanism: We are *NOT* advertising the mechanism to clients
# connecting from the Intranet.
# Note 1: The Exim docs imply that this option belongs to auth driver
# definitions, but in reality it is a global option.
# Note 2: For unknown reasons, it is not possible to use here the
# variable relay_from_hosts, nor the macro MAIN_RELAY_NETS, even if
# this file is processed after the variable/macro have been defined.
#auth_advertise_hosts = ! 192.168.0.0/16

Outdated notes which I like to keep around for historical reasons. The notes are outdated because pelargir is now a Dedicated Server and no longer resides in my intranet.

  • I advertise the authentication mechanism even to intranet clients (auth_advertise_hosts remains undefined). At first I wanted SMTP authentication to be available only from the extranet, but it's more difficult to configure a device such as my iPhone that wants to connect from both intranet and extranet.


Enforcing TLS and authentication

In order to enforce TLS and authentication we need to add the following to the "rcpt_to" ACL: 

  # Accept if the message arrived over an authenticated connection,
  # from any host. Again, these messages are usually from MUAs, so
  # recipient verification is omitted.
  accept
    authenticated = *
    logwrite      = accepted for user $authenticated_id

(this currently resides in acl/63_exim4-config_pelargir_rcpt_to)


Notes:

  • The snippet must appear somewhere at the beginning of the ACL, together with the relay check
  • A similar "accept" verb must appear in every ACL where we would otherwise deny relaying to external clients. Currently this is the case in whatever ACLs are set for the acl_smtp_mail and acl_smtp_data options
  • Enforcing authentication automatically also enforces TLS, since further up we have configured authentication to be advertised only for TLS sessions


Storing credentials in LDAP

Currently I don't know how to configure Exim so that it can both bind to and query the LDAP directory. This doesn't even need to be in the same configuration statement, I just have to overcome the fact that my LDAP directory does not allow anonymous queries.


Binding alone with the user specified for SMTP authentication is not sufficient since there are all sorts of users in the directory, and I don't want them all to be able to relay mail. So I would need to make a check before or after the bind, e.g. check if the DN has a certain object class...


This presentation contains a lot of spiffy Exim/LDAP stuff, maybe I can get inspiration from here.


Using openssl to test both TLS and authentication

Connect to the Exim server with TLS: 

openssl s_client -connect smtp.herzbube.ch:25 -starttls smtp

Note that this is a real connection to a live server! With TLS, simulating a connection is not possible. If you want to simulate a connection via exim -bh 1.2.3.4 you have to temporarily re-configure Exim to allow SMTP authentication over unencrypted connections.


Next, issue the "EHLO" command. If you have connected via TLS then you will probably have to wait 20 seconds before you see a response because the server is delaying you due to "remote host presented unverifiable HELO/EHLO greeting".


The response to the "EHLO" command should advertise the "AUTH PLAIN" mechanism. You are now ready to authenticate. To do so, you first have to Base64-encode both username and password like this (see this StackOverflow answer): 

echo -en "\0username\0secret" | base64

The SMTP session snippet then looks like this: 

AUTH PLAIN
334 
AHVzZXJuYW1lAHNlY3JldA==
235 Authentication succeeded


Using GnuTLS to test the TLS handshake

TODO Seeing as I have a complete test scenario in the previous section, this section seems a bit superfluous. Check if maybe GnuTLS is more flexible than OpenSSL when it comes to delaying the TLS handshake. 

gnutls-cli --crlf --insecure --starttls --port 25 smtp.herzbube.ch
[...]
EHLO herzbube.ch
[...]
STARTTLS
[...]
Press Ctrl+D
[...]  <----- here gnutls-cli prints information about the TLS handshake
QUIT


Steps when adding a new domain

Manually update the file /etc/exim4/update-exim4.conf.conf and set the variable 

dc_other_hostnames='herzbube.ch ; moser-naef.ch ; francescamoser.ch ; grunzwanzling.ch'

If the "Virtual domains" solution presented above is active:

  • Create a new file named like the domain in /etc/mail/virtual
    • The file contains the mapping of message recipients to local users
    • The file should contain at least one line looking like this: *: patrick@localhost
  • Create a new file named like the domain in /etc/mail/bounce-rcpts
    • The file contains a list of message recipients for which bounce message are allowed
    • The file must contain at least one line with recipient "postmaster" so that sender/callout verification by receiving MTAs will succeed

If the "Virtual domains" solution is not active:

  • Create a new verify router
  • Create a new spamcheck router
  • Create a new spamcheck transport
  • Create a new "smartuser" router
  • Create a new "smartuser" transport

After all modifications have been made, execute the commands 

update-exim4.conf
/etc/init.d/exim4 restart


Steps when adding a new user

TBD


Mail administration

Thaw frozen messages (message_id can be looked up in /var/spool/exim/msglog): 

exim4 -Mt <message_id>

List all messages in the mail queue: 

exim -bp

Remove messages from the queue: 

exim -Mrm <message_id>


Upgrade procedure

At the time of writing, the following packages are involved in an upgrade of Exim:

  • exim4
  • exim4-base
  • exim4-config
  • exim4-daemon-heavy

The upgrade procedure works as follows:

  • NO LONGER APPLIES, CHECK WITH NEW PROVIDER IF SOMETHING SIMILAR IS POSSIBLE If paranoid about a change, create a new backup MX DNS record a few hours or a day before the upgrade is planned. The MX record should point to virusscan.solnet.ch and have a rather low priority, e.g. 20.
  • Use aptitude to upgrade the packages. Warning: This will automatically and immediately restart the Exim daemon!
  • If something goes wrong, it is still possible to add the backup MX DNS record
  • Then try to fix the errors in the config files
  • Run update-exim4.conf to generate an up-to-date config file /var/lib/exim4/config.autogenerated
  • The server does not need to be (re)started, the following command can be used for a dry-run:
exim4 -bh 212.101.18.224
  • The above command invokes a pseudo-MTA that uses the generated config file, then it simulates a connection from IP address 212.101.18.224
  • Obviously, don't forget to remove any unnecessary MX records after a successful upgrade


Only outgoing mail from local accounts

A server system does not need to receive mails, nor does it need to relay mails for others, but it needs to be able send mail. In effect this means that the SMTP port is closed on all external interfaces, but open on 127.0.0.1 so that local processes can submit mail.


After installing Exim, no mail can be sent or received. The "send" part can be tested by sending a mail on the command line (using the mail command line utility) and then looking at the log file. If mails cannot be sent it will look something like this. Note the "Mailing to remote domains not supported" message. 

root@pelargir:/var/log# tail exim4/mainlog
2016-06-17 15:28:49 Start queue run: pid=47102
2016-06-17 15:28:49 End queue run: pid=47102
2016-06-17 15:58:49 Start queue run: pid=47614
2016-06-17 15:58:49 End queue run: pid=47614
2016-06-17 16:15:13 1bDuYD-000Cb7-8g <= root@pelargir.herzbube.ch U=root P=local S=386
2016-06-17 16:15:13 1bDuYD-000Cb7-8g ** herzbube@herzbube.ch R=nonlocal: Mailing to remote domains not supported
2016-06-17 16:15:13 1bDuYD-000CbB-E4 <= <> R=1bDuYD-000Cb7-8g U=Debian-exim P=local S=1255
2016-06-17 16:15:13 1bDuYD-000Cb7-8g Completed
2016-06-17 16:15:13 1bDuYD-000CbB-E4 => patrick <root@pelargir.herzbube.ch> R=local_user T=maildir_home
2016-06-17 16:15:13 1bDuYD-000CbB-E4 Completed

Start by configuring Exim: 

dpkg-reconfigure exim4-config

The very first question is "General type of mail configuration", and the default answer which reflects the current setting probably is "local delivery only; not on a network". Change the answer to "internet site; mail is sent and received directly using SMTP". This looks dangerous, because we don't really want to receive mail using SMTP, but the answers to the subsequent questions will make sure that SMTP is set up correctly.

  • System mail name = herzbube.ch (default is pelargir.herzbube.ch)
  • IP-addresses to listen on for incoming SMTP connections = 127.0.0.1 ; ::1
    • This is the answer that makes sure that we can't receive mail from external senders.
  • Other destinations for which mail is accepted = <empty>
    • Leaving this empty makes sure that Exim only treats pelargir.herzbube.ch and localhost addresses as local, so when mails are sent to herzbube.ch addresses they are really sent to the proper mail server
  • Domains to relay mail for = <empty>
  • Machines to relay mail for = <empty>
  • Keep number of DNS-queries minimal (Dial-on-Demand)? = No
  • Delivery method for local mail = Maildir format in home directory
  • Split configuration into small files? = Yes (default = No)
    • I prefer a split configuration, this entire Wiki page is based on the assumption that the configuration is split and not monolithic


Now Exim should be properly configured to send mails. The next problem, however, that may occur is that mails are refused by a receiving MTA. For instance, I found this in the log file: 

2016-06-17 16:37:48 1bDuu3-000CuV-P1 ** herzbube@herzbube.ch R=dnslookup T=remote_smtp: SMTP error from remote mail server after initial connection: host mail.mail-ch.ch [81.221.254.153]: 554 impin02.agrinet.ch (mail.messaging.ch) ESMTP server not available (no DNS Reverse)

If this occurs, then the reason is that no reverse DNS entry exists for the server's IP address. These days a reverse DNS entry is mandatory because it is checked by mail hosting services all over the world in order to counteract spam. The solution: Contact your provider and tell them to add a reverse DNS entry that points to the name of the server (in my case pelargir.herzbube.ch) not the entire domain (herzbube.ch)!


Also see this very good article, which explains the responsibilities for setting up reverse DNS: http://www.dnsstuff.com/reverse-dns-faq

Retrieved from "https://wiki.herzbube.ch/index.php?title=Exim&oldid=7051"