Mailman

This page is about the Mailman software, used to host and manage mailing lists.

Debian package

The following Debian packages need to be installed:

mailman

References

http://wiki.list.org/

The mailman wiki

http://www.list.org/mailman-install/index.html

Installation

http://www.exim.org/howto/mailman21.html

Exim integration

Configuration

Important note: Do not start the mailman daemon until the site-wide administrative-purpose mailing list named mailman has been created !!!

DebConf

DebConf configures mailman like this:

• user/group = list/list
• installation directory = /var/lib/mailman

/etc/mailman/mm_cfg.py

Note: I have not found real documentation about options that can be placed in this file. The next best thing is to study the default settings in /usr/lib/mailman/Mailman/Defaults.py, because mm_cfg.py is used to override those defaults.

Edit the following configuration file in a text editor:

/etc/mailman/mm_cfg.py

Already configured by DebConf

DEFAULT_EMAIL_HOST = 'herzbube.ch'
DEFAULT_URL_HOST = 'lists.herzbube.ch'

(note that DEFAULT_URL_HOST defines the host that must be used later on for web administration)

At a later stage, Exim will be configured to automatically work with every new mailing list created by mailman. Because of this, we can make the following adjustment:

MTA=none

Change the following options to allow Mailman to work out of the root of its own Apache vhost. Make sure that the DEFAULT_URL_PATTERN path has a trailing slash ("/") character, otherwise various URLs will be generated incorrectly:

DEFAULT_URL_PATTERN = 'http://%s/'
PRIVATE_ARCHIVE_URL = '/private'
IMAGE_LOGOS         = '/images/'

Important note: If you have already created some lists, you cannot just edit the configuration file and be done with it. Rather you must fix the existing mailing lists to match your new configuration. One example is that you have to run /var/lib/mailman/bin/fix_url.py after you have changed the DEFAULT_URL_PATTERN option.

Integration with Exim

General stuff

Create the following file

/etc/exim4/conf.d/main/50_exim4-config_pelargir

with the following content:

MM_HOME=/var/lib/mailman
MM_UID=list
MM_GID=list
domainlist mm_domains=herzbube.ch
MM_WRAP=MM_HOME/mail/mailman
MM_LISTCHK=MM_HOME/lists/${lc::$local_part}/config.pck

Router configuration

It is important that the mailman router appears before (!) the virtualdomains router, so that it has a chance to catch mailing list messages before they are handled by virtualdomains.

To effect this, the mailman router configuration file simply must have a prefix that lets it be sorted in front of the file containing the virtualdomains router:

/etc/exim4/conf.d/router/040_exim4_config_pelargir_mailman

File content:

mailman_router:
  driver = accept
  domains = +mm_domains
  require_files = MM_LISTCHK
  local_part_suffix_optional
  local_part_suffix = -admin     : \
    -bounces   : -bounces+* : \
    -confirm   : -confirm+* : \
    -join      : -leave     : \
    -owner    : -request   : \
    -subscribe : -unsubscribe
  transport = mailman_transport

Transport configuration

Place the transport configuration in this file:

/etc/exim4/conf.d/transport/40_exim4_config_pelargir_mailman

File content:

mailman_transport:
  driver = pipe
  command = MM_WRAP \
            '${if def:local_part_suffix \
               {${sg{$local_part_suffix}{-(\\w+)(\\+.*)?}{\$1}}} \
               {post}}' \
            $local_part
  current_directory = MM_HOME
  home_directory = MM_HOME
  user = MM_UID
  group = MM_GID

No tuning

Because my mailing lists are small and private and do not cause much traffic, I do not take any of the "Mailing List MTA Tuning" measure suggested by the Exim integration manual.

• I do not limit how many recipients mailman is allowed to use per message (default = 500)
• I do not limit how many messages mailman is allowed to send during the same SMPT session (default = ?)
• I do not limit how many SMTP sessions mailman is allowed to initiate during the same TCP connection (default = ?)

The Exim document for integrating mailman suggests that recipient verification should be turned off for mailman traffic in the rcpt_to ACL. It is currently unclear whether this is true or not in my current Exim setup. I suspect that recipient verification occurs, but I have not felt any negative effects so far, probably again because of the low traffic on my mailing lists.

Tests

After a mailing list has been created, the following tests can be done on the command line

exim -bt testlist@list.example.com
exim -bt testlist-request@list.example.com

Integration with Apache

I have assigned an Apache vhost to Mailman that is accessible under http://lists.herzbube.ch/. These are the configuration details:

# --------------------------------------------------------------------------------
# lists.herzbube.ch
# --------------------------------------------------------------------------------
<VirtualHost *:80>
  ServerName lists.herzbube.ch
  ServerAlias lists.lan.herzbube.ch lists.wifi.herzbube.ch
  ServerAdmin webmaster@herzbube.ch
  ErrorLog /var/log/apache2/herzbube.ch/error.log
  CustomLog /var/log/apache2/herzbube.ch/access.log combined

  DocumentRoot /usr/lib/cgi-bin/mailman
  Alias /robots.txt /var/www/lists.herzbube.ch/robots.txt

  # Redirect / to the default CGI that will enumerate public mailing lists
  <IfModule mod_rewrite.c>
    RewriteEngine on
    RewriteCond %{REQUEST_URI} ^/$
    RewriteRule ^.*$ /listinfo/ [R,L]
  </IfModule>

  # Scripts in this folder do not have the .cgi extension, therefore they
  # are not recognized by the CGI handler that is connected to the extension
  # somewhere else in the global Apache configuratio. We therefore have
  # to explicitly tell Apache that all files in this directory are CGI
  # scripts.
  <Directory /usr/lib/cgi-bin/mailman/>
    Require all granted
    SetHandler cgi-script
  </Directory>

  # Make a couple of logos available
  Alias /images/ /usr/share/images/mailman/
  <Directory /usr/share/images/mailman/>
    Require all granted
  </Directory>

  # Make the public mailing lists' archives available. We don't have to
  # do anything special for The private archives: These are available
  # only through the CGI script "private" after the user has authenticated.
  Alias /pipermail/ /var/lib/mailman/archives/public/
  <Directory /var/lib/mailman/archives/public/>
    Require all granted
    Options FollowSymLinks
    # For reasons I have not investigated, the archive content is not UTF-8
    # encoded. I am not sure if in the future everything will be ISO-8859-1,
    # but at the moment this is how it works.
    <IfModule mod_mime.c>
      AddDefaultCharset iso-8859-1
    </IfModule>
  </Directory>

  # We have to add the charset via <Location> directive because the private
  # archives are accessed through a CGI script in that location (not by simply
  # browsing through a directory structure, as with the public archives)
  <Location /private>
    <IfModule mod_mime.c>
      AddDefaultCharset iso-8859-1
    </IfModule>
  </Location>

  <Directory /var/www/lists.herzbube.ch/>
    Require all granted
  </Directory>
</VirtualHost>

# --------------------------------------------------------------------------------
# SSL Host
# --------------------------------------------------------------------------------
<IfModule mod_ssl.c>
  <VirtualHost *:443>
    ServerName lists.herzbube.ch
    ServerAlias lists.lan.herzbube.ch lists.wifi.herzbube.ch
    ServerAdmin webmaster@herzbube.ch
    ErrorLog /var/log/apache2/herzbube.ch/error.log
    CustomLog /var/log/apache2/herzbube.ch/access.log combined

    DocumentRoot /usr/lib/cgi-bin/mailman
    Alias /robots.txt /var/www/lists.herzbube.ch/robots.txt

    <IfModule mod_rewrite.c>
      RewriteEngine on
      RewriteCond %{REQUEST_URI} ^/$
      RewriteRule ^.*$ /listinfo/ [R,L]
    </IfModule>

    <Directory /usr/lib/cgi-bin/mailman/>
      Require all granted
      SetHandler cgi-script
    </Directory>

    Alias /images/ /usr/share/images/mailman/
    <Directory /usr/share/images/mailman/>
      Require all granted
    </Directory>

    Alias /pipermail/ /var/lib/mailman/archives/public/
    <Directory /var/lib/mailman/archives/public/>
      Require all granted
      Options FollowSymLinks
      <IfModule mod_mime.c>
        AddDefaultCharset iso-8859-1
      </IfModule>
    </Directory>
    <Location /private>
      <IfModule mod_mime.c>
        AddDefaultCharset iso-8859-1
      </IfModule>
    </Location>

    <Directory /var/www/lists.herzbube.ch/>
      Require all granted
    </Directory>

    SSLEngine on
    SSLCertificateFile /etc/ssl/certs/herzbube.ch.crt
    SSLCertificateKeyFile /etc/ssl/private/herzbube.ch.key.unsecure
    SSLCertificateChainFile /etc/ssl/certs/cacert.org.certchain
    SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown
  </VirtualHost>
</IfModule>

Setup after initial configuration

Site-wide passwords

mailman has 2 special kinds of site-wide passwords that need to be set up after the initial configuration has been made:

• site password: this is the "root" password for mailman; wherever a password is required, the site password gives you access; among other things, this password can be used to create new mailing lists through the web interface
• list creator password = is a password to delegate the ability to create new mailing lists

To setup these passwords, use the utility

mmsitepass

Note: I don't use the list creator password because I do all the mailing list management myself.

Site-wide mailing list

Also required after the initial configuration is the creation of a site-wide mailing list used for administrative purposes. For instance, this mailing list is the one that password reminders will appear to come from. Usually this special mailing list should be called mailman. If a different name is necessary, the configuration file /etc/mailman/mm_cfg.py needs to be changed.

Create the site-wide mailing list like this:

newlist mailman

Or through the web interface at

http://herzbube.ch/cgi-bin/mailman/create

The attributes for the mailing list are:

• list owner = herzbube@herzbube.ch
• password = <secret>

After the mailing list has been created, you can finally start the daemon:

/etc/init.d/mailman start

You should now subscribe yourself to the mailing list.

Languages

When the Debian package is installed for the first time, there is no option to choose any languages, so mailman will support only English. To fix this:

• either enable languages through the web interface, then update the Debian package
• or say dpkg-reconfigure mailman

I have installed the additional languages

• German
• French
• Italian
• Dutch

and made "German" the default language.

Command line administration

List all existing mailing lists

Run the following command, which should be in your PATH:

list_lists

List members of a mailing list

Run the following command, which should be in your PATH:

list_members <listname>

Perform operations on a mailing list

If a mailman operation does not have a script in your PATH, the command

withlist

can be used to perform the operation. The name of the operation to perform is the name of a script from the folder

/usr/lib/mailman/bin

The general syntax is this:

withlist -l -r <command> <listname> [options]

The -l option locks the list object so that nothing else can operate on that list. Although it may be safe to omit this option when performing read-only operations, I still use it just to be on the safe side. The -r option names the command to run.

Hint: To find out which mailman operations are in your PATH, try the command ls -l /usr/sbin | grep -i mailman

Fix URL of existing mailing lists

If an URL setting has changed in /etc/mailman/mm_cfg.py, existing mailing lists have to be updated to reflect the change.

withlist -l -r fix_url <listname> -v

Unfortunately, the mailing list archive is not touched by this at all. It needs to be regenerated with this command:

mmarch --wipe <listname>

Important: Regenerating the archive must be done after running fix_url on a mailing list.

Web administration

URL

It is important to note that web administration can be done only with URLs that use the domain lists.herzbube.ch. The reason is that mailman CGI scripts always generate absolute URL names that explicitly use the host DEFAULT_URL_HOST specified in /etc/mm_cfg.py.

So the various CGI scripts are available at

http://herzbube.ch/cgi-bin/mailman/<scriptname>/<listname>

For instance, the following URL allows administration of the mailman mailing list:

http://herzbube.ch/cgi-bin/mailman/admin/mailman

Adding members

When members are added to a mailing list (also when mass adding members) a password is automatically generated for every member. The password is then sent out inside the welcome message. If no welcome message is sent, members can use the "lost password" feature of the web interface to get at the automatically generated password.

Deleting mailing lists

It is not possible to delete a mailing list through the web interface as long as /etc/mailman/mm_cfg.py does not contain the setting

OWNERS_CAN_DELETE_THEIR_OWN_LISTS = Yes

By default, this option is set to "No". When the option has been enabled, the following URL allows to delete a mailing list:

http://herzbube.ch/cgi-bin/mailman/rmlist/<liste>

To delete a mailing list on the command line:

rmlist <list>

To also remove the archives

rmlist -a <liste>

Making a list private/public

A public list is one that is advertised on the page where all lists of a site are enumerated (the "listinfo" page). A private list is not advertised, but still accessible to those who know that it exists.

To make a list private, login to the list's administrative interface, then select "Privacy options > Subscription rules" and set the "advertised" flag to false. If the web interface is displayed in German, the category to look for is (unintuitively) named "Abo-Regeln und Adressfilter > Abo-Regeln".

My preferred list settings

When a new mailing list is created, I like to change some of its settings as follows:

• general options
  • real_name = change the first letter of the list name to lower case
  • description = <something useful>
  • subject_prefix = change the first letter of the list name to lower case
  • admin_immed_notify = yes (immediately notify list owner when "something special" happens)
  • admin_notify_mchanges = yes (send notifications about subscribes/unsubscribes)
  • max_message_size = <xx> (40 KB is the default, which might be too low for certain lists)

• language options
  • add german, italian, french, dutch
  • preferred_language = german

• digest options
  • digestable = false

• privacy options
  • subscription rules
    • subscribe_policy = require approval
  • sender filters
    • generic_nonmember_action = discard|reject|accept (if non-members should be disallowed|allowed to post; if the action is "discard", posters do not get notified; this is most useful to gobble up spam)
    • forward_auto_discards = yes | no (usually no, because yes means that spam is routed to the list admin)

• archiving options
  • archive_volume_frequency = yearly

Special list settings

travel-news

• general options
  • send_reminders = yes (because this list will be very low-traffic and people are practically guaranteed to forget that they are on this list)
  • max_message_size = 100 (just to be on the safe side, although I do not intend to send attachments)

• privacy options
  • subscription rules
    • subscribe_policy = confirm (I want the list to be open for anybody)
  • sender filters
    • default_member_moderation = yes (I and Francesca should be the only ones that can send emails to the list)
    • member_moderation_action = reject (if somebody still attempts to post, the message will be sent back so that they see that why it was not sent)

Troubleshooting

senddigests error

The senddigests tool which is run nightly by cron (see /etc/cron.d/mailman) barfs with this message:

Traceback (most recent call last):
  File "/usr/lib/mailman/cron/senddigests", line 94, in ?
    main()
  File "/usr/lib/mailman/cron/senddigests", line 86, in main
    mlist.send_digest_now()
  File "/usr/lib/mailman/Mailman/Digester.py", line 60, in send_digest_now
    ToDigest.send_digests(self, mboxfp)
  File "/usr/lib/mailman/Mailman/Handlers/ToDigest.py", line 142, in send_digests
    send_i18n_digests(mlist, mboxfp)
  File "/usr/lib/mailman/Mailman/Handlers/ToDigest.py", line 324, in send_i18n_digests
    msg = scrubber(mlist, msg)
  File "/usr/lib/mailman/Mailman/Handlers/Scrubber.py", line 393, in process
    replace_payload_by_text(msg, sep.join(text), charset)
  File "/usr/lib/mailman/Mailman/Handlers/Scrubber.py", line 175, in
replace_payload_by_text
    msg.set_payload(text, charset)
  File "email/Message.py", line 218, in set_payload
  File "email/Message.py", line 242, in set_charset
TypeError: iso-8859-1

Found some help here: http://mail.python.org/pipermail/mailman-users/2007-July/057772.html

When looking at the file

/var/lib/mailman/lists/ludenti-announce/digest.mbox

I don't see any headers with apostrophes. After unsuccessfully rooting around for more information that could help me, I ended up simply deleting the digest.mbox file. senddigests now runs through again without errors, I just hope I didn't break anything. Update: So far, deleting the digest.mbox file did not seem have any negative effects.

Another error message by senddigests (which actually prevented the upgrade from mailman 2.1.9 to 2.1.11):

Traceback (most recent call last):
  File "/usr/lib/mailman/cron/senddigests", line 94, in <module>
    main()
  File "/usr/lib/mailman/cron/senddigests", line 86, in main
    mlist.send_digest_now()
  File "/var/lib/mailman/Mailman/Digester.py", line 60, in send_digest_now
    ToDigest.send_digests(self, mboxfp)
  File "/var/lib/mailman/Mailman/Handlers/ToDigest.py", line 142, in send_digests
    send_i18n_digests(mlist, mboxfp)
  File "/var/lib/mailman/Mailman/Handlers/ToDigest.py", line 324, in send_i18n_digests
    msg = scrubber(mlist, msg)
  File "/var/lib/mailman/Mailman/Handlers/Scrubber.py", line 196, in process
    ctype = part.get_type(part.get_default_type())
AttributeError: Message instance has no attribute 'get_type'

Google led me to this message: http://mail.python.org/pipermail/mailman-users/2008-March/060822.html. It seems as if the original poster's problem was not related to senddigests, but the final error message "AttributeError: [...]" indicates that the two problems might have the same root. It is suggested that mailman uses an "email" package from the Python system installation instead of from the Mailman installation.

I checked and found that there exists a directory /usr/lib/mailman/pythonlib/email. Unfortunately, I don't know enough about python to easily find out whether senddigest uses this mailman-owned email package. In the end, I again had no choice but to delete the offending digest.mbox file.

Update: Debian bug report #443069 confirmed that this is indeed about the mailman-owned email package. It also provides a solution, see the next Troubleshooting entry about my woes when upgrading from mailman 2.1.9 to 2.1.11.

"Old queue files present" when upgrading the mailman package

When upgrading from mailman 2.1.9 to 2.1.11, dpkg presents the following message box and refuses to do the upgrade:

Old queue files present

The directory /var/lib/mailman/qfiles contains files. It needs to be empty for the upgrade to work properly. You can try to handle them by:
 - Stop new messages from coming in (at the MTA level).
 - Start a mailman queue runner: /etc/init.d/mailman start
 - Let it run until all messages are handled.
   If they don't all get handled in a timely manner, look at the logs
   to try to understand why and solve the cause.
 - Stop it: /etc/init.d/mailman stop
 - Retry the upgrade.
 - Let messages come in again.
You can also decide to simply remove the files, which will make Mailman forget about (and lose) the corresponding emails.

If these files correspond to shunted messages, you have to either delete them or unshunt them (with /var/lib/mailman/bin/unshunt). Shunted messages are messages on which
Mailman has already abandoned any further processing because of an error condition, but that are kept for admin review. You can use /var/lib/mailman/bin/show_qfiles to
examine the contents of the queues.

The investigation revealed the following:

• the directory /var/lib/mailman/qfiles/shunt contains 8 files
• the content of each file can be examined with /var/lib/mailman/bin/show_qfiles
• the files are messages that were, to my knowledge, correctly delivered to all subscribers (at least I got the messages myself, and I did not hear any complaints from users)
• the messages, however, do not appear in the mailing list archives
• so I try to unshunt the messages, hoping they will be transferred to the archives: /var/lib/mailman/bin/unshunt
• unfortunately, this does not work: the messages still sit in the shunt directory; when looking at the error log (/var/lib/mailman/logs/error) I can see that unshunt probably tried to do the right thing (putting the messages into the archive), but that the archiving tool seems to have the same problem as senddigest (see above):

Aug 23 18:42:14 2008 (2947) SHUNTING: 1191069753.370122+1b56436ae05d348ffc4b41f6495ddc1f85e8feed
Aug 23 18:42:14 2008 (2947) uncaught archiver exception at filepos: 0
Aug 23 18:42:14 2008 (2947) Uncaught runner exception: Message instance has no attribute 'get_type'
Aug 23 18:42:14 2008 (2947) Traceback (most recent call last):
  File "/var/lib/mailman/Mailman/Queue/Runner.py", line 112, in _oneloop
    self._onefile(msg, msgdata)
  File "/var/lib/mailman/Mailman/Queue/Runner.py", line 170, in _onefile
    keepqueued = self._dispose(mlist, msg, msgdata)
  File "/var/lib/mailman/Mailman/Queue/ArchRunner.py", line 73, in _dispose
    mlist.ArchiveMail(msg)
  File "/var/lib/mailman/Mailman/Archiver/Archiver.py", line 214, in ArchiveMail
    h.processUnixMailbox(f)
  File "/var/lib/mailman/Mailman/Archiver/pipermail.py", line 564, in processUnixMailbox
    m = mbox.next()
  File "/usr/lib/python2.5/mailbox.py", line 1933, in next
    return self.factory(_PartialFile(self.fp, start, stop))
  File "/var/lib/mailman/Mailman/Mailbox.py", line 89, in scrubber
    return mailbox.scrub(msg)
  File "/var/lib/mailman/Mailman/Mailbox.py", line 109, in scrub
    return self._scrubber(self._mlist, msg)
  File "/var/lib/mailman/Mailman/Handlers/Scrubber.py", line 196, in process
    ctype = part.get_type(part.get_default_type())
AttributeError: Message instance has no attribute 'get_type'

• I decide to move the shunted messages out of the way, then do the upgrade, move them back and try to unshunt again with the new version 2.1.11; I hope that the new version will have fixed the "has no attribute get_type" problem...
• the upgrade now works, but unshunting still does not work
• after some more unsuccessful googling and checking the mailman wiki, I finally decide that the real problem might be related to the Debian package, not mailman itself, so I check the Debian bugs for mailman (http://bugs.debian.org/cgi-bin/pkgreport.cgi?package=mailman)
• I find this bug: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=443069 which seems to describe my problem; so the real bugger is that on my system I have an empty /var/lib/mailman/pythonlib directory instead of a symlink
• I fix the problem:

pelargir:~# cd /var/lib/mailman
pelargir:/var/lib/mailman# ls -ld pythonlib
drwxrwsr-x 2 root list 4096 Aug 17  2007 pythonlib
pelargir:/var/lib/mailman# rmdir pythonlib
pelargir:/var/lib/mailman# ln -s /usr/lib/mailman/pythonlib /var/lib/mailman/pythonlib

• I try unshunting again, which - unbelievable! - barfs again, this time with the following message:

Cannot unshunt message 1191069753.370122+41ad516cbe95b497441272430bb2546c7c2e21d2, skipping:
No module named header

• the /var/lib/mailman/qfiles/shunt directory now contains *.bak files instead of *.pck files, but the unshunted messages still are not in the archive; what the heck!
• I try unshunting once more, this time the /var/lib/mailman/qfiles/shunt directory gets emptied, but there are errors in the log:

Aug 23 19:56:53 2008 (15094) Unpickling .bak exception: No module named header
preserving file: 1191069753.370122+41ad516cbe95b497441272430bb2546c7c2e21d2

• I now find the messages in the /var/lib/mailman/qfiles/bad directory, and I am now really out of ideas; could it be that the format of the files is for mailman 2.1.9, and that the format did not get upgraded because the files were not present when I upgraded the package?
• I give up! I don't want to waste more time on this, and it's not that important to me that the messages are in the archive; hopefully, everything works smoothly from now on