L'Xtreme

Are we there yet?

Dovecot, Postfix with PAM authentication

Table of Contents

Introduction ⇑

In this document, I'll describe how Dovecot and Postfix can authenticate users using PAM and PostgreSQL as backend. The foundation of this document is described in the Dovecot wiki. However, this Wiki document contains some flaws and causes authentication problems in some versions of PAM. The latter is caused to the fact that PAM regards '@'-signs as seperator between a username and its realm. Due to this, it is not possible to use an user's email-address for authentication purposes.

Other versions of this document ⇑

Goals of this document ⇑

The purpose of this document is to explain how to use PAM, with a PostgreSQL backend, to lookup and authenticate Postfix and Dovecot users, for multiple virtual domains, with any number of email-addresses (aliases). In addition, we want to allow users to send email through Postfix from any computer, using SMTP authentication.

The goals of the used approach are:

  1. Simplicity: we want to (re-)use as much of the available packages/software as possible, without custom compilations;
  2. Safety: virtual users should be able to access their mail, and nothing
    1. Potential security flaws in the used software should not crack our system wide open;
  3. Maintainability: changing or adding users should be as centralised as possible;
  4. Extensibility: we want to leave room to add new features, like virus-, spam-filtering, or accounting functionality.

Used software and tools ⇑

This document was written for, and tested on a Debian Linux unstable/testing machine, with the following software installed (Debian packages are enclosed in parenthesises):

In addition, some administration scripts for Postfix were written in Perl and Bash. These scripts are mandatory.

Configuring PostgreSQL ⇑

If not done already, we first must allow clients from the local machine to access PostgreSQL databases using password-based authentication. This is done by adding the following entry in the /etc/postgresql/pg_hba.conf file:

host    all    all    127.0.0.1     255.255.255.255    password

Note that our used approach requires that all used packages run on the same machine. With some effort, it is possible to create a seperation between the different packages, allowing them to run on different machines. However, for performance reasons, this should not be necessary as there are reports of people hosting email for 2000+ users on a single machine with a similiar configuration as used in this document.

We now create a database (mail_lxtreme) to store various user and transport information. Furthermore, we create a (restricted) user (mailreader) to allow read-only access to the stored information.

[localhost]$ su - postgres
[localhost]$ createdb mail_lxtreme
[localhost]$ createuser -A -D -P -E mailreader
[localhost]$ psql mail_lxtreme

Next, we create the different tables:

CREATE TABLE domains
(
 gid       INTEGER UNIQUE NOT NULL,
 domain    VARCHAR(128) NOT NULL,
 transport VARCHAR(64) NOT NULL DEFAULT 'virtual:',
 PRIMARY KEY ( gid )
);

The domains table provides allows literal domain-names to be mapped to numerical “group ids” and provides transport information to Postfix.

CREATE TABLE user_accounts
(
 username  VARCHAR(12) UNIQUE NOT NULL,
 uid       INTEGER NOT NULL,
 gid       INTEGER REFERENCES domains ( gid ),
 password  VARCHAR(32),
 email     VARCHAR(128) NOT NULL,
 mailbox   VARCHAR(128) NOT NULL,
 quota     INTEGER NOT NULL DEFAULT 25,
 active    CHAR(1) NOT NULL DEFAULT '0',
 PRIMARY KEY ( uid, gid )
);

Basic user information is stored in the user_accounts table. It allows users to be grouped in a similar manner as “conventional” UNIX users. The uids/gids used should reflect the ownership of the mailbox directory, otherwise users will be denied access to their mailbox. Note that the mailbox field is relative to the virtual_mailbox_base configuration parameter used by Postfix.
The password field should store the encrypted user password. The encryption algorithm used/expected is (standard UNIX) crypt.
The email field should reflect (final) email address for this user. This should (though might not) be an email address handled by our server.

CREATE TABLE user_addresses
(
 username  VARCHAR(12) REFERENCES user_accounts ( username ),
 alias     VARCHAR(128) UNIQUE NOT NULL,
 PRIMARY KEY ( username, alias )
);

The user_addresses table contains all email addresses known to the system for the handled users. This also includes the real email address of each user.

CREATE VIEW postfix_mailboxes AS
     SELECT alias AS address, mailbox||'/' AS mailbox
     FROM user_accounts RIGHT JOIN user_addresses ON
            user_accounts.username = user_addresses.username;
CREATE VIEW postfix_aliases AS
     SELECT email AS address, alias
     FROM user_accounts RIGHT JOIN user_addresses ON
            user_accounts.username = user_addresses.username;

The views postfix_mailboxes and postfix_aliases provide easy access to user information for Postfix. Note that the mailbox field is concatenated with a '/'. This makes Postfix to deliver all user mail in a Maildir style, used by IMAP.

Now our tables are defined, we restrict the access to our data even more:

GRANT SELECT ON domains, user_accounts, user_addresses,
     postfix_mailboxes, postfix_aliases TO mailreader;

If everything went alright, we can insert some data:

INSERT INTO domains ( gid, domain )
     VALUES ( 5001, 'lxtreme.nl' );
INSERT INTO user_accounts ( username, uid, gid, password,
                           email, mailbox, active )
     VALUES ( 'jawi_lx', 5001, 5001, 'my_crypted_pw',
	         'janwillem.janssen@lxtreme.nl',
              'lxtreme.nl/janwillem.janssen', '1' );
INSERT INTO user_accounts ( username, uid, gid, password,
                           email, mailbox, active )
     VALUES ( 'info_lx', 5002, 5001, 'an_crypted_pw',
              'info@lxtreme.nl',
              'lxtreme.nl/info', '1' );
INSERT INTO user_addresses ( username, alias )
     VALUES ( 'jawi_lx', 'janwillem.janssen@lxtreme.nl' );
INSERT INTO user_addresses ( username, alias )
     VALUES ( 'jawi_lx', 'j.w.janssen@lxtreme.nl' );
INSERT INTO user_addresses ( username, alias )
     VALUES ( 'jawi_lx', 'postmaster@lxtreme.nl' );
INSERT INTO user_addresses ( username, alias )
     VALUES ( 'jawi_lx', 'hostmaster@lxtreme.nl' );
INSERT INTO user_addresses ( username, alias )
     VALUES ( 'jawi_lx', 'webmaster@lxtreme.nl' );
INSERT INTO user_addresses ( username, alias )
     VALUES ( 'info_lx', 'info@lxtreme.nl' );

Note that if you set a user's password, that this should be in its encrypted, using the (standard) crypt algorithm. I've written a simple Perl-module, that asks for a (plain-text) password and outputs the crypt'ed result (read the file for documentation).

Configuring Postfix ⇑

Postfix provides several hooks for querying external sources for its mail delivery information. These external sources are called maps by Postfix, and can be regarded as lookup tables: a single input value should result in a single output value. For this document, we define and used the following hooks:

To use an external source for a certain map, Postfix uses a URI-like syntax. For example, to use a PostgreSQL database as map-source, you would use the following: pgsql:/path/to/query_file, in which the “query_file” contains the actual lookup-query for PostgreSQL.

Putting everything together, we add the following entries to Postfix's main.cf file:

transport_maps = pgsql:/etc/postfix/transport.cf
virtual_minimum_uid = 5000
virtual_uid_maps = pgsql:/etc/postfix/uids.cf
virtual_minimum_gid = 5000
virtual_gid_maps = pgsql:/etc/postfix/gids.cf
virtual_alias_maps = pgsql:/etc/postfix/virtual.cf
virtual_mailbox_domains = pgsql:/etc/postfix/transport.cf
virtual_mailbox_base = /var/local/mail
virtual_mailbox_maps = pgsql:/etc/postfix/mailboxes.cf
virtual_mailbox_limit = 51200000
mydestination = $myhostname

From this snippet, we can see that PostgreSQL is extensively used for looking up all sorts of information. Furthermore, we see that all mailboxes are stored relative to /var/local/mail, with a maximum limit of around 50 megabytes. The last entry tells Postfix that all numerical userids are counted from 5000 upwards.

Without going much into their specific details (they can be found here), we present the used “query_files”:

transport.cf:

# domain transport information
hosts = localhost
user = mailreader
password = secret
dbname = mail_lxtreme
table = domains
select_field = transport
where_field = domain

uids.cf:

# user id information
hosts = localhost
user = mailreader
password = secret
dbname = mail_lxtreme
table = user_accounts
select_field = uid
where_field = email

gids.cf:

# group id information
hosts = localhost
user = mailreader
password = secret
dbname = mail_lxtreme
table = user_accounts
select_field = gid
where_field = email

virtual.cf:

# email address (real+virtual) information
hosts = localhost
user = mailreader
password = secret
dbname = mail_lxtreme
table = postfix_aliases
select_field = address
where_field = alias

mailboxes.cf:

# user information
hosts = localhost
user = mailreader
password = secret
dbname = mail_lxtreme
table = postfix_mailboxes
select_field = mailbox
where_field = address

Now, make sure that the virtual_mailbox_base directory exists and has the proper permissions. It should be readable, writeble and browseable by Postfix. Any virtual domains located underneat virtual_mailbox_base, should be owned by the Postfix-user and have the correct (numerical) gid. Virtual users should have the proper uid and gid. Below you find an example listing showing the correct permissions:

[localhost]$ ls -al /var/local/mail/lxtreme.nl/
lxtreme.nl/
drwxrwx---    4 postfix  5001         4096 Apr 18 13:11 ./
drwxrwxr-x    4 postfix  postfix      4096 Apr 18 13:11 ../
drwx------    5 5002     5001         4096 Apr 18 13:14 info/
drwx------    5 5001     5001         4096 Apr 18 13:11 janwillem.janssen/

Note that you need to create a seperate mailbox for each virtual user you add to the database! I'm currently working on a script that synchronises the database and “real” mailboxes and sets the correct permissions.

At this point, Postfix is sufficiently configured to successfully accept and deliver email for many (virtual) users on many different (virtual) domains. However, those users are at this point not able to access their mail after delivery.

Configuring Dovecot ⇑

Dovecot is configured through a single configuration file, /etc/dovecot.conf. Since most of the default settings of Dovecot are perfectly reasonable, we only need to modify the following entries:

default_mail_env = maildir:/var/local/mail/%d/%n
auth_userdb = pgsql /etc/dovecot-pgsql.conf
auth_passdb = pgsql /etc/dovecot-pgsql.conf
# Tell Dovecot more about the used uids/gids.
first_valid_uid = 5000
last_valid_uid = 65535
first_valid_gid = 5000
last_valid_gid = 65535

These entries tell Dovecot were to find a user's mailbox and what method to use for verifying users and their passwords using the /etc/dovecot-pgsql.conf file as source for the queries. Based on our used database schema, we use the following /etc/dovecot-pgsql.conf:

# Dovecot queries
connect = host=localhost dbname=mail_lxtreme user=mailreader password=secret
default_pass_scheme = CRYPT
password_query = SELECT password FROM user_accounts WHERE username='%u'
user_query = SELECT '/var/local/mail/'||mailbox AS mail, uid, gid FROM
	user_accounts WHERE username='%u'

Remember that when adding or modifying users, their passwords should be encrypted using the “crypt” algorithm!

Additionally, you might want to refine your Dovecot configuration to reflect your own situation.

First test: mail delivery and access ⇑

Both Postfix and Dovecot can now be restarted to test the current configuration. Detailed explanations about testing the configuration is beyond the scope of this document. However, you should check the following (possibly incomplete) list:

If all tests are positive, i.e., your configuration reacts accordingly, we can proceed to the next phase: allow relaying for hosted users and tightening security.

Some of the more common pitfalls:

Allow relaying for hosted users ⇑

In most cases, you do not want your Postfix configuration to act as “open relay” (unless you're explicitly creating one). However, we do want our hosted users to sent email through our server using their favourite email client at home. We need to make Postfix aware of those who can and those who cannot sent email through our servers. By default, Postfix only allows machines from the “local network”, as set by mynetworks in main.cf, to sent email. However, we do not know a priori from which hosts our users want to sent their email (they could have a dynamic IP address for instance).

The solution to the sketched problem is to use SMTP authentication, described in RFC 2554. Using this authentication method, we restrict access to our SMTP-server to those which have an account.

Enable Postfix SASL support ⇑

Postfix uses SASL (RFC 2222, Cyrus SASL2 implementation) to provide SMTP authentication to its clients.

To enable SMTP authentication using SASL, we need to add the following entries to main.cf:

# Enable SMTP authentication support
smtp_sasl_auth_enable = no
smtpd_sasl_auth_enable = yes
smtpd_sasl_security_options = noanonymous
smtpd_sasl_local_domain = $mydomain
unknown_local_recipient_reject_code = 450
smtpd_recipient_restrictions =
     permit_mynetworks,
     permit_sasl_authenticated,
     reject_unauth_destination

Basically, this tells Postfix to require non-anonymous SMTP authentication for any recipient that is not locally known. Email from unauthorized clients to unknown recipients is simply rejected.

Configuring SASL2 ⇑

The SASL authentication daemon (saslauthd) uses UNIX sockets to communicate with other programs. Because Postfix runs in a chroot'ed environment, we need to make sure that the SASL daemon and Postfix are capable of talking to eachother. In other words: we must enforce SASL to create a UNIX socket somewhere in Postfix's chroot environment. To do so, we create (or modify) the following saslauthd configuration file /etc/default/saslauthd:

START=yes
MECHANISMS=pam
PARAMS="-m /var/spool/postfix/etc"

Also, we must tell Postfix's SASL layer that it can talk to the saslauthd daemon via an UNIX socket. Note that the saslauthd_path parameter is relative to the chroot directory! We create the following /etc/postfix/sasl/smtpd.conf:

pwcheck_method: saslauthd
saslauthd_path: /etc/mux
mech_list: login plain

The last option narrows the possible authentication mechanisms to either login or plain.

Configuring PAM ⇑

Debian heavily relies on PAM for all of its authentication purposes. In /etc/default/saslauthd we told SASL to use PAM as its authentitcation mechanism. However, since all our users' passwords are stored in a PostgreSQL database, we need to enable PAM to look them up. The pam_pgsql module is configured in /etc/pam_pgsql.conf:

database = mail_lxtreme
host = localhost
user = mailreader
password = secret
table = user_accounts
user_column = username
pwd_column = password
pw_type = crypt

The above configuration file simply tells the pam_pgsql module to use the user_accounts table, located in the mail_lxtreme database. In this table, it should check the user's name and its password against the username and password columns. Furthermore, all passwords are mangled using the crypt algorithm.

PAM must also be told which programs should use what authentication method(s). This is done by creating /etc/pam.d/smtp:

auth        required    pam_pgsql.so 
account     required    pam_pgsql.so
password    required    pam_pgsql.so

Testing authentication ⇑

We should now be able to test whether SMTP authentication is working correctly (or not). First, we need to (re)start our saslauthd and postfix daemons.

/etc/init.d/saslauthd restart
/etc/init.d/postfix restart

Check your processes and logs to see whether both daemons are running properly!

If everything went OK, sending email to non-hosted users without authentication should fail! You should check this on a machine that is not trusted by Postfix, i.e., not described by mynetworks. Of course, email to hosted users should be accepted normally.

Now, modify your email client's configuration to use SMTP authentication (SMTP AUTH) for sending email through your server. Again, try to send an email through your server. It should succeed. If not, check /var/log/auth.log and /var/log/mail.log to see what's going wrong.

Tightening your Postfix configuration ⇑

At this point, you should have a working Postfix configuration, capable of authenticating authorized users and avoiding relaying access to spammers. We now can tighten the security and access to our mail server. What we want to do is to make it even harder for spammers to use our system as potential relay and to restrict access from blacklisted hosts. Modify the /etc/postfix/main.cf file as follows (Note: you should do this only if you've got a proper working Postfix configuration! Do not use these settings while debugging your Postfix configuration):

disable_vrfy_command = yes
smtpd_helo_required = yes
smtpd_recipient_restrictions =
     permit_mynetworks,
     permit_sasl_authenticated,
     reject_unauth_pipelining,
     reject_unknown_recipient_domain,
     reject_non_fqdn_sender,
     reject_non_fqdn_recipient,
     reject_unauth_destination
smtpd_client_restrictions =
     permit_mynetworks,
     reject_rbl_client relays.ordb.org,
     reject_rbl_client ipwhois.rfc-ignorant.org,
     permit_sasl_authenticated
smtpd_sender_restrictions =
     permit_mynetworks,
     reject_rhsbl_sender dsn.rfc-ignorant.org,
     permit_sasl_authenticated,
     reject_sender_login_mismatch
# Setup a spam tarpit...
smtpd_error_sleep_time = 60
smtpd_soft_error_limit = 60
smtpd_hard_error_limit = 10
default_process_limit = 3

If you came this far, and everything seems to work perfectly: congratulations, you've got a full functional email-server allowing you to host email for many different users on many different domains!

Monitoring and maintainance of Postfix ⇑

Most of the time, your mail server's queue will be empty after sending an email through it. However, it could occur that messages keep lingering in the queues longer than expected, for instance because its recipient does no longer exists. If the sender's email address exists, this problem will resolve on its own when Postfix sends a failure message to the sender after which the failing message is removed from the queue. However, when the sender's email address does not exist (e.g. because it is fake), Postfix won't be able to send a failure message, keeping the failing message indefinitely in the mail queue. Lots of these “deferred” emails can degrade your server's performance significantly. Therefore, we should clean it up once in a while using the following cronjob (taken from [6]):

#!/bin/sh
mailq | grep MAILER-DAEMON | awk '{printf $1} {print " deferred"}' | \
	tr -d '*!' | xargs -n 2 postsuper -d >/dev/null 2>&1

Although there is a very small possibility that the above script deletes the wrong message due to Postfix' reuse of queue IDs, this possibility is really small (your server needs a very high load of email for this to happen).

If you're following this howto for setting up your own mail server, you might want to retrieve some interesting stats from it. Jim Seymour has crafted a nice script which does just that:

pflogsummary v1.1.0

I've created a small wrapper script that mails the output of pflogsummary to the mail server's postmaster user. It runs as a cron-job each night and gives me good insight in my servers statistics:

#!/bin/sh
PATH=/bin:/usr/bin:/usr/local/sbin
LOG_FILE=/var/log/mail.log.0*
PFLOGSUMM=/usr/local/sbin/pflogsummary.pl
$PFLOGSUMM -d yesterday --problems_first `ls -rt $LOG_FILE*` | \ 
	mail -s "Postfix :: report" postmaster

The above script extracts yesterdays' statistics from Postfix' log-files, reporting (critical) problems first.

Downloads ⇑

For your convenience, you can download the above mentioned scripts:

Postfix report scripts, v1.1 (11/06/04)

References ⇑

  1. http://wiki.dovecot.org/moin.cgi/DovecotPostgresql
  2. http://www.postfix.org/PGSQL_README.html
  3. http://www.faqs.org/rfcs/rfc2554.html
  4. http://www.faqs.org/rfcs/rfc2222.html
  5. http://asg.web.cmu.edu/sasl/sasl-library.html
  6. http://kancer.978.org/lawmonkey.org/anti-spam.html
  7. http://jimsun.linxnet.com/postfix_contrib.html

Document history ⇑

Friday, 15 October 2004 (15:16:55) :: © copyright 2004 - L'Xtreme. LXcms v1.0b5.   [valid XHTML 1.1]   [valid CSS]