Mail server scalability
What problems do we come across?
Linear password files
On some systems, every mail delivery and pop3 connection requires a scan
through the whole /etc/passwd file.
- This is a problem with many Linux distributions
- FreeBSD uses searchable databases /etc/pwd.db, /etc/spwd.db
- Don't give mail users a Unix account; have a separate user database.
(Better for security, too)
- Or, use an external account database such as LDAP or RADIUS
Linear mbox files
If a user keeps their mail on the server, every POP3 or IMAP connection
and every mailbox change requires the POP3 daemon to read the entire mail
file. 30,000 messages is a 500MB mailbox file that takes 30 seconds to
scan.
Maildir is an alternative which stores each message in its own
file:
- Solves the problem of reading a large file.
- Creates a new problem of reading a large directory!
- Choice of filesystem and options makes a big difference (e.g.
reiserfs or htree)
- "Creating 100,000 files in a single directory took 38 minutes
without directory indexing... and 11 seconds with the directory indexing
turned on." (Theodore
Ts'o)
- Can run out of inodes with millions of small files
Too many files in one directory
- Can happen with either mbox (too many users) or maildir (too many
emails for one user)
- Use a hashed directory structure, e.g.
/home/12/34/user
CPU limits
- Be sure that CPU is really the limit!
- use top when server runs slowly, check %cpu user/system, >50%
may be a problem
- Use a graphing/trending tool like munin or
cacti to spot heavy usage times, and correlate
with high disk/CPU/memory usage
- Put in a faster CPU/multiple CPUs (SMP), and more RAM
- Ensure kernel parameters correctly tweaked (sockets, filehandles)
- Distribute the load across multiple boxes - clustering
- Use an efficient MTA and POP/IMAP server
- Use IMAP rather than POP3 for accounts with webmail, and for
large mailboxes
Disk performance
- Use softupdates (FreeBSD) or a high-performance filesystem (Linux)
- Use multiple disks, spread your mail directories across them
- Use pairs of mirrored disks (RAID1), not RAID5
- Distribute the load across multiple boxes - clustering
- Enforce quotas to limit disk space and inodes used by each
customer
Separate SMTP (smarthost) and POP3/IMAP services separate
Separating SMTP and POP3/IMAP on separate machines is an easy way to
scale your mail service. Plan for it by using separate hostnames (e.g.
pop3 and smtp) that may initially point to the same machine.
pop3.example.com |
does not relay, accepts incoming SMTP for delivery to local mailboxes
only |
smtp.example.com |
relays, has no local mailboxes |
There is an additional advantage: mail routing works correctly even if
one of your customers leaves (moves their domain's MX records to point
somewhere else) without telling you.