SPAM/Virus Filter on Sarge

Posted by ncb on Fri 3 Mar 2006 at 10:01

This documents my adventure of setting up a spam/virus fighting mail server on Sarge. It is not the most fancy way of doing it, but should be suitable for an environment with not too many users, nor a bunch of road warriors. It's also probably a decent starting point for building more advanced configurations. The end product is a server that dicards virus-infected mail, identifies spam, and sorts it into a local user's junk folder. Courier's Maildrop is used instead of procmail, Maildir directories are used instead of mbox, and the SSL enabled version of Courier IMAP is used for mail retrieval by users.

I should note that there are many other, and far better, howtos out in the world for doing this, but many are more involved and I was shooting for relative simplicity. For example this howto does not cover the setup of SMTP Auth, TLS, or SASL, and so does not allow users that are external to the network to relay mail through the server. For this, a similar but more indepth howto is this article at Fat of the LAN. If however, you want the full-on (and kick-ass) ISP style server, with virtual domains and MySQL to keep track of everything, you'll like this article at workaround.org.

UPDATE: After I initially posted this article, some commenters politely pointed out concerns with the configuration, which I think can be summarized into three main points:

  1. The configuration in this article uses after-queue content filtering, instead of before-queue content filtering. I've not tried before-queue filtering, but some commenters have said it is the newer/better way. You can read more about it on the Postfix Before-Queue Content Filter page. Following is the the Pros and Cons section from that page (the cons sound a bit scary to me, but obviously some are having good results).
    • Pro: Postfix can reject mail before the incoming SMTP mail transfer completes, so that Postfix does not have to send rejected mail back to the sender (which is usually forged anyway). Mail that is not accepted remains the responsibility of the remote SMTP client.
    • Con: The remote SMTP client expects an SMTP reply within a deadline. As the system load increases, fewer and fewer CPU cycles remain available to answer within the deadline, and eventually you either have to stop accepting mail or you have to stop filtering mail. It is for this reason that the before-queue content filter can be used only on low-traffic sites.
    • Con: Content filtering software can use lots of memory resources. In order to not run out of memory you have to reduce the number of before-filter SMTP server processes so that a burst of mail will not drive your system into the ground with too many content filter processes. This, in turn, means that SMTP clients have to wait for a long time before they receive service.

    Note also that if you go the before-queue route, doing it with amavisd-new is not recommended (thanks to jooray for pointing it out).

  2. This configuration tags and sorts all spam to a junk folder, rejecting none of it. The rationale for this choice is to minimize the likelihood of losing wanted mail (especially since no bounce messages are being sent), and also to negate the need for any quarantine. Someone pointed out that anything Spamassassin scores higher than 10 or so is something you'll likely never miss, so accepting it is unnecessary. If this is what you want it is easily remedied. Assuming 10 is your magic number, set amavisd.conf's $sa_kill_level_deflt = 10 and make sure to set $spam_quarantine_to = undef. Anything scored 10 or higher will just go away.
  3. There was also at least one mention of the need to never ever send bounce messages to virus or spam senders. I totally agree. Strictly speaking this configuration didn't do that, but I had stuck with the default of sending a bounce message for banned file attachments. I've never been burned by this, but it's probably a risk one shouldn't take, and so I've changed the article to set amavisd.conf's $final_banned_destiny from the default of D_BOUNCE to D_DISCARD.

Also, I initially left out the step of configuring Postfix's mynetwoks directive, so I added it.

Before you start

  • First off, don't try this on an existing Debian mail server. Installing a new SMTP server package will remove the one you already have, i.e Postfix replaces Exim.
  • Get a firewall: Firehol or Shorewall are good candidates. You'll need to allow TCP port 25 for SMTP, and TCP port 993 for IMAP-SSL.
  • Configure DNS: The FQDN of your server needs to be resolvable via DNS, and the host name you gave your server during the Debian install should match, i.e., both should be something like smtp.example.com. After your new mail server is working you can make it a mail handler for your domain by adding a new MX record to DNS (but don't do that yet!)
  • Synchronize your clock: Mail servers should have an accurate clock. The easiset way is apt-get install ntpdate.
  • Add Debian volatile to sources.list: Clamav is constatntly changing to keep up with viruses, and unless you keep it updated more often than Sarge typically would, you will eventually be annoyed by the message "WARNING: Your ClamAV installation is OUTDATED!" To keep clamav current, add one of the debian volatile mirrors to /etc/apt/sources.list. Like so ...
       deb http://volatile.debian.net/debian-volatile stable/volatile main
       
    Look here for more info on Debian Volatile.

What gets installed

These are the main players and versions being installed at the time of writing (March 2006).

Installing Postfix and Company

This installs postfix, amavisd-new, clamav, and some recommended packages (zoo, unarj, and unzip let amavisd-new access archives). You know the drill . . .

:# apt-get update
:# apt-get upgrade
:# apt-get install postfix amavisd-new spamassassin clamav clamav-daemon clamav-docs zoo unzip unarj 

Some guidance on the ensuing configuration questions ...

  • Choose daemon mode for freshclam (the default)
  • Choose a local mirror from the list for clamav updates
  • Choose Yes for notifying clamd of virus defitions updates (the default)
  • Choose "Internet Site" for Postfix. (the default)
  • At this Postfix config question ...

    Give a comma-separated list of domains that this machine should consider itself the final destination for...

    edit the list to look like something like this.

    smtp.example.com, localhost.example.com, example.com, localhost

Installing Courier-IMAP and Company

Now lets install all the Courier tools, plus some of their recommended packages. This will pull in everything you need for IMAP, including the courier-imap and openssl packages. You can just accept all the defaults on the upcoming config questions.

:# apt-get install courier-maildrop courier-imap-ssl courier-doc ca-certificates fam

Creating Maildir directories

Now we need to create the Maildir directories for incoming mail to land in. We do this with the tool maildirmake, and we do it within the user's home directories. Do the following for existing users.

:# su user
:# cd ~
:# maildirmake Maildir (creates ~/Maildir plus cur, new, and tmp subdirs)
:# maildirmake -f Drafts Maildir (creates ~/Maildir/.Drafts)
:# maildirmake -f Junk Maildir (creates ~/Maildir/.Junk)
:# maildirmake -f Sent Maildir (creates ~/Maildir/.Sent)
:# maildirmake -f Templates Maildir (creates ~/Maildir/.Templates)
:# exit (back to being root)

Now repeat the above process as root in /etc/skel so new users will get Maildir directories automatically.

Configuring Maildrop

Maildrop is a replacement for procmail that includes it's own perl-like filtering language. We can use this filtering language to sort spam to the Junk folder we created. We do this by creating the file /etc/courier/maildroprc and adding some system-wide configuration there. Create the file, and then add the following.

# /etc/courier/maildroprc

# Our shell
SHELL="/bin/bash"

# The default path
DEFAULT = "$HOME/Maildir"

# Our log file
logfile "/var/log/maildrop"

# Our verbosity in the log file
VERBOSE="5"

# This get's added above each entry in the log file.
# It's just a visual aid.
log "------------------------"

# If the Spam-Flag is YES, sort to Junk folder
# ---------------------------------------------
if ( /^X-Spam-Flag: YES/)
{
       to $DEFAULT/.Junk/
}

The above tells maildrop where it's log file is, and also to sort anything with "X-Spam-Flag: YES" in the headers to the Junk folder. Note that users can create their own ~/.mailfilter file to further sort mail. What I have above is a very basic use of maildrop filtering. For more info try man maildropfilter or look here.

We also need to create the log file we mentioned in the maildroprc file above.

:# touch /var/log/maildrop
:# chown root.adm /var/log/maildrop
:# chmod 642 /var/log/maildrop

This log should be rotated. Create /etc/logrotate.d/maildrop and add the following to it.

/var/log/maildrop {
   weekly
   missingok
}

Sanity Checking Postfix

Let's take a look at our default postfix config. Open up /etc/postfix/main.cf in an editor. Look it over and give it a sanity check. Look for your own domain whenever you see example.com. It should look something like the following.

# See /usr/share/postfix/main.cf.dist for a commented, more complete version

smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h

myhostname = smtp.example.com
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = smtp.example.com, localhost.example.com, example.com, localhost
relayhost =
mynetworks = 127.0.0.0/8
mailbox_command = procmail -a "$EXTENSION"
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all

Setting mynetworks

This is a good point to add any networks you want to relay mail for to the mynetworks list. For example, if your local LAN is on network 192.168.10.0/24 do this ...

mynetworks = 127.0.0.0/8, 192.168.10.0/24

Configuring Postfix to use Maildrop

Now let's edit this file so that Postifx will use maildrop (instead of the default procmail), and deliver to our user's Maildirs. First, comment out the existing mailbox_command line.

# mailbox_command = procmail -a "$EXTENSION"

Then add these lines at the bottom of the file.

# Use maildrop instead of procmail
mailbox_command = /usr/bin/maildrop

# Deliver to user's Maildir (trailing slash is required)
home_mailbox = Maildir/

Save the file, and reload Postfix.

/etc/init.d/postfix reload

Testing Postfix and Maildrop

Now, assuming DNS is working, you should be able to send a test mail from another machine. Remember there is no MX record for this box yet, so use the FQDN in the e-mail address (i.e. user@smtp.example.com not user@example.com). To watch the message go through you can tail the mail log while sending it.

:# tail -f /var/log/mail.log

Feb 28 14:44:18 smtp postfix/smtpd[5843]: connect from someserver.somedomain.com[123.456.78.9]
Feb 28 14:44:18 smtp postfix/smtpd[5843]: 43F4C11E65C: client=someserver.somedomain.com[123.456.78.9]
Feb 28 14:44:18 smtp postfix/cleanup[5846]: 43F4C11E65C: message-id=<4404D1BA.2060504@somedomain.com>
Feb 28 14:44:18 smtp postfix/qmgr[5838]: 43F4C11E65C: from=<user@somedomain.com>, size=1414, nrcpt=1 (queue active)
Feb 28 14:44:18 smtp postfix/smtpd[5843]: disconnect from someserver.somedomain.com[123.456.78.9]
Feb 28 14:44:18 smtp postfix/local[5847]: 43F4C11E65C: to=<user@smtp.example.com>, relay=local, delay=0, status=sent (delivered to command: /usr/bin/maildrop)
Feb 28 14:44:18 smtp postfix/qmgr[5838]: 43F4C11E65C: removed

You should also be able to see something like this in /var/log/maildrop

------------------------
Date: Wed Feb  28 14:44:20 2006
From: Some Name <user@example.com>
Subj: testing 
File: /home/user/./Maildir                                            (1531)

Further, you should be able to find the message in the user's new mail folder under a really long file name.

:# ls ~username/Maildir/new
1141137858.M361578P5848V0000000000000301I000578F3_0.smtp,S=1508

If this test fails check the mail.log for errors, check for typos, make sure you reloaded postfix, and look for firewall or DNS issues. This needs to work before you proceed.

Configuring Amavisd-new

Now that we know Postfix and maildrop are working, we want to add spam and virus filtering to the mix. This is where amavisd-new comes in to play. amavisd-new acts as the glue between postifx and content checkers such as spamassassin and clamav. We need to configure it, and after that we'll configure postifx to forward mail through it.

The config file for amavisd-new is /etc/amavis/amavisd.conf. This is a huge (and a bit daunting) config file, with lots of options and comments. There are a number of changes to make, so I'll just go right down the file and make some brief explanations along the way.

Before we start, here is an overview of what we want amavisd-new to do for us:

  • Check mail for viruses using clamd (or clamscan if clamd is unavailable)
  • If a mail is infected with a virus, don't quarantine it but discard it instead. Let the postmaster know what happened, and don't try to notify the almost certainly spoofed sender.

    Note: The reason I don't want to quarantine btw, is that I don't want to manage what will quickly become a huge archive of virus-infected messages from spoofed sender addresses. My opinion is that if the message came from a legitimate sender, and is really important, then the sender can re-send it after cleaning their system, or send a damned fax. The default behaviour, however, is to quarantine. If you decide to quarantine I suggest a cron job that regularly deletes infected files from the quarantine that are over some reasonable age, like 2 weeks for example.

  • Just like with a virus, if a mail has a banned filename attached, discard it, let the postmaster know, and do not try to notify the sender.
  • Assign spam scores to each message using spamassassin.
  • If the spam score is above a certain level (default is 6.3), prepend the string "***SPAM***" to the subject, add "X-Spam-Flag:YES" to the headers, and pass the message back to Postfix (eventually getting sorted to user's Junk folder by maildrop).
  • Don't notify the postmaster of every spam message.

OK, open up amavisd.conf in an editor and let's get started. First, set the domain and host names. Find these lines and change them.

$mydomain = 'example.com';      # (no useful default) -->
$myhostname = 'smtp.example.com';  # fqdn of this host, default by uname(3)

Uncomment the following two lines, these tell amavisd-new on what port to look for Postfix (port 10025, we'll configure postifx for this a bit later).

$forward_method = 'smtp:127.0.0.1:10025';  # where to forward checked mail
$notify_method = $forward_method;          # where to submit notifications

Comment out this line. Debian's amavisd-new does not use spamassassin by defualt, but we want it.

# @bypass_spam_checks_acl  = qw( . );    # No default dependency on spamassassin

The following 4 parameters tell amavisd-new what to do with mail that is virus infected or spam. They are somewhat self explanatory, but reading the notes just above these in amavisd.conf will give you a good idea of how they work. Set them like so:

$final_virus_destiny      = D_DISCARD; # (defaults to D_BOUNCE)
$final_banned_destiny     = D_DISCARD;  # (defaults to D_BOUNCE)
$final_spam_destiny       = D_PASS;  # (defaults to D_REJECT)
$final_bad_header_destiny = D_PASS;  # (defaults to D_PASS), D_BOUNCE suggested

Note: D_BOUNCE is the default for $final_banned_destiny, which I've found useful and never had ill effects from. Setting it to D_DISCARD however will ensure you are sending no automatic bounce messages, and is probably the safest way to go.

The $virus_admin variable sets the email address that will receive notifications of virus mails. No change is needed on this line unless you want this to be someone other than postmaster.

$virus_admin = "postmaster\@$mydomain";         # due to D_DISCARD default

We don't want to keep infected mail, and are going to pass tagged spam on to the user. So, there's nothing to quarnatine. Make the following few changes.

Find the line defining the $QUARANTINEDIR. Comment out the first line here, and then add the second one.

# $QUARANTINEDIR = '/var/lib/amavis/virusmails';
$QUARANTINEDIR = undef;

Then, comment out this line.

# $virus_quarantine_to  = 'virus-quarantine';    # traditional local quarantine

And then uncomment this line.

$virus_quarantine_to = undef;

If you want to adhere to the principle that postmaster should be able to receive all mail (except viruses), find and uncomment the following line.

$spam_lovers{lc("postmaster\@$mydomain")} = 1;

The $sa_local_tests_only variable controls whether Spamassassin can do checks that require it to access the internet. To allow it, change the value from 1 to 0.

$sa_local_tests_only = 0;   # (default: false)

The next three settings determine at what Spamassassin score a message's headers will be altered. Set them like so.

$sa_tag_level_deflt  = -999; # add spam info headers if at, or above that level
$sa_tag2_level_deflt = 6.3; # add 'spam detected' headers at that level
$sa_kill_level_deflt = 999; # triggers spam evasive actions

Here's a brief explanation of these settings.

$sa_tag_level_deflt = -999;
Setting this to a big negative number causes every message to be tagged as either spam or not (X-Spam-Status:Yes/No in the mail headers).
$sa_tag2_level_deflt = 6.3
This is the spamassassin score at which a message is considered to be spam, the string "***SPAM***" will be prepended to the subject line of a message, and "X-Spam-Flag:YES" is added to the headers. You can fine tune this number over time. I eventualy wound up at 5.0.
$sa_kill_level_deflt
This is the spamassassin score at which actions such as DISCARD or QUARANTINE will occur for spam. When taking such actions, this is set to the same value as $sa_tag2_level_deflt, but since we are just tagging spam we can set it very high.

Note: Over time you may decide that every message scoring above a certain level is definitely spam and should be discarded. This can greatly reduce the amount of spam in your user's junkf folder, and may decrease your exposeure to any virus clamav might have missed. If so you can tune the $sa_kill_level_deflt to a lower number, like somewhere between 10 or 20 for example.

Now go down to the section beginning with @av_scanners = (. Here you will find stanza after stanza defining the various virus scanners amavisd-new will look for at start up (several can be used at the same time). Following that is another large section defining the backup scanners that will be used if none of the primary scanners can be found. Personally, I don't like the idea of churning through all this at every startup, so I just delete every stanza I don't need leaving only those related to clamd and clamscan. But that's just me, and leaving this section alone entirely or just commenting out the sections you don't need is fine too. Anyway, below is what it looks like when I'm through with it.

@av_scanners = (

### http://www.clamav.net/
['Clam Antivirus-clamd',
  \&ask_daemon, ["CONTSCAN {}\n", "/var/run/clamav/clamd.ctl"],
  qr/\bOK$/, qr/\bFOUND$/,
  qr/^.*?: (?!Infected Archive)(.*) FOUND$/ ],
# NOTE: run clamd under the same user as amavisd;  match the socket
# name (LocalSocket) in clamav.conf to the socket name in this entry
# When running chrooted one may prefer: ["CONTSCAN {}\n","$MYHOME/clamd"],
);

# If no virus scanners from the @av_scanners list produce 'clean' nor
# 'infected' status (e.g. they all fail to run or the list is empty),
# then _all_ scanners from the @av_scanners_backup list are tried.
# When there are both daemonized and command-line scanners available,
# it is customary to place slower command-line scanners in the
# @av_scanners_backup list. The default choice is somewhat arbitrary,
# move entries from one list to another as desired.

@av_scanners_backup = (

  ### http://www.clamav.net/
  ['Clam Antivirus - clamscan', 'clamscan',
    "--stdout --no-summary -r --tempdir=$TEMPBASE {}", [0], [1],
    qr/^.*?: (?!Infected Archive)(.*) FOUND$/ ],
);

That's it for amavisd.conf. Save and close the file, but don't reload amavisd-new quite yet.

One Last Thing

The notes in the above clamd stanza are important. amavisd-new and clamd both need to agree on the location of clamd's socket file (/var/run/clamav/clamd.ctl), and both need permission to see it. The file /usr/share/doc/amavisd-new/NEWS.Debian tells us how to handle this, and indeed nearly everything is already taken care of. If you look at /etc/clamav/clamd.conf you will see that clamd's LocalSocket setting already matches what is in amavisd.conf, that clamd runs as user clamav, and the setting AllowSupplementaryGroups is active. So, all that's left to do is to add the clamav user to the amavis group.

:# adduser clamav amavis
Adding user `clamav' to group `amavis'...
Done.

Testing Amavisd-new

I can't remember what was required here, just a reload, a restart, or what. So let's just do full restarts on both amavisd-new and clamd.

:#/etc/init.d/clamav-daemon restart
Stopping ClamAV daemon: clamd
Starting ClamAV daemon: clamd
:# /etc/init.d/amavis restart
Stopping amavisd: amavisd-new.
Starting amavisd: amavisd-new.

That oughta do it. Check /var/log/syslog and /var/log/clamav/clamav.log for errors. If all is well we should be able to telnet to amavisd-new on port 10024.

:# telnet 127.0.0.1 10024
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
220 [127.0.0.1] ESMTP amavisd-new service ready
quit
221 2.0.0 [127.0.0.1] (amavisd) closing transmission channel
Connection closed by foreign host.

Configuring Postfix to use Amavisd-new

OK, now we just need amavisd-new and postfix to start working together. We need to edit two postfix config files (More info on the following steps is in /usr/share/doc/amavisd-new/README.postfix).

Open up /etc/postfix/master.cf, paste the following at the end of the file, then save and close.

# Added per /usr/share/doc/amavisd-new/README.postfix
smtp-amavis unix -      -       n     -       2  smtp
    -o smtp_data_done_timeout=1200
    -o smtp_send_xforward_command=yes
    -o disable_dns_lookups=yes

127.0.0.1:10025 inet n  -       n     -       -  smtpd
    -o content_filter=
    -o local_recipient_maps=
    -o relay_recipient_maps=
    -o smtpd_restriction_classes=
    -o smtpd_client_restrictions=
    -o smtpd_helo_restrictions=
    -o smtpd_sender_restrictions=
    -o smtpd_recipient_restrictions=permit_mynetworks,reject
    -o mynetworks=127.0.0.0/8
    -o strict_rfc821_envelopes=yes
    -o smtpd_error_sleep_time=0
    -o smtpd_soft_error_limit=1001
    -o smtpd_hard_error_limit=1000

Then, open /etc/postfix/main.cf, add the following at the end of the file, then save and close.

# Added per /usr/share/doc/amavisd-new/README.postfix
content_filter=smtp-amavis:[127.0.0.1]:10024

Now reload postfix, (tail -f /var/log/mail.log to watch for errors).

:# /etc/init.d/postfix reload

Testing it All Together

Alrighty then, let's see if this really works. We'll use telnet, the EICAR virus test string, and the GTUBE spam test string. These strings are handily provided in /usr/share/doc/amavisd-new/README.postfix and /usr/share/doc/spamassassin/examples/sample-spam.txt.

Let's telnet to port 10024, and have an SMTP conversation like the following. Note, however, that the empty line which precedes both the EICAR and GTUBE strings is needed, as is the following period on the last line. Hitting enter after the final period sends the message. Also, it's important to use a real user, and your own domain in place of example.com.

Note: You should get similar results using port 25, but postfix will not give you as much info.

:# telnet 127.0.0.1 10024
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
220 [127.0.0.1] ESMTP amavisd-new service ready
MAIL FROM:<test@somedomain.com>
250 2.1.0 Sender test@somedomain.com OK
RCPT TO:<realuser@example.com>
250 2.1.5 Recipient realuser OK
DATA
354 End data with <CR><LF>.<CR><LF>
Subject: test virus

X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*
.
250 2.7.1 Ok, discarded, id=01336-01 - VIRUS: Eicar-Test-Signature
quit
221 2.0.0 [127.0.0.1] (amavisd) closing transmission channel
Connection closed by foreign host.

The virus was discarded. That's good. Now let's try a spam messasge.

:# telnet 127.0.0.1 10024
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
220 [127.0.0.1] ESMTP amavisd-new service ready
MAIL FROM:<test@somedomain.com>
250 2.1.0 Sender test@somedomain.com OK
RCPT TO:<realuser@example.com>
250 2.1.5 Recipient realuser OK
DATA
354 End data with <CR><LF>.<CR><LF>
Subject: test spam

XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X
.
250 2.6.0 Ok, id=01337-01, from MTA: 250 Ok: queued as 69F6811E65D
quit
221 2.0.0 [127.0.0.1] (amavisd) closing transmission channel
Connection closed by foreign host.

The spam was queued, which is also good, since we are passing it along.

After the above two SMTP sessions, the postmaster should have a virus notification message in ~/Maildir/new/ with something like the following in it:

A virus (Eicar-Test-Signature) was found.

Scanner detecting a virus: Clam Antivirus-clamd

The mail originated from: <test@somedomain.com>

There should also be a message in the user's ~/Maildir/.Junk/new/ folder with ***SPAM*** prepended to the subject, and X-Spam-Flag:YES appended to the headers. The bottom part should look like this.

Subject: ***SPAM*** test spam
Message-Id: <20060302231720.F267411E65D@smtp.example.com>
Date: Thu,  2 Mar 2006 15:17:20 -0800 (PST)
From: test@somedomain.com
To: undisclosed-recipients: ;
X-Virus-Scanned: by amavisd-new-20030616-p10 (Debian) at example.com
X-Spam-Status: Yes, hits=998.7 tagged_above=-999.0 required=6.3
        tests=ALL_TRUSTED, GTUBE, NO_REAL_NAME, UNDISC_RECIPS
X-Spam-Level: ****************************************************************
X-Spam-Flag: YES

XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X

Now, isn't that cool?

If all that worked, and your DNS and firewall cooperate, you should now be able to send mail to any real user at your machine's FQDN (user@smtp.example.com) from a machine outside your network, like a yahoo or gmail account for example.

Configuring Courier-IMAP-SSL

Now let's get IMAP working so we can check our mail. We're using SSL so the first thing we need is a cert. First move the existing imapd.pem out of the way, and make a backup copy of imapd.cnf.

#: cd /etc/courier
#: mv imapd.pem imapd.pem.original
#: cp imapd.cnf imapd.cnf.original

Now edit imapd.cnf to look something like the following, putting in the correct information for city, state, etc, and making sure to set the CN to the FQDN of your server. Then save and close.

RANDFILE = /usr/lib/courier/imapd.rand

[ req ]
default_bits = 1024
encrypt_key = yes
distinguished_name = req_dn
x509_extensions = cert_type
prompt = no

[ req_dn ]
C=US
ST=WA
L=Tacoma
O=Example.com
OU=Example.com SMTP Server
CN=smtp.example.com
emailAddress=postmaster@example.com


[ cert_type ]
nsCertType = server

Now just run the command mkimapdcert with no arguments. Your new cert will be created as /etc/courier/imapd.pem, and will be good for a year.

:# mkimapdcert
Generating a 1024 bit RSA private key
........++++++
.......++++++
writing new private key to '/usr/lib/courier/imapd.pem'
-----
1024 semi-random bytes loaded
Generating DH parameters, 512 bit long safe prime, generator 2
This is going to take a long time
....+......................................+........+.......+..................+..............+...+.............................+..............................+...................+......+........+.............+.....+..................+.....................+......+...........................................+......................+......................................................+...............................+.+............................+....................................+...+..............+.+...............................................................+..+.+.......................+...+...........................+..............+..+.........................+................+.................................................+.+....................................+..+..............+........................................................................................................+.........................................................+.........................++*++*++*++*++*++*
subject= /C=US/ST=WA/L=Tacoma/O=Example.com/OU=Example.com Mail Server/CN=smtp.example.com/emailAddress=postmaster@example.com
notBefore=Mar  3 01:42:28 2006 GMT
notAfter=Mar  3 01:42:28 2007 GMT

Now restart courier-imap.

:# /etc/init.d/courier-imap restart
Stopping Courier IMAP server: imapd.
Starting Courier IMAP server: imapd.
:# /etc/init.d/courier-imap-ssl restart
Restarting Courier IMAP-SSL server: done.

Checking Your Mail

Now all you should have to do is create an account in a mail client, point it at smtp.example.com, and tell it to use IMAP. Don't forget that little check box about using SSL (port 993). The first time you connect, your mail client will whine about the self-signed cert, but you can tell it to remember it permanently (or get a real cert). Also, you may need to tell your mail client to subscribe to all the Maildir folders we created. If you're using Thunderbird, just right-click on the account, select Subscribe..., and check all the little boxes.

That is all. Hope it helps someone.

 

 


Posted by parsingphase (87.74.xx.xx) on Fri 3 Mar 2006 at 12:22
[ View Weblogs ]
See also http://www.phase.org/journal/byjid/8550 for an alternative (if slightly older) solution.

[ Parent | Reply to this comment ]

Posted by Anonymous (84.194.xx.xx) on Fri 3 Mar 2006 at 12:37
I'm using the same software for virtual domains, but stored in MySQL and with vacation. Otherwise you never have vacation :)))

Postfix, Courier, MySQL, Vacation mini-HOWTO

Fred
Linox.be

[ Parent | Reply to this comment ]

Posted by Anonymous (200.125.xx.xx) on Fri 3 Mar 2006 at 12:57
Do you try DSPAM?

Packages for debian here: http://pkg-dspam.alioth.debian.org/ and dspam website here: http://dspam.nuclearelephant.com/

[ Parent | Reply to this comment ]

Posted by ncb (66.224.xx.xx) on Fri 3 Mar 2006 at 18:45
Looks very cool. Thanks for the tip, maybe next time around ...

[ Parent | Reply to this comment ]

Posted by JulienV (86.213.xx.xx) on Sat 18 Mar 2006 at 15:46
[ View Weblogs ]

I'd be very happy to give a try to dspam but cannot find any up-to-date backport for Sarge. I can't build it because of a build-dependency to libmysqlclient... which isn't available on Sarge. I don't want to loose my time if somebody has already done the job!
Any hint?

Cheers,
Julien

[ Parent | Reply to this comment ]

Posted by JulienV (86.213.xx.xx) on Sat 18 Mar 2006 at 20:39
[ View Weblogs ]

well, backporting dspam was in fact really easy, I just had to revert build-dep back to libmysql14-dev.

My backported packages are available from my repository.

Cheers,
Julien

[ Parent | Reply to this comment ]

Posted by Anonymous (15.195.xx.xx) on Fri 3 Mar 2006 at 14:48
I wanted to do something similar, but I don't know how to use maildrop with postfix virtual users... do you have any ideas?

[ Parent | Reply to this comment ]

Posted by ncb (66.224.xx.xx) on Fri 3 Mar 2006 at 17:00

I've not done it, but perhaps these two links can help. The first one is an overview of virtual hosting on Postfix. I followed a link from it to the second one, which specifically talks about Postfix + Maildrop.

Postfix Virtual Domain Hosting

Postfix + Maildrop Howto

Hope that helps.

[ Parent | Reply to this comment ]

Posted by bezell (66.92.xx.xx) on Fri 3 Mar 2006 at 15:21
For a quick and easy solution, try ASSP (http://assp.sourceforge.com). One can install and configure it in 5m. It has an extremely well commented web based configuration page as well.

ASSP is a simple perl script (a single file, assp.pl) that as acts as a proxy between your mail server and the wild. It uses the clamav antivirus, performs greylisting, whitelisting, RBL, HELO and bayesian checks--just about everything short of razor/pyzor. Users can update the serverwide bayesian filter by forwarding the messages to an address like nospam/spam@example.com. It is server independent and can run with any server, including MS Exchange. Enable/Disable Couple it with a daemon-monitoring-daemon like monit or through xinetd for added reliablility.

[ Parent | Reply to this comment ]

Posted by Anonymous (84.50.xx.xx) on Fri 3 Mar 2006 at 20:07
IMHO there is a much better way to do it. Rationale: why should you accept SPAM and viruses into your system tag them and let users sort them, when you should be rejecting them outright? The key is rejecting them at SMTP level and not queueing and discarding or tagging them.

The components are all good - ie. postfix, amavis, sa, clamav, but the setup should be tweaked. Postfix supports before-queue proxys. I'm running a 100 user mailboxes and can assure that its good to reject viruses and completely safe to reject everything that SA w/ network tests gives more than 10 points. At SMTP level of course. Even if you happen to have that 1 in 100 000 mis-classified e-mail, it will not be silently discarded or filed into obscure spam folder - the sender will be notified.

Also, never ever automatically bounce spam of virus mails. The from: field is always forged and some poor schmuck will get all the heat, and you might even get blacklisted yourself.

The changes? The key is smtpd_proxy_filter keyword.

Simple:
--- master.conf
smtp inet n - n - 10 smtpd
-o smtpd_proxy_filter=127.0.0.1:10024
-o smtpd_client_connection_count_limit=5

127.0.0.1:10025 inet n - n - - smtpd
-o smtpd_authorized_xforward_hosts=127.0.0.0/8
-o smtpd_client_restrictions=
-o smtpd_helo_restrictions=
-o smtpd_sender_restrictions=
-o smtpd_recipient_restrictions=permit_mynetworks,reject
-o smtpd_data_restrictions=
-o mynetworks=127.0.0.0/8
-o receive_override_options=no_unknown_recipient_checks



another tip: safe and good spam filter- don't accept foo@nonexistant.no
--- main.conf
smtpd_client_restrictions = reject_unknown_sender_domain
unknown_address_reject_code = 550




frn, debian admin

[ Parent | Reply to this comment ]

Posted by ncb (66.224.xx.xx) on Fri 3 Mar 2006 at 22:31

That's very interesting food for thought. I have comments...

snip ...
why should you accept SPAM and viruses into your system tag them and let users sort them, when you should be rejecting them outright? The key is rejecting them at SMTP level and not queueing and discarding or tagging them.
snip...

To be clear, this setup I outlined discards viruses, does all the sorting of spam for the user, and nothing is quarantined. My rationale for tagging spam and passing it on is to avoid those instances where wanted mail is lost or quarantined when a user really does want that spammy mail from a certain source. This does happen (NRA mailing lists, and users who know a mortgage broker, for example.)

The idea is to sort spam to a folder a user will only look at when wanted mail is somehow getting tagged as spam (at which point a per_recipient whitelist could come into play). MUA's can be set to delete the spam at some reasonable interval so it doesn't build up. The key points are that there is very little chance of losing wanted mail, user's don't have to wade through spam to read their good mail, and I don't have to deal with a quarantine.

Still I know that's not what you meant by "accept", and your point is good. Why let all this crap into the queue and force your server to deal with it? If I understand correctly, your setup would reject at the SMTP level, leaving the responsibility of notifying the sender with the connecting client. That's a cool idea.

A wanted spammy mail would still be lost (from my users point of view), but the sender could still be whitelisted. I didn't know postfix could work this way with Amavis. Thanks for pointing it out.

From a quick googling it does look like there are some caveats. For example this page talks about the need to reduce your number of SMTP sessions to keep your server from "running into the ground with too many content filter processes". For a small site though, I doubt that's a real problem.

Also note, according to that same page, that unless you are running Postfix 2.2 that smtpd_client_connection_count_limit=5 is being ignored. Sarge's Postfix is 2.1.5-9.

[ Parent | Reply to this comment ]

Posted by Anonymous (84.50.xx.xx) on Sat 4 Mar 2006 at 00:07
: To be clear, this setup I outlined discards viruses,
I've noticed that this tends to be a problem with users. Scenario: Users GF sends a .doc file with a macro virus and it will be discarded silently. GF correctly assumes mail went trhu. Recipient never receives it nor notice about it beeing discarded.
But if this mail was to be rejected at SMTP level, GF would get a mailer message about mail beeing undeliverable because "550 VIRUS" and will call the user and ask why her e-mail does not work.

:This does happen (NRA mailing lists, and users who know a mortgage broker, for example.)
Well, i'm from non-english speaking country, so everything above SA 5 points is spam for me. But if you reject from 20 points up, you will not ever lose legit mail even from NRA, brokers etc..) You can still tag mail below that.

My server has been running more than 5 years with about 100 users, and only one user contacted me after i set up SA and clamav and wanted to be unfiltered. You can put those users into amavis config under "spam lovers" or something, and they will get all the spam they've ordered. Anyway, that same user contacted me again after 2 months and asked to be on the filtered list again : )


:From a quick googling it does look like there are some caveats. For example this page talks about the need to reduce your number of SMTP sessions to keep your server from "running into the ground with too many content filter processes". For a small site though, I doubt that's a real problem.
That depends. My 450MHz 256MB celeron handles the load very well. If you have only 128MB of RAM, you probably will get problems.


:Also note, according to that same page, that unless you are running Postfix 2.2 that smtpd_client_connection_count_limit=5 is being ignored. Sarge's Postfix is 2.1.5-9.
Im actually still running woody with some backports. If it ain't broken, don't fix it.

frn

[ Parent | Reply to this comment ]

Posted by simonw (84.45.xx.xx) on Sat 4 Mar 2006 at 11:18
[ View Weblogs ]
What anonymous said... ;)

The Postfix greylisting article I posted here earlier rejects common virus extensions, and 98.99% of our junk email (except backscatter - see Postfix site for advice on backscatter), and requires only a 3 line change to main.cf, with no mysterious folders for genuine email to go unread in.

Afraid I thought the whole config here over complex, and the wrong approach.

The reason not to do complex Bayesian spam checks, and complex virus inspection, before queuing is purely performance. A big ISP can put through more email by queuing it at peak times, as long as the spam/virus checks can complete the load over a 24 hour cycle, then they can get away with less hardware. This was a key target area for Postfix when it was written, so most of the early advice is about queuing first, filtering second.

However the queue first, filter second model, is just out of date, and even big ISPs deploy better upfront filterings (you can still do some post queue checks if you keep the queues small, and the resulting backscatter smaller).

The assumption you can reliably identify viruses, and thus bin them, is definitely a bad choice. Even if the virus scanners were 100% accurate, one day the environment it is in will get messed up and email will be lost. If the only decisions are accept/reject, then all that would happen is a reject of good email, if the decision is accept/reject/delete eventually something will get deleted in error.

Since the end user desktop (well vulnerable desktops) must have virus protection to avoid the case of encrypted or otherwise obscured viruses getting through, I've always regarded virus checking in the MTA as misplaced. Especially since it often leads to a bottleneck.

[ Parent | Reply to this comment ]

Posted by ncb (66.224.xx.xx) on Sat 4 Mar 2006 at 16:38
Ok y'all. Your comments are humbly received. This setup has worked quite well for me, but I'm not marreid to it. Since every single comment has pointed out a different or better wayto do it I think I'll be re-visiting the drawing board :)

[ Parent | Reply to this comment ]

Posted by Steve (82.41.xx.xx) on Sat 4 Mar 2006 at 19:40
[ View Steve's Scratchpad | View Weblogs ]

Hey don't worry.

Even if there are better, or different, approaches for other people if you have a setup that works for you, then you keep it!

My setup is very simple:

  • Exim4
  • Drop known viruses at SMTP time.
  • Delivering to procmail for all users.

The procmail setup uses pyzor to check for bad mail, and automatically sends identified mail to be learnt as spam.

After pyzor the spambayes bayasian filter runs and tags anything it things are spam. (This is trained automatically by pyzor-mails as mentioned).

Finally all mail is delivered locally to users.

The setup isn't ideal, but it works well for me.

[ Parent | Reply to this comment ]

Posted by simonw (84.45.xx.xx) on Sat 4 Mar 2006 at 21:27
[ View Weblogs ]
I just want to evangelize what I think are the best practices. As Steve says if it works for you that is good, but I'm always seeking the simplest ways of achieving the best outcomes.

Also different things work for different situations, I think greylisting is great, but if you routinely deal with lots of different people in a time critical fashion than greylisting isn't for you (although perhaps Instant Messaging is better that email in such environments). Greylisting is also easily circumvented, so having the ability to filter spam before accepting it with a tool like spam assassin is probably useful for when the spammers decide to try and defeat greylisting.

Although when I last checked my log files, I saw a number of cases of;

IP not in blocklist
IP SMTP connection greylisted
IP tries again but is now in blocklist

(realtime block lists - don't you love em).

[ Parent | Reply to this comment ]

Posted by jooray (85.216.xx.xx) on Mon 6 Mar 2006 at 02:03
from amavisd-new homepage:

The Postfix Before-Queue Content Filter setup, also known as smtpd_proxy setup, is not a supported or recommended setup with amavisd-new, which is not a transparent SMTP proxy by design. See caveats in README_FILES/SMTPD_PROXY_README. This setup might work amavisd-new for low-traffic sites which do not use authentication, but is not recommended.

I have had several problems using this setup with Sarge's amavisd-new.

[ Parent | Reply to this comment ]

Posted by jooray (85.216.xx.xx) on Mon 6 Mar 2006 at 02:04
just one more comment -- if it worked without problems, rejecting on smtp level is the best way to do it of course. but it's not recommended and not working well either :(.

[ Parent | Reply to this comment ]

Posted by Anonymous (84.58.xx.xx) on Sat 4 Mar 2006 at 19:14
dont forget to configure your firewall to reset ident request (tcp/113) - if the firewall silently drop these you will have a small delay in the mail delivery.

regards

gnjb

[ Parent | Reply to this comment ]

Posted by ncb (66.224.xx.xx) on Mon 6 Mar 2006 at 15:48
I added a section labled UPDATE: near the top of this article today. It addresses some of the concerns raised in the comments, and makes the reader aware of some configuration choices/tweaks they have to accept less spam and eliminate all bounce messages. I also added some small notes in the body of the article to point out these choices where I thought it was appropriate.

It's still an "after-queue" filter, but now the article points out to readers from the start that they have a choice in this regard, and I think overall it's a little better.

[ Parent | Reply to this comment ]

Posted by SwampThing (62.79.xx.xx) on Sat 25 Mar 2006 at 23:28
I have another setup, also with clamav and spamassassin, but instead I use exim.

Take a look at http://swampthing.dk/local_exim.php.

Using my status page at http://swampthing.dk/local_status.php I have added statistics on how much spam and virus I get daily, and how much I need to cleanup at some point in time.

Brgds
Torben

[ Parent | Reply to this comment ]

Posted by Anonymous (65.61.xx.xx) on Thu 6 Apr 2006 at 16:42
This howto is great and has helped me "wade through" some of the stumbling blocks I came upon while configuring this kind of setup. However, now (not sure as of when (2.3.3)) there is no /etc/amavis/amavisd.conf. There is now a /etc/amavis/conf.d/ directory containing various individual configuration files. While somewhat descriptive in name it is unclear what aspect some of these files are controlling. Also, I could only find about half of the mentioned variables mentioned in this article throughout these various files.

I hate to ask but would it be possible to update this howto to reflect these changes as it has been a valuable resource for myself? In the meantime I am still digging through attempting to figure out a (my) translation of the differences between the variables mentioned in the article.

Thank you again for a clearly laid out document.

[ Parent | Reply to this comment ]

Posted by ncb (66.224.xx.xx) on Thu 6 Apr 2006 at 21:10

Funny you should ask...

Since receiving all the constructive feedback on this article, I've done little else but play with mail servers and re-evaluate my approach. What I came up with basically uses this configuration in the post-queue stage, but also adds postfix's smtpd_*_restrictions, greylisting, SPF checks, and RBL checks in the pre-queue stage. I tried it both with standard Sarge packages, and also with backports of everything available from backports.org. I wrote a howto as I went along which is only just now complete (more or less). It has some general guidance on the differences in the config required by the backports, including amavisd-new. You can find it here.

[ Parent | Reply to this comment ]

Posted by Anonymous (65.61.xx.xx) on Fri 7 Apr 2006 at 21:28
You Rock! ...again - Thanks

[ Parent | Reply to this comment ]

Posted by ncb (66.224.xx.xx) on Tue 11 Apr 2006 at 00:23
Well thanks! FYI, I thought the section on configuring the backported amavisd-new needed some beefing up, so I added some more on that today.

[ Parent | Reply to this comment ]

Posted by Anonymous (212.165.xx.xx) on Tue 13 Jun 2006 at 07:13
classsic!

[ Parent | Reply to this comment ]

Posted by BlaecHrim (192.89.xx.xx) on Tue 27 Jun 2006 at 13:59
Some useful hints and how-tos conserning postfix can be found at http://www.thoron.iki.fi/postfix/

[ Parent | Reply to this comment ]

Posted by Anonymous (81.183.xx.xx) on Tue 27 Jun 2006 at 15:12
Off: for exim powerusers. You dont need to use amavisd if you already have exim4-heavy with built-in exiscan hence exim can call directly clamav by it. Sorry if you have already mentioned it somewhere here...

[ Parent | Reply to this comment ]

Sign In

Username:

Password:

[Register|Advanced]

 

Flattr

 

Current Poll

What do you use for configuration management?








( 496 votes ~ 5 comments )

 

 

Related Links