Secure Spam/Virus filtering system with Debian and MailScanner
Posted by ugob on Wed 29 Jun 2005 at 20:46
Despite the fact that I have more experience with RedHat, I configured a Spam/Virus filtering system on Debian recently and I thought I should share some knowledge with the community.
My package of choice is MailScanner (and its friends) and I thought I could offer some guidance to people who whish to configure a similar system. MailScanner is a very powerful, scalable and robust, open-source e-mail security package. It processes more than 500 million e-mail messages every day, and is used in more than 20,000 sites around the world.
MailScanner scans all e-mail for viruses, spam and attacks against security vulnerabilities. It is not tied to any particular virus scanner, but can be used with any combination of 14 different virus scanners, allowing sites to choose the "best of breed" virus scanner. (http://www.mailscanner.info)
I'll base this article on Sarge, as it was just released and include a rather recent version of the tools we need. The server was running Postfix, so this is the MTA I'll use here as well.
The first thing to do is to make sure your system is up to date, using apt or aptitude. Then, the fun begins:
- Make sure your MTA (postfix) is configured properly for your needs
- Stop the MTA (/etc/init.d/postfix stop)
- Install the packages:
- aptitude install mailscanner clamav dcc-client pyzor razor spamassassin
- Answer the questions the best you can. It shouldn't be too hard and you can always re-configure those settings later if needed
- Install bitdefender (free)
- wget ftp://ftp.bitdefender.com/pub/linux/free/bitdefender-console/en/BitDefender-Console-Antivirus-7.0.1-3.linux-gcc3x.i586.deb
- dpkg -i BitDefender-Console-Antivirus-7.0.1-3.linux-gcc3x.i586.deb
- Run 'freshclam' to update the virus definition for clamav.
- Run 'bdc --update' to update the virus definitions for bitdefender
- Run 'pyzor discover' and 'razor-client discover' to update pyzor's and razor's servers list
- A few tricks are needed since we're using postfix as MTA:
- mkdir /var/spool/MailScanner/spamassassin
- chown postfix:postfix /var/spool/MailScanner/spamassassin
- chown postfix:postfix /var/spool/MailScanner/spamassassin
- Make a copy of your MailScanner.conf file
- cd /etc/MailScanner
- cp MailScanner.conf MailScanner.conf.dist
- Edit your MailScanner.conf file to make sure you set those parameters:
- Run As User = postfix
- Run As Group = postfix
- Incoming Queue Dir = /var/spool/postfix/hold
- Outgoing Queue Dir = /var/spool/postfix/incoming
- MTA = postfix
- Virus Scanners = clamav bitdefender
- Make a copy of your main.cf file
- cd /etc/postfix/
- cp main.cf main.cf.dist
- Edit your main.cf to add this line
- header_checks = regexp:/etc/postfix/header_checks
- Create a file /etc/postfix/header_checks with only this in it:
- /^Received:/ HOLD
For the curious, this tells postfix to accept incoming mail and put it in the hold queue. Then, MailScanner takes the messages there, process them, and then put it back into the incoming queue, so that postfix can deliver them to the recipients .
- In the file /etc/default/mailscanner, make sure this parameter is at 1:
- run_mailscanner=1
- You can now start the system
- /etc/init.d/mailscanner start
- /etc/init.d/postfix start
- Check your logs for errors 'tail -f /var/log/mail.log'
- You can now configure MailScanner by editing /etc/MailScanner/MailScanner.conf
- Want stats? The simplest reporting packages to install on Sarge is Vispan - http://www.while.homeunix.net/
There, you now have a mail filtering system. Every e-mail is scanned by 2 virus engines, by MailScanner for HTML and other vulnerabilities, and by SpamAssassin to filter out spam.
Is it finished yet?
Not really. MailScanner is very powerfull and complex, so you have to learn about what you can do with it and how. The first step is to read MailScanner.conf and do some tests. Also, e-mail security is an ever-evolving topic so you must update your system often, and try to find the more recent version of software. Debian Volatile can help with that, or sometimes you may be better compiling from source.
Then, there is a wiki where you can get a lot of information. I suggest you start by reading the MAQ page (which I inciendally created and maintain), and then go in the documentation section for more in-depth tricks.
BTW, MailScanner can work with Exim, Sendmail, Qmail and Zmailer as well.
If you have any questions, you'll find that the MailScanner mailing list is very helpfull.
Finally, please let me know if you find an error in this procedure and I'd appreciate to have any feedback on this article.
Ugo
chown -R postfix:postfix /var/spool/MailScanner/
chown -R postfix:postfix /var/lib/MailScanner
chown -R postfix:postfix /var/run/MailScanner/
chown -R postfix:postfix /var/lock/subsys/MailScanner
Also, the hyperlink for http://www.while.homeunix.net does not work, it has a "BR" tag at the end of the link. Vispan is a great reporting package. I'll install it later.
[ Parent | Reply to this comment ]
[ Send Message | View Steve's Scratchpad | View Weblogs ]
I've fixed the broken link now, sorry for not spotting this previously.
Steve
-- Steve.org.uk
[ Parent | Reply to this comment ]
chown postfis:postfix /var/spool/MailScanner/spamassassin
How hard is it to setup vispan on Debian?
[ Parent | Reply to this comment ]
[ Send Message | View Steve's Scratchpad | View Weblogs ]
Thanks for that, fixed too now.
As for vispan I guess Ugo would know .. I've never heard of it before!
Steve
-- Steve.org.uk
[ Parent | Reply to this comment ]
- perl -MCPAN -e 'install GD::Graph'
- perl -MCPAN -e 'install Mail::Sendmail'
- perl -MCPAN -e 'install Number::Format'
[ Parent | Reply to this comment ]
Is there a way I can edit directly?
Thanks,
[ Parent | Reply to this comment ]
[ Send Message | View Steve's Scratchpad | View Weblogs ]
Fixed now - Sorry but right now authors cannot edit their pieces. I need to write that still!
All being well it will be possible in the next few days, the change isn't very difficult I don't think.
There are quite a few things that need to be updated in the code, it's just a matter of getting them all done.
Steve
-- Steve.org.uk
[ Parent | Reply to this comment ]
SpamAssassin User State Dir = /var/spool/MailScanner/spamassassin
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
http://wiki.mailscanner.info/doku.php?id=maq:index#anti-virus
Answer to the second question:
Yes, so if you a want system that is completely open, don't use bitdefender.
[ Parent | Reply to this comment ]
I'm set up right now with amavisd-new, which seems to do the same job.
Is mailscanner better?
[ Parent | Reply to this comment ]
I don't know if one is better than the other. I don't think so. MailScanner has limitations and advantages that Amavis hasn't and vice-versa. I never used Amavis, so I can't really tell. I can only tell that I manage ~15 MailScanner servers and they do a really, really good job at keeping the networks spam/virus/(e-mail related) vulnerability exploit free.
They also have a different design. Amavis works as a network daemon and that is how it communicates with the MTA. MailScanner picks up the mail on the filesystem and re-feeds the MTA through the filesystem.
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
See http://wiki.mailscanner.info/doku.php?id=documentation:configurat ion:mta:postfix:politics
[ Parent | Reply to this comment ]
Debian Anti-Spam Anti-Virus Gateway Email Server using Postfix 2.1, Amavisd-new, SpamAssassin, Razor, DCC, Pyzor, and ClamAV
[ Parent | Reply to this comment ]
Jun 30 19:57:37 fea MailScanner[27371]: Messages found but no hashed queue directories. Please enable hashed queues for incoming and deferred with a depth of 1 or 2. See the Postfix documentation for hash_queue_names and hash_queue_depth
I can't seem to figure out how to configure postfix to correct this problem. I think it is postfix because at least it has a hash_queue command appearing in the post_install script. How do I correct this problem ?
[ Parent | Reply to this comment ]
# postconf hash_queue_names hash_queue_depth
hash_queue_names = deferred, defer
hash_queue_depth = 1
So use:
postconf -e "hash_queue_names = deferred, defer, incoming"
postfix stop
postfix start
[ Parent | Reply to this comment ]
However, I tried this and it did not fix the problem. I continue to get those same messages and it is making for a rather large /var/log/mail.log file
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
Ugo
[ Parent | Reply to this comment ]
logfile=/var/spool/postfix/hold/razor-agent.log
which was totally different from the messages sent to the log file and I would never have figured this out. Things seem to be working here now although I still have one annoying warning message each time postfix is started: Jul 2 21:38:52 fea postfix/postsuper[18773]: warning: bogus file name: hold/razor-agent.log Also, there is a good explanation for the apparent postfix/mailscanner issue here:
http://www.sng.ecs.soton.ac.uk/mailscanner/serve/cache/382.html
One thing I have learned about all this: it needs to me much simpler ! This is difficult to implement (even for us Linux hackers) !
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
This is due to the fact that undocumented Postfix internals (e.g. the queue file format) are subject to change. Since these changes cannot be anticipated by the MailScanner author, it's definitely not safe to use.
[ Parent | Reply to this comment ]
But hey, if you don't want to use Postfix, just replace the section about postfix in the article by something exim-specific. It apparently works out-of the box. Or use Amavis. With open-source you have the choice...
Ah, and see http://wiki.mailscanner.info/doku.php?id=documentation:configurat ion:mta:postfix:politics
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
# Still deliver (after cleaning) messages that contained viruses listed
# in the above option ("Silent Viruses") to the recipient?
# Setting this to "yes" is good when you are testing everything, and
# because it shows management that MailScanner is protecting them,
# but it is bad because they have to filter/delete all the incoming virus
# warnings.
#
# Note: Once you have deployed this into "production" use, you should set
# Note: this option to "no" so you don't bombard thousands of people with
# Note: useless messages they don't want!
#
# This can also be the filename of a ruleset.
Still Deliver Silent Viruses = no
If you realy want to notify all the people that didn't send the virus ... change it to yes.
For the MailScanner daemon, I found an interesting article about extending it with a front-end: MailWatch for MailScanner: a PHP web front-end to MailScanner, creates graphics and statistics about spam, virii and other. Does a great job in creating statistics, learning spam to spamassassin, surfing the archives, ...
[ Parent | Reply to this comment ]
I would like to inform the sender that he sended a virus.
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
99,99999% of the viruses are send without the knowledge of the suer, and 99,999% of these viruses the sender isn't the user in the "From:".
Do you still believe it's important for person X that person Y send a virus ... ??? Fred
[ Parent | Reply to this comment ]
I have installed Mailscanner according to this realy nice step-by-step instruction. However i feel MailScanner is not behaving as it should. It often hangs so that I have to restart it (/etc/init.d/mailscanner restart), and then it floods in with messages. Has anybody else experienced that ?
Second question: My postfix is using procmail (mailbox_command = procmail -a "$EXTENSION" ). How is this working together with MailScanner, and should I avoid use of /etc/procmailrc settings for spamassassin ?
[ Parent | Reply to this comment ]
http://wiki.mailscanner.info/doku.php?id=documentation:test_troub leshoot:mailscanner
For the second question, you should avoid using procmail with MailScanner, since MailScanner calls SpamAssassin itself. It might be possible, but you should probably ask the MailScanner list to ask people before. I can't tell since I've never used procmail.
What actions are you performing with procmail? Please let me know and I'll tell you if you can do that directly within MailScanner.
[ Parent | Reply to this comment ]
As far as I can read the procmailrc that comes along with spamassassin as an example, the procmail is doing a pre-selection of mails of small size (less than 256k) and checks these for spam directly. Assuming that viruses are greater size, I guess (which is not nessecery true...?).
:0fw: spamassassin.lock
* < 256000
| spamassassin
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
I got the same problem. Usually mailscanner (or something else) hangs once a day. I suspect that procmail is stoping somehow with large attachments. I could not find any information in the log files.
I disabled procmail now. Let's see what will happen.
[ Parent | Reply to this comment ]
- How does the filtering system work ?
- How can I know if the filtering system works well ? I looked mail.log and I have :
MailScanner[22457]: MailScanner E-Mail Virus Scanner version 4.55.10 starting...
MailScanner[22457]: Read 748 hostnames from the phishing whitelist
MailScanner[22457]: Using SpamAssassin results cache
MailScanner[22457]: Connected to SpamAssassin cache database
MailScanner[22457]: Enabling SpamAssassin auto-whitelist functionality...
MailScanner[22457]: ClamAV scanner using unrar command /usr/bin/unrar
MailScanner[22457]: Using locktype = flock
Does it work ?
Thanks !
[ Parent | Reply to this comment ]