Secure Spam/Virus filtering system with Debian and MailScanner

Posted by ugob on Wed 29 Jun 2005 at 20:46

Despite the fact that I have more experience with RedHat, I configured a Spam/Virus filtering system on Debian recently and I thought I should share some knowledge with the community.

My package of choice is MailScanner (and its friends) and I thought I could offer some guidance to people who whish to configure a similar system. MailScanner is a very powerful, scalable and robust, open-source e-mail security package. It processes more than 500 million e-mail messages every day, and is used in more than 20,000 sites around the world.

MailScanner scans all e-mail for viruses, spam and attacks against security vulnerabilities. It is not tied to any particular virus scanner, but can be used with any combination of 14 different virus scanners, allowing sites to choose the "best of breed" virus scanner. (http://www.mailscanner.info)

I'll base this article on Sarge, as it was just released and include a rather recent version of the tools we need. The server was running Postfix, so this is the MTA I'll use here as well.

The first thing to do is to make sure your system is up to date, using apt or aptitude. Then, the fun begins:

  • Make sure your MTA (postfix) is configured properly for your needs
  • Stop the MTA (/etc/init.d/postfix stop)
  • Install the packages:
    • aptitude install mailscanner clamav dcc-client pyzor razor spamassassin
  • Answer the questions the best you can. It shouldn't be too hard and you can always re-configure those settings later if needed
  • Install bitdefender (free)
    • wget ftp://ftp.bitdefender.com/pub/linux/free/bitdefender-console/en/BitDefender-Console-Antivirus-7.0.1-3.linux-gcc3x.i586.deb
    • dpkg -i BitDefender-Console-Antivirus-7.0.1-3.linux-gcc3x.i586.deb
  • Run 'freshclam' to update the virus definition for clamav.
  • Run 'bdc --update' to update the virus definitions for bitdefender
  • Run 'pyzor discover' and 'razor-client discover' to update pyzor's and razor's servers list
  • A few tricks are needed since we're using postfix as MTA:
    • mkdir /var/spool/MailScanner/spamassassin
    • chown postfix:postfix /var/spool/MailScanner/spamassassin
    • chown postfix:postfix /var/spool/MailScanner/spamassassin
  • Make a copy of your MailScanner.conf file
    • cd /etc/MailScanner
    • cp MailScanner.conf MailScanner.conf.dist
  • Edit your MailScanner.conf file to make sure you set those parameters:
    • Run As User = postfix
    • Run As Group = postfix
    • Incoming Queue Dir = /var/spool/postfix/hold
    • Outgoing Queue Dir = /var/spool/postfix/incoming
    • MTA = postfix
    • Virus Scanners = clamav bitdefender
  • Make a copy of your main.cf file
    • cd /etc/postfix/
    • cp main.cf main.cf.dist
  • Edit your main.cf to add this line
    • header_checks = regexp:/etc/postfix/header_checks
  • Create a file /etc/postfix/header_checks with only this in it:
    • /^Received:/ HOLD

For the curious, this tells postfix to accept incoming mail and put it in the hold queue. Then, MailScanner takes the messages there, process them, and then put it back into the incoming queue, so that postfix can deliver them to the recipients .

  • In the file /etc/default/mailscanner, make sure this parameter is at 1:
    • run_mailscanner=1
  • You can now start the system
    • /etc/init.d/mailscanner start
    • /etc/init.d/postfix start
  • Check your logs for errors 'tail -f /var/log/mail.log'
  • You can now configure MailScanner by editing /etc/MailScanner/MailScanner.conf
  • Want stats? The simplest reporting packages to install on Sarge is Vispan - http://www.while.homeunix.net/

There, you now have a mail filtering system. Every e-mail is scanned by 2 virus engines, by MailScanner for HTML and other vulnerabilities, and by SpamAssassin to filter out spam.

Is it finished yet?

Not really. MailScanner is very powerfull and complex, so you have to learn about what you can do with it and how. The first step is to read MailScanner.conf and do some tests. Also, e-mail security is an ever-evolving topic so you must update your system often, and try to find the more recent version of software. Debian Volatile can help with that, or sometimes you may be better compiling from source.

Then, there is a wiki where you can get a lot of information. I suggest you start by reading the MAQ page (which I inciendally created and maintain), and then go in the documentation section for more in-depth tricks.

BTW, MailScanner can work with Exim, Sendmail, Qmail and Zmailer as well.

If you have any questions, you'll find that the MailScanner mailing list is very helpfull.

Finally, please let me know if you find an error in this procedure and I'd appreciate to have any feedback on this article.

Ugo

 

 


Posted by rbelk (192.231.xx.xx) on Thu 30 Jun 2005 at 15:40
[ Send Message ]
It only took me 15 minutes to set this up on my current Postfix mailer, but I ran into some minor issues. I had to change ownership for the following directories because mailscanner did not like them being owned by mail. I changed them to postfix:postfix and it appears to be working ok.

chown -R postfix:postfix /var/spool/MailScanner/
chown -R postfix:postfix /var/lib/MailScanner
chown -R postfix:postfix /var/run/MailScanner/
chown -R postfix:postfix /var/lock/subsys/MailScanner

Also, the hyperlink for http://www.while.homeunix.net does not work, it has a "BR" tag at the end of the link. Vispan is a great reporting package. I'll install it later.

[ Parent | Reply to this comment ]

Posted by Steve (82.41.xx.xx) on Thu 30 Jun 2005 at 16:21
[ Send Message | View Steve's Scratchpad | View Weblogs ]

I've fixed the broken link now, sorry for not spotting this previously.

Steve
-- Steve.org.uk

[ Parent | Reply to this comment ]

Posted by rbelk (192.231.xx.xx) on Thu 30 Jun 2005 at 16:47
[ Send Message ]
This is exactly the steps/howto I've been looking for, thanks for your time and effort. I did find another spelling error

chown postfis:postfix /var/spool/MailScanner/spamassassin

How hard is it to setup vispan on Debian?

[ Parent | Reply to this comment ]

Posted by Steve (82.41.xx.xx) on Thu 30 Jun 2005 at 16:56
[ Send Message | View Steve's Scratchpad | View Weblogs ]

Thanks for that, fixed too now.

As for vispan I guess Ugo would know .. I've never heard of it before!

Steve
-- Steve.org.uk

[ Parent | Reply to this comment ]

Posted by ugob (216.113.xx.xx) on Thu 30 Jun 2005 at 17:22
[ Send Message ]
Vispan was called mailstats before. It is not too hard to install. You just need a couple of perl modules and the rest is pretty simple. Read the README file in the tarball for the details. These are the commands I used to install the perl modules dependencies:
  • perl -MCPAN -e 'install GD::Graph'
  • perl -MCPAN -e 'install Mail::Sendmail'
  • perl -MCPAN -e 'install Number::Format'

[ Parent | Reply to this comment ]

Posted by ugob (216.113.xx.xx) on Thu 30 Jun 2005 at 17:15
[ Send Message ]
Steve, could you fix that (the chown), it is in the section " A few tricks are needed since we're using postfix as MTA:"

Is there a way I can edit directly?

Thanks,

[ Parent | Reply to this comment ]

Posted by Steve (82.41.xx.xx) on Thu 30 Jun 2005 at 17:33
[ Send Message | View Steve's Scratchpad | View Weblogs ]

Fixed now - Sorry but right now authors cannot edit their pieces. I need to write that still!

All being well it will be possible in the next few days, the change isn't very difficult I don't think.

There are quite a few things that need to be updated in the code, it's just a matter of getting them all done.

Steve
-- Steve.org.uk

[ Parent | Reply to this comment ]

Posted by Knobbe (213.197.xx.xx) on Thu 27 Oct 2005 at 12:49
[ Send Message ]
Also do not forget to ad this line;

SpamAssassin User State Dir = /var/spool/MailScanner/spamassassin

[ Parent | Reply to this comment ]

Posted by jrasmussen0 (67.52.xx.xx) on Thu 30 Jun 2005 at 20:19
[ Send Message ]
What benefit is using bitdefender? Is that the only proprietary package installed?

[ Parent | Reply to this comment ]

Posted by ugob (216.113.xx.xx) on Thu 30 Jun 2005 at 20:24
[ Send Message ]
Answer to the first question:

http://wiki.mailscanner.info/doku.php?id=maq:index#anti-virus

Answer to the second question:

Yes, so if you a want system that is completely open, don't use bitdefender.

[ Parent | Reply to this comment ]

Posted by medwayman (212.159.xx.xx) on Thu 30 Jun 2005 at 21:14
[ Send Message | View Weblogs ]
Good article!

I'm set up right now with amavisd-new, which seems to do the same job.

Is mailscanner better?

[ Parent | Reply to this comment ]

Posted by ugob (209.161.xx.xx) on Fri 1 Jul 2005 at 01:16
[ Send Message ]
Thanks !

I don't know if one is better than the other. I don't think so. MailScanner has limitations and advantages that Amavis hasn't and vice-versa. I never used Amavis, so I can't really tell. I can only tell that I manage ~15 MailScanner servers and they do a really, really good job at keeping the networks spam/virus/(e-mail related) vulnerability exploit free.

They also have a different design. Amavis works as a network daemon and that is how it communicates with the MTA. MailScanner picks up the mail on the filesystem and re-feeds the MTA through the filesystem.

[ Parent | Reply to this comment ]

Posted by hildeb (193.175.xx.xx) on Fri 1 Jul 2005 at 09:08
[ Send Message ]
And that's why it's not safe to use with Postfix (according to Wietse, the author).

[ Parent | Reply to this comment ]

Posted by ugob (216.113.xx.xx) on Fri 1 Jul 2005 at 17:31
[ Send Message ]
According to the author, yes. And Julian Field, MailScanner's author, recently changed the way MailScanner processes the messages, and I think it should be ok now since it doesn't process the active queue anymore.

See http://wiki.mailscanner.info/doku.php?id=documentation:configurat ion:mta:postfix:politics

[ Parent | Reply to this comment ]

Posted by ManxStef (80.65.xx.xx) on Thu 30 Jun 2005 at 23:41
[ Send Message ]
Not Mailscanner but vaguely related (uses AMaViS), I found this guide immensely useful when configuring a Postfix filter/relay to protect my work's Exchange system:
Debian Anti-Spam Anti-Virus Gateway Email Server using Postfix 2.1, Amavisd-new, SpamAssassin, Razor, DCC, Pyzor, and ClamAV

[ Parent | Reply to this comment ]

Posted by freelsjd (68.34.xx.xx) on Fri 1 Jul 2005 at 02:08
[ Send Message ]
Thanks for posting this article. I have been wanting to implement this for a while now, but did not have the time to dig in a figure out how. I also had the directory protection issue and changed it exactly the same way. Also, I have been examining my /var/log/mail.log file as you suggested. I find the following warning message printed infinite times:

Jun 30 19:57:37 fea MailScanner[27371]: Messages found but no hashed queue directories. Please enable hashed queues for incoming and deferred with a depth of 1 or 2. See the Postfix documentation for hash_queue_names and hash_queue_depth

I can't seem to figure out how to configure postfix to correct this problem. I think it is postfix because at least it has a hash_queue command appearing in the post_install script. How do I correct this problem ?

[ Parent | Reply to this comment ]

Posted by hildeb (193.175.xx.xx) on Fri 1 Jul 2005 at 09:10
[ Send Message ]
The solution IS described in the error message you got. What could be easier?

# postconf hash_queue_names hash_queue_depth
hash_queue_names = deferred, defer
hash_queue_depth = 1

So use:
postconf -e "hash_queue_names = deferred, defer, incoming"
postfix stop
postfix start

[ Parent | Reply to this comment ]

Posted by freelsjd (160.91.xx.xx) on Fri 1 Jul 2005 at 12:43
[ Send Message ]
Perhaps to you it is easy, but not I. Thank you for the tip.

However, I tried this and it did not fix the problem. I continue to get those same messages and it is making for a rather large /var/log/mail.log file

[ Parent | Reply to this comment ]

Posted by hildeb (193.175.xx.xx) on Fri 1 Jul 2005 at 12:51
[ Send Message ]
But my commands did exactly what the error message wanted me to do... Anyway: Mailscanner is not safe to use with Postfix anyway, so the point is rather moot.

[ Parent | Reply to this comment ]

Posted by ugob (216.113.xx.xx) on Fri 1 Jul 2005 at 17:34
[ Send Message ]
Please send any problem reports to the MailScanner mailing list. A lot of people will help you there (including me). I don't want to start a long troubeshooting thread here please.

Ugo

[ Parent | Reply to this comment ]

Posted by freelsjd (68.34.xx.xx) on Sun 3 Jul 2005 at 01:49
[ Send Message ]
Thanks for the post on the mailscanner mailing list. I was able to see just from the list archives that the problem was the following entry needed to be in the /etc/razor/razor-agent.conf file:

logfile=/var/spool/postfix/hold/razor-agent.log

which was totally different from the messages sent to the log file and I would never have figured this out. Things seem to be working here now although I still have one annoying warning message each time postfix is started: Jul 2 21:38:52 fea postfix/postsuper[18773]: warning: bogus file name: hold/razor-agent.log Also, there is a good explanation for the apparent postfix/mailscanner issue here:

http://www.sng.ecs.soton.ac.uk/mailscanner/serve/cache/382.html

One thing I have learned about all this: it needs to me much simpler ! This is difficult to implement (even for us Linux hackers) !

[ Parent | Reply to this comment ]

Posted by ugob (209.161.xx.xx) on Sun 3 Jul 2005 at 13:44
[ Send Message ]
Well, all I can say is that MailScanner is a powerful, but complex piece of software. There are always little glitches in complex software packages, especially when it is interacting with many other sofware programs like that. I'm sorry if you found it difficult, but I think that you did great and you only had one little problem that you could solve by your own by searching the list archive. Maybe using another MTA would have been easier (Exim...), maybe not. BTW, going to the mailing list when you install MailScanner is part of the process, IMHO. Personally, I find that the hardest part of installing and configuring MailScanner is to understand all the options in MailScanner.conf. The next step is to read some of the online doc and... Buy the book! http://www.mailscanner.info/store/

[ Parent | Reply to this comment ]

Posted by hildeb (193.175.xx.xx) on Fri 1 Jul 2005 at 09:07
[ Send Message ]
MailScanner's way of accessing Postfix's queue is unsafe and Wietse STRONGLY recommends not to use it with Postfix. Cite: "Mailscanner is not safe to use with Postfix".

This is due to the fact that undocumented Postfix internals (e.g. the queue file format) are subject to change. Since these changes cannot be anticipated by the MailScanner author, it's definitely not safe to use.

[ Parent | Reply to this comment ]

Posted by ugob (216.113.xx.xx) on Fri 1 Jul 2005 at 17:41
[ Send Message ]
MailScanner+Postfix setup are filtering millions of e-mail messages around the globe daily, without problem. MailScanner's author has always been very, very quick to deal with any issue about this kind of setup and I've been succesfully running this setup on 4 machines for more than 2 years with only one little problem. This was a little bug in one version, so I rolled back and the issue has been fixed within 2 weeks.

But hey, if you don't want to use Postfix, just replace the section about postfix in the article by something exim-specific. It apparently works out-of the box. Or use Amavis. With open-source you have the choice...

Ah, and see http://wiki.mailscanner.info/doku.php?id=documentation:configurat ion:mta:postfix:politics

[ Parent | Reply to this comment ]

Posted by GeorgeLazar (81.196.xx.xx) on Tue 16 Aug 2005 at 06:07
[ Send Message ]
Nice article, but how can I notify the sender about the virus?

[ Parent | Reply to this comment ]

Posted by Anonymous (212.224.xx.xx) on Tue 16 Aug 2005 at 15:24
In the MailSCanner.conf file you have

# Still deliver (after cleaning) messages that contained viruses listed
# in the above option ("Silent Viruses") to the recipient?
# Setting this to "yes" is good when you are testing everything, and
# because it shows management that MailScanner is protecting them,
# but it is bad because they have to filter/delete all the incoming virus
# warnings.
#
# Note: Once you have deployed this into "production" use, you should set
# Note: this option to "no" so you don't bombard thousands of people with
# Note: useless messages they don't want!
#
# This can also be the filename of a ruleset.
Still Deliver Silent Viruses = no


If you realy want to notify all the people that didn't send the virus ... change it to yes.


For the MailScanner daemon, I found an interesting article about extending it with a front-end: MailWatch for MailScanner: a PHP web front-end to MailScanner, creates graphics and statistics about spam, virii and other. Does a great job in creating statistics, learning spam to spamassassin, surfing the archives, ...

[ Parent | Reply to this comment ]

Posted by Anonymous (81.196.xx.xx) on Wed 17 Aug 2005 at 06:46
Thank you, no :-)
I would like to inform the sender that he sended a virus.

[ Parent | Reply to this comment ]

Posted by Anonymous (84.194.xx.xx) on Thu 18 Aug 2005 at 09:51
You have 99,999% chance that the person who receives the virus warning didn't send the virus ... that's what I hate.

[ Parent | Reply to this comment ]

Posted by Anonymous (81.196.xx.xx) on Mon 22 Aug 2005 at 09:13
Yes, but in the other cases the sender believes that the message was delivered.

[ Parent | Reply to this comment ]

Posted by Anonymous (213.193.xx.xx) on Sat 10 Sep 2005 at 08:02
lol

99,99999% of the viruses are send without the knowledge of the suer, and 99,999% of these viruses the sender isn't the user in the "From:".

Do you still believe it's important for person X that person Y send a virus ... ??? Fred

[ Parent | Reply to this comment ]

Posted by tessem (143.97.xx.xx) on Tue 20 Sep 2005 at 15:14
[ Send Message ]
Hi.
I have installed Mailscanner according to this realy nice step-by-step instruction. However i feel MailScanner is not behaving as it should. It often hangs so that I have to restart it (/etc/init.d/mailscanner restart), and then it floods in with messages. Has anybody else experienced that ?

Second question: My postfix is using procmail (mailbox_command = procmail -a "$EXTENSION" ). How is this working together with MailScanner, and should I avoid use of /etc/procmailrc settings for spamassassin ?

[ Parent | Reply to this comment ]

Posted by ugob (216.113.xx.xx) on Tue 20 Sep 2005 at 15:22
[ Send Message ]
For your first problem, I recommend you check the troubleshooting part of the MailScanner wiki.

http://wiki.mailscanner.info/doku.php?id=documentation:test_troub leshoot:mailscanner

For the second question, you should avoid using procmail with MailScanner, since MailScanner calls SpamAssassin itself. It might be possible, but you should probably ask the MailScanner list to ask people before. I can't tell since I've never used procmail.

What actions are you performing with procmail? Please let me know and I'll tell you if you can do that directly within MailScanner.

[ Parent | Reply to this comment ]

Posted by tessem (62.179.xx.xx) on Tue 20 Sep 2005 at 18:00
[ Send Message ]
Hi, and thanks for quick answer. I'll check out the wiki.

As far as I can read the procmailrc that comes along with spamassassin as an example, the procmail is doing a pre-selection of mails of small size (less than 256k) and checks these for spam directly. Assuming that viruses are greater size, I guess (which is not nessecery true...?).

:0fw: spamassassin.lock
* < 256000
| spamassassin

[ Parent | Reply to this comment ]

Posted by ugob (216.113.xx.xx) on Tue 20 Sep 2005 at 18:16
[ Send Message ]
No, both viruses and spam are usually fairly small. MailScanner uses a different technique: It only scans the first x bytes of a message for spam. That is usualy more than enough to detect a spam message. Conclusion: you don't need procmail anymore :).

[ Parent | Reply to this comment ]

Posted by tepegoz (144.122.xx.xx) on Wed 8 Mar 2006 at 06:20
[ Send Message | View Weblogs ]
Hello,
I got the same problem. Usually mailscanner (or something else) hangs once a day. I suspect that procmail is stoping somehow with large attachments. I could not find any information in the log files.

I disabled procmail now. Let's see what will happen.

[ Parent | Reply to this comment ]

Posted by Anonymous (213.41.xx.xx) on Fri 11 Jan 2008 at 21:18
I've two questions :

- How does the filtering system work ?
- How can I know if the filtering system works well ? I looked mail.log and I have :

MailScanner[22457]: MailScanner E-Mail Virus Scanner version 4.55.10 starting...
MailScanner[22457]: Read 748 hostnames from the phishing whitelist
MailScanner[22457]: Using SpamAssassin results cache
MailScanner[22457]: Connected to SpamAssassin cache database
MailScanner[22457]: Enabling SpamAssassin auto-whitelist functionality...
MailScanner[22457]: ClamAV scanner using unrar command /usr/bin/unrar
MailScanner[22457]: Using locktype = flock

Does it work ?

Thanks !

[ Parent | Reply to this comment ]

Sign In

Username:

Password:

[Register|Advanced]

 

Flattr

 

Current Poll

Which init system are you using in Debian?






( 1068 votes ~ 7 comments )