1) Qmail. Hmmmm. Personally I wouldn't touch qmail these days, unless sendmail was the only other option, but even then it'd be a tight call. It's been way too long since qmail's had an official update from what I can tell, and due to the licence restrictions imposed by djb getting modern functionality can be a pain (although this is second hand info from a work-mates previous pains with qmail).

Essentially the more uptodate options and probably the best supported hereabouts are exim (the debian default, and my personal favorite), and postfix (which also has alot of fans here).

2) One BIG precaution -- make sure it's not an OPEN relay server -- in other words, only accept mail from or for your users, do not let external users send mail to other external users. One of the really nice things about exim are all the different testing modes that it has (commandline options, but well worth learning about); these enable you to firewall off your mail ports, build the mail configuration the way you want it, test it thoroughly, and only then open it up for use. Other mailservers have similar abilities, but exim just makes it really easy.

3) See the beginning of 2) re. being an OPEN relay. Also, use SpamAssassin (or something similar like DSPAM). Again, exim is really useful here, as its SMTP time acls can do inline scanning for spam (and malware if you use something like clamav), and these tools can go a long way to defeding you from spammers.

4) Reject as much as possible at SMTP time, that way you're doing much of the scanning you would anyway, but without the effort of actually accepting the message, and then bouncing it later.

5) I know that you didn't even ask a 5), but here it is: Once you've chosen which mail server you're going to use, take the docs home for the weekend, and read cover to cover before you configure it -- it's much easier to organise your thoughts once you've got a bird's eye view of how it all fits together.

Cheers.

relaying in qmail is very simple. Just add the domainname into the /var/control/rcpthosts file, reload qmail-send. That's it. Just don't include the domain into the locals or vitualdomains files.

Emails for domains not listed in rcpthosts or morercpthosts are rejected (relay not allowed). Domains in (more)rcpthosts and not listed in locals or virtualdomains are relayed.

If you need to relay to a special (e.g. not listed as an MX record) server on a non-smtp port you also need to add an entry to smtproutes. If the file doesn't exist, just create it.
Each entry has the following syntax:
domain:new domain or ip[:port]
example.org:example.com
example.org:example.net:26

Development of Qmail self is finished. It receives, sends and deliver emails. According to Dan Bernstein that's what a mail server is supposed to do. No extra whissles means less code equals minor changes of bugs (exploits). But the qmail community (www.qmail.com) have developed a lot of addon's for it. Basicly all of them are some kind of filter.

Have fun!