Blocking a DNS DDOS using the fail2ban package

Posted by vlm on Sat 31 Jan 2009 at 14:20

Are you tired of getting multi-thousand line emails from the logcheck package that contain multiple reports of denied queries from named? If so this article will show how you can reject these DDOS attempts via the fail2ban package.

These events look something like this:

System Events
=-=-=-=-=-=-=
Jan 21 06:02:13 www named[32410]: client 66.230.128.15#15333: query (cache)
+'./NS/IN' denied

You can get the whole story about this DOS attack at http://isc.sans.org/diary.html?storyid=5713

However, in summary, the source ip address is falsified by a large bot army. Each soldier of the bot army sends one DNS packet per second to your DNS server. Your dns server replies with a fail message to the falsified source address, causing a DOS attack on that source address.

Tired of your DNS server being used as someone's DOS amplifier weapon? Try Debian's fail2ban package. The homepage for fail2ban is http://www.fail2ban.org

First install the Debian fail2ban package. By default it only watches and bans ssh. That is probably a good idea, further discussion of which is somewhat beyond the scope of this article.

apt-get install fail2ban

Then inspect the contents of /etc/fail2ban/jail.conf
As per the notes at the end of that file, you'll need to modify your bind logging so fail2ban can understand it.

First make the directory for the bind log file.

mkdir /var/log/named
chmod a+w /var/log/named

I'm sure a reader will complain about making a log file a+w, but it is the simplest way to make this demo work. In your spare time, once everything works, find a better way.

Next, edit /etc/bind/named.conf.local and add the following lines

logging {
    channel security_file {
        file "/var/log/named/security.log" versions 3 size 30m;
        severity dynamic;
        print-time yes;
    };
    category security {
        security_file;
    };
};

Restart Bind using /etc/init.d/bind9 restart
Test bind to make sure it's still working and also verify the log file /var/log/named/security.log is filling up with lines like this:

21-Jan-2009 07:19:54.835 client 66.230.160.1#28310: query (cache) './NS/IN' denied

OK, now to set up fail2ban. Edit the /etc/fail2ban/jail.conf file and change from:

[named-refused-udp]

enabled  = false

to:

[named-refused-udp]

enabled  = true

and from:

[named-refused-tcp]

enabled  = false

to:

[named-refused-tcp]

enabled  = true

Then restart fail2ban in the usual manner,

/etc/init.d/fail2ban restart

Now verify that fail2ban is doing something by checking out the log file located at /var/log/fail2ban.log it should contain something like

2009-01-21 07:34:32,800 fail2ban.actions: WARNING [named-refused-udp] Ban 76.9.16.171
2009-01-21 07:34:32,902 fail2ban.actions: WARNING [named-refused-tcp] Ban 76.9.16.171

Verify that fail2ban is modifying the iptables rules

iptables -L

Now verify that fail2ban's iptables rules are actually stopping access

tail -f /var/log/named/security.log

DNS error messages should be several minutes apart rather than multiple per second.

Now for some fine tuning.

First we have to modify logcheck to look at the new location of named error messages. Edit /etc/logcheck/logcheck.logfiles and add this to the end of the file:

/var/log/named/security.log

Next modify logcheck to report what fail2ban is doing. edit the same file, /etc/logcheck/logcheck.logfiles and add this line to the end of the file:

/var/log/fail2ban.log

Now verify you are getting both named and fail2ban messages in your hourly logcheck emails.

It would be a good idea to research, change, and test the [DEFAULT] ignoreip stanza in the /etc/fail2ban/jail.conf file. Maybe the package default should change to ignore all RFC1918 addresses. Probably you should ignore your LAN, just in case. At least ignore the source ip address of the machine that you usually use to SSH into your DNS server.

It would also be a good idea to think about the bantime = 600 setting in the /etc/fail2ban/jail.conf file. Maybe more than ten minutes would be appropriate. Since the DDOS attacks last "days", perhaps "hours" would be most appropriate.

As per above, think of a better way to assign permissions to the logfile directory /var/log/named. Maybe modify /etc/bind/named.conf.local and /etc/logcheck/logcheck.logfiles and /etc/fail2ban/jail.conf to not use that directory... maybe just use a file named /var/log/named.log

Last, but certainly not least, please compliment the maintainer of the debian package Yaroslav Halchenko and also the author of fail2ban, Cyril Jaquier

Thanks, and have a pleasant day.

Vince Mulhollon vince@mulhollon.com aka vlm@debian.org

 

 


Posted by Anonymous (213.226.xx.xx) on Sat 31 Jan 2009 at 19:29
I did it couple of days ago too - in jail.conf is an example of part which you have to put into bind's config file

no idea why but in this example is

logging {
...
}

instead of

logging {
...
};

(semicolon in the end) so you can't start bind afterwards - I spent so many hours to find it :( in this article it's ok

[ Parent | Reply to this comment ]

Posted by Serge (78.21.xx.xx) on Sat 31 Jan 2009 at 19:48
[ Send Message | View Serge's Scratchpad | View Weblogs ]
I think instead of doing

# chmod a+w /var/log/named

you'd better issue

# chown bind:adm /var/log/named



--
Serge van Ginderachter

[ Parent | Reply to this comment ]

Posted by yarikoptic (69.125.xx.xx) on Tue 3 Feb 2009 at 04:05
[ Send Message ]
sounds like a bug... my bug

I wonder -- why didn't you typed 'reportbug fail2ban' in your favorite terminal?

;-)

in any case -- it is fixed in 0.8.3-4 which is on the way now to experimental ;-) also will upstream know

[ Parent | Reply to this comment ]

Posted by existenz (2a01:0xx:0xx:0xxx:0xxx:0xxx:xx) on Sun 1 Feb 2009 at 12:31
[ Send Message ]
If you use apparmor and get syslog entries like:
Feb 1 13:23:04***** kernel: [69837.875962] type=1503 audit(1233490984.050:540): operation="inode_create" requested_mask="a::" denied_mask="a::" fsuid=115 name="/var/log/named/security.log" pid=23748 profile="/usr/sbin/named"

Simply add the following two line to /etc/apparmor.d/usr.sbin.named:
/var/log/named/** rw,
/var/log/named/ rw,

and load the new profile:
cat /etc/apparmor.d/usr.sbin.named | sudo apparmor_parser -r
sudo /etc/init.d/apparmor reload

[src: https://bugs.launchpad.net/ubuntu/+source/bind9/+bug/294935 ]

+

[ Parent | Reply to this comment ]

Posted by philcore (72.218.xx.xx) on Sun 1 Feb 2009 at 13:35
[ Send Message | View Weblogs ]
It's worth noting that by blocking these ip addresses, you could be blocking *legitimate* queries from the actual victim domain, causing a DoS for the victim. The victim domain will not be able to reach your network and your services. Isn't UDP fun?

phil

[ Parent | Reply to this comment ]

Posted by Anonymous (207.237.xx.xx) on Tue 3 Feb 2009 at 05:31
Note that the version of fail2ban in etch 0.7.5-2etch1 does not have the [named] sections in /etc/fail2ban/jail.conf. You must be using a newer version...

[ Parent | Reply to this comment ]

Posted by Anonymous (2001:0xx:0xx:0xxx:0xxx:0xxx:xx) on Wed 4 Feb 2009 at 23:07
Hi,

I personally think this method on UDP streams is quite dangerous since it can lead to block legitimate servers/users.

You'd better patch your BIND config (additional-from-cache no;).

Laurent (lc+debian-admin@unix-scripts.info)

[ Parent | Reply to this comment ]

Posted by simonw (84.45.xx.xx) on Sun 12 Apr 2009 at 10:12
[ Send Message | View Weblogs ]
Seconded, anyone who implements this is deliberately adding a Denial of Service vulnerability and that would be very daft.

I note the servers for debian-administration.org don't answer rogue queries, so there is a clearly a better solution.

Perhaps Steve will tell us what they are doing?

I did a survey - but it is a work in progress still, this is how I spotted that Microsoft think they are the Internet root servers.

http://simonwaters.technocool.net/dns.html

Feedback welcome!

[ Parent | Reply to this comment ]

Posted by Anonymous (66.76.xx.xx) on Wed 18 Feb 2009 at 17:06
Where these instructions say to edit /etc/fail2ban/jail.conf, I think that "violates" the Debian Way. The instructions at the top of the conf file state that changes from the default conf should rather be placed in jail.local. This helps to prevent collisions on package updates and still works as expected since Debian checks the *.local for conf overrides.

Also, regarding the /var/log/named/ directory, my bind9 is chrooted under /var/lib/named/ because of following a tutorial from howtoforge.com. There are probably others who set up DNS on Debian following the same instructions. Those people need to create the directory under the chroot /var/lib/named. The subdirectory var is already present, but you need to create log and named, perhaps like this:

cd /var/lib/named/var
mkdir -p log/named


Then adjust the ownership to match the other directories:

chown -R bind:bind log

And to make the directory appear to be in the "expected" location, a symlink completes the effect:

cd /var/log
ln -s /var/lib/named/var/log/named named


Works fine here.

[ Parent | Reply to this comment ]

Posted by tictacbum (87.223.xx.xx) on Thu 26 Feb 2009 at 20:32
[ Send Message ]
There's any way to disable sysklogd log compression functionality?
last message repeated 6 times
As it groups identical log lines, fail2ban doesn't count this lines.

[ Parent | Reply to this comment ]

Posted by Anonymous (91.219.xx.xx) on Mon 8 Oct 2012 at 07:40
my fail2ban is not ban this ddos, in logs only this messages:
INFO Log rotation detected for /var/log/named/security.log

[ Parent | Reply to this comment ]

Sign In

Username:

Password:

[Register|Advanced]

 

Flattr

 

Current Poll

Which init system are you using in Debian?






( 1066 votes ~ 7 comments )