Weblog entry #39 for simonw
See all weblog entries made by simonw.
We are dropping packets from IPs identified as involved, but this is error prone at best (we already trapped a Google bot that tried to index the offending pages).
Our ISPs response so far hasn't been great.
Wondering how others have got on in the same circumstance. Clearly this was a small network of bots, and a minor inconvenience, but enough to prompt us to establish a proper procedure.
Comments on this Entry
We had a similar problem with a suspected Virus that was guessing passwords on the SSH port. The attack was coming from multiple addresses.
The solution is, I think, rather elegant thanks to some excellent work by the iptables/netfilter contributors. You will need the ipt_recent kernel module for this to work.
This is what I added to my /etc/network/interfaces files for the external device:
# only allow 3 ssh connections every 5 minutes up /sbin/iptables -A allowed -m state --state NEW -p tcp --dport 22 -m recent --update --seconds 300 --hitcount 5 -j DROP up /sbin/iptables -A allowed -m state --state NEW -p tcp --dport 22 -m recent --set -j ACCEPT
What it does is drop connections with a hitcount of 5 (within 300s). If not, it will accept the connection and update the hitcount (--set).
This allows 5 login attempts to port 22 (SSH) before futher attempts will be denied for 5 minutes. As soon as these rules were added, the password guessing stopped for good.
In our configuration, we use the rules in the allowed chain for INPUT and FORWARD and everything else is dropped. I have left out some other rules in the allow chain for brevity, but hopefully you get the idea.
For more information, google on the iptables recent module.
[ Parent | Reply to this comment ]
[ Send Message | View Steve's Scratchpad | View Weblogs ]
For more information, google on the iptables recent module.
Or read this ;)
I've only ever seen these accidental DDoS attacks hitting Apache a couple of times. In my case each had an identical "bot" User-Agent, so I could drop them fairly easily with the mod_security module for Apache.
(I'm ignoring dictionary attacks on SSH, since they are pretty standard these days.)
[ Parent | Reply to this comment ]
Yes good point. Perhaps that could have saved me some time... or is that where I got the idea from?
[ Parent | Reply to this comment ]
I think the issue is that web traffic is huge in comparison to most other traffic, so HTTP misuse tends to hurt a lot more.
We identified two types of abusive HTTP traffic causing problems (we think they were related), both identified by requested URL and appropriate rules added by parsing the squid accelerator logs and adding iptables rules as required.
Initially removing the pages being requested help reduce traffic, but eventually serving 404 messages was eating too much bandwidth (we don't have a lot spare).
I'm more interested in what people do when this level of defence isn't enough. The ISP has been responsive, but not especially helpful. They claim they have broad network defences in place to prevent large scale DDoS attacks, which may keep their own network working, but is not much good for us.
One of my previous employers policy was to host routers at ISPs, and thus to control both ends of any bottleneck links.
[ Parent | Reply to this comment ]