Question: Preventing Apache referer spam?

Posted by Steve on Mon 29 Aug 2005 at 16:00

Referer spam is something that has only affected weblogs until recently. However it is now on the rise generally and many webservers are seeing incoming requests with HTTP Referer spam.

Referer spam is simply described as incoming requests to your webserver with a website being listed in the "referer" field. The intention of submitting requests is that these logs will be archived somewhere and that search engines will spider these logs and increase the score of the spammed websites.

There are two popular approaches to dealing with Referer spam on Apache webservers - both of which require you to maintain a blacklist of referer strings, or IP addresses, you wish to ignore.

Using mod_rewrite to redirect bogus requests.
Using mod_security to deny incoming requests.

Each of these approaches suffers from the same problem: You must have a list of the invalid referers to block.

For example with mod_security you can block referers which mention "poker" with rules like this:

SecFilterSelective "HTTP_REFERER" "(holdem|poker|casino)"

This will match on all incoming requests which have a referer string containing the words "poker", "holdem", or "casino".

The mod_rewrite equivilent is :


  RewriteEngine   on
  RewriteCond %{HTTP_REFERER} poker  [OR]
  RewriteCond %{HTTP_REFERER} holdem [OR]
  RewriteCond %{HTTP_REFERER} casino 
  RewriteRule .* - [F,L]

Both of these solutions are simple to setup if you're using one of the modules already. (We've previously covered installing mod_security and enabling mod_rewrite for Apache/Apache2.)

The real problem is keeping the blacklists/rules current.

So, my question is how do you deal with this problem?

This article can be found online at the Debian Administration website at the following bookmarkable URL:

http://www.debian-administration.org/articles/232