Blocking ad servers with dnsmasq

Posted by lindenle on Tue 19 Jun 2007 at 08:59

Tags: ,

I was chatting with a colleague over IRC on Tuesday and he was complaining about the new update for Bind9 that broke his automatic blocking of ad servers. Naturally I was curious and asked him what he was talking about..

He pointed me to the site pgl.yoyo.org/adservers that provides a list of the currently known banner ad distribution domains. The avid web surfer in me immediately decided I could no longer live with banner adds while web browsing at home and I set out to solve this problem. The easiest solution would have been to ask "Bob" for his scripts and install Bind9 on my server. I however already run dnsmasq on said server and figured there must be a way to bend it to my will. Reading the dnsmasq manpage I found the following:

A, --address=/<domain>/ [domain/] <ipaddr>
              Specify an IP address to  return  for  any  host  in  the  given
              domains.   Queries in the domains are never forwarded and always
              replied to with the specified IP address which may  be  IPv4  or
              IPv6.  To  give  both  IPv4 and IPv6 addresses for a domain, use
              repeated -A flags.  Note that /etc/hosts and DHCP  leases  over-
              ride this for individual names. A common use of this is to redi-
              rect the entire doubleclick.net domain to  some  friendly  local
              web  server  to avoid banner ads. The domain specification works
              in the same was as for --server, with  the  additional  facility
              that  /#/  matches  any  domain.  Thus --address=/#/1.2.3.4 will
              always return 1.2.3.4 for any query not answered from /etc/hosts
              or  DHCP  and  not sent to an upstream nameserver by a more spe-
              cific --server directive."

This seemed to be exactly what I wanted. It redirects any server in a given domain to a specified ip address. Now all I had to do was write a script to download the list, rewrite my dnsmasq.conf file and restart dnsmasq at regular intervals.

The following bash script is the result of about an hour of hacking yesterday. It downloads a plain text list of ad servers from yoyo, rewrites "/etc/dnsmasq.conf" and restarts the server:

#!/bin/sh

### short script that downloads a list of ad servers for use with
### dnsmasq to block ads.
###

# the ipaddress where we want to send the requests to, instead of the
# bannerservers

addcatcherip='192.168.1.4'
configfile=/etc/dnsmasq.conf

# the args to add to the request to the yoyo server, to tell it that we want
# a hosts file and that we want to redirect to the addcatcher
listurlargs="hostformat=nohtml&showintro=0&mimetype=plaintext"

# URL of the ad server list to download
listurl="http://pgl.yoyo.org/adservers/serverlist.php?${listurlargs}"

# location of a file where hostnames not listed can be added
extrasfile='/etc/banner_add_hosts.manual'

## command to reload dnsmasq - change according to your system
## not sure if we need this for dnsmasq
reloadcmd='/etc/init.d/dnsmasq restart'  

# temp files to use
tmpfile="/tmp/.adlist.$$"
tmpconffile="/tmp/.dnsmasq.conf.$$"

# command to fetch the list (alternatives commented out)
fetchcmd="/usr/bin/wget -q -O $tmpfile $listurl"

$fetchcmd 

# add the extras
 [ -f "$extrasfile" ]  && cat $extrasfile >> $tmpfile

# check the temp file exists OK before overwriting the existing list
if  [ ! -s $tmpfile ] 
then
echo "temp file '$tmpfile' either doesn't exist or is empty; quitting"
exit
fi

# get a fresh list of ad server addresses for dnsmasq to refuse
cat $configfile | grep -v "address=" > $tmpconffile

while read line; do
    ADDRESS="/${line}/${addcatcherip}"
    echo "address=\"${ADDRESS}\"" >> $tmpconffile
done < $tmpfile 

mv $tmpconffile $configfile
$reloadcmd
rm $tmpfile
exit

The script has the nice feature that hosts can be added by hand in the file /etc/banner_add_hosts.manual. I installed it in /usr/local/bin and changed the permissions to 700 and ran the script:


$ sudo cp update_bannerhosts /usr/local/bin
$ sudo chown root.root /usr/local/bin/update_bannerhosts
$ sudo chmod 700 /usr/local/bin/update_bannerhosts
$ sudo  /usr/local/bin/update_bannerhosts
Restarting DNS forwarder and DHCP server: dnsmasq.

Looking at my /etc/dnsmasq.conf revealed the addition of many lines of new address entries:

address="/ac.rnm.ca/192.168.1.4"
address="/accelerator-media.com/192.168.1.4"
address="/action.ientry.net/192.168.1.4"
address="/actionsplash.com/192.168.1.4"
address="/actualdeals.com/192.168.1.4"

Next I ran a few quick tests to see that all ad domains were being redirected to my addcatcherip.

$ nslookup doubleclick.net
Server:         192.168.1.4
Address:        192.168.1.4#53

Name:   doubleclick.net
Address: 192.168.1.4
$ nslookup ads.doubleclick.net
Server:         192.168.1.4
Address:        192.168.1.4#53

Name:   ads.doubleclick.net
Address: 192.168.1.4

Sure enough any server in the domain doubleclick.net is now resolving to my local ip address. The next big test was to open a web page and see if the banner ads were gone, I chose heise.de and sure enough the big banner add at the top was now gone. Success!

The last step in this project was to add crontab entry to run the new script every 4 hours to update the list of ad servers:


$ cat /etc/cron.d/update_bannerhosts

#Update the banner hosts...
0 0,4,8,12,16,18,20 * * * root /usr/local/bin/update_bannerhosts

I hope this helps some people get rid of banner ads without having to install and learn Bind9.

 

 


Posted by Alucard (24.147.xx.xx) on Tue 19 Jun 2007 at 14:56
[ View Weblogs ]

Cool :). Been doing this for over a year, and it's amazing how muhc better the Internet looks. For most servers though, you don't have to jump through the hoops of changing /etc/dnsmasq and a script. Download the list in hosts file, plain-text format, and put it in, say, /etc/hosts.ads. Then in /etc/dnsmasq.conf, find+uncomment+change/put the following line:

addn-hosts=/etc/hosts.ads
and restart dnsmasq.

This is easier for probably 99% of ad servers, but some (like intellitxt) use a different subdomain for *every* site they serve, so that's when you need to take advantage of dnsmsq's address=/../.. feature. I have a modest list of those (plus about 2000 servers in /etc/hosts.ads):

address=/doubleclick.net/192.168.0.1
address=/intellitxt.com/192.168.0.1
address=/vibrantmedia.com/192.168.0.1
address=/kontera.com/192.168.0.1
address=/tribalfusion.com/192.168.0.1
address=/adbrite.com/192.168.0.1
This will block, for example, *all* intellitxt subdomains.

Once you have that, the next step is to set up something local to actually answer the requests, so you don't have to wait for the connection to time out or get Connection Refused or whatnot. A simple web server setup will do this; I use lighttpd but I won't get much into the actual configuration here. I set it up to answer 404s from local IPs with a very small "ad blocked" image.

As a side note, the computer this is running on is my LAN's router, thus the 192.168.0.1 address. If your machine is not a DNS server for anyone else you want the suggested 127.0.0.1.

You can do this on any Linux machine (like your laptop, say), and as part of a router setup it's very powerful -- my not-very-techy roomate was quite pleased when suddenly the ads on her AIM client disappeared.

And finally, the local solution for Windows machines is %WINDIR%\system32\drivers\etc\hosts and eDexter, though I don't know how to do wildcard matching.

[ Parent | Reply to this comment ]

Posted by neofpo (200.185.xx.xx) on Tue 19 Jun 2007 at 15:54
[ View Weblogs ]
There is a click'n use solution: Adblock Plus.
I know DNS has other advantages (specially on a server of a mixed clients environment) but on a single computer, this extension is easier to use.

[ Parent | Reply to this comment ]

Posted by Alucard (24.147.xx.xx) on Tue 19 Jun 2007 at 16:31
[ View Weblogs ]
That Firefox extension has the big disadvantage that it only works for Firefox, and not for AIM, IE (when you have to use it..), $random_spyware_app, etc, but if you just want to block ads in FF, it's fine.

[ Parent | Reply to this comment ]

Posted by Anonymous (76.199.xx.xx) on Thu 21 Jun 2007 at 20:47
Good point about the web server reply I run apache and added the follwing configuration and file.


<VirtualHost *:80>
        ServerAdmin  root@localhost
        DocumentRoot  ;/home/lindenle/public_html/ad_block
        <Directory&nbs p;/home/lindenle/public_html/ad_block>
            ;     Options -Indexes
            ;     AllowOverride All
            ;     DirectoryIndex doku.php
            ;     ErrorDocument 404 "/erro r/error_404.html"
        </Directory> ;

</VirtualHost>


This answers any 404 errors with a file that contains:


<html>
<body>
Ad Blocked!
</body>
</html>


and the body of the ad is replaced with the text "Ad Blocked!"

[ Parent | Reply to this comment ]

Posted by Anonymous (65.95.xx.xx) on Sun 18 May 2008 at 20:13
I have a question about running the apache server. I am currently running a server. and would like to point requests to a different IP address. I was wondering if you would be able to provide any information about how to host multiple IPs?

I keep getting the following error message:

* Forcing reload of web server (apache2)... httpd (no pid file) not running
(99)Cannot assign requested address: make_sock: could not bind to address 192.168.0.35:80
no listening sockets available, shutting down
Unable to open logs

Thank you

Philippe

[ Parent | Reply to this comment ]

Posted by Anonymous (151.46.xx.xx) on Tue 19 Jun 2007 at 19:27
and what's the solution if I want to use that list with bind9?

[ Parent | Reply to this comment ]

Posted by matej (213.160.xx.xx) on Sat 23 Jun 2007 at 16:40
I'd suggest to create empty zone file or a file which points to a webserver retunring nothing immediately (see previous or even later posts). then, you should modify your named.conf (named.conf.local on debian's bind9) to include the following:

...
zone "doubleclick.net" { type master; file "/etc/bind/empty.zone"; }
zone "advertising.com" { type master; file "/etc/bind/empty.zone"; }
...

the empty.zone file:


$TTL 3600
@ in soa localhost. hostmaster.localhost. (2007062301 120 1 15 120)
@ in ns localhost.
* in a 192.168.0.1

[ Parent | Reply to this comment ]

Posted by TRS-80 (203.59.xx.xx) on Wed 20 Jun 2007 at 07:11
At work I have squid configured to use adzapper (which is in Debian). I don't have it transproxying but all browsers are configured to use the squid proxy. At home I use NoScript for other reasons but it does a good job at blocking many ads that rely on javascript.

[ Parent | Reply to this comment ]

Posted by Anonymous (213.164.xx.xx) on Wed 20 Jun 2007 at 14:37
> At work I have squid configured to use adzapper

How do you expect the websites that rely on adverts to pay their bills?

[ Parent | Reply to this comment ]

Posted by Anonymous (130.214.xx.xx) on Thu 21 Jun 2007 at 19:21
>How do you expect the websites that rely on adverts to pay their bills?

Cash or check, I suppose. Joking aside, I've never clicked an ad, so I'm definitely not helping them pay their bills either way.

[ Parent | Reply to this comment ]

Posted by Anonymous (62.77.xx.xx) on Sat 23 Jun 2007 at 09:03
Anonymous you idiot, your dumb comment cleanly shows that you try to finance your pathetic site from ads, well maybe not everyone install adblocker and maybe not everyone want to see ungly/obscene/disgusting banners all day.

This script is similar to another which can be used with squidguard.

# URL of the ad server list to download
listurl='http://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml 9;

It's a good workaround if you don't have proxy on your network but is it faster than block these sites with squidguard or dansguard?

zeratul

[ Parent | Reply to this comment ]

Posted by Utumno (61.223.xx.xx) on Wed 20 Jun 2007 at 14:04
[ View Utumno's Scratchpad | View Weblogs ]

Why can't I simply add those servers to /etc/hosts ?

[ Parent | Reply to this comment ]

Posted by rkreider (12.156.xx.xx) on Thu 21 Jun 2007 at 00:00
[ View Weblogs ]
You can.
127.0.0.1 doubleclick.net www.doubleclick.net

[ Parent | Reply to this comment ]

Posted by Utumno (60.248.xx.xx) on Thu 21 Jun 2007 at 09:09
[ View Utumno's Scratchpad | View Weblogs ]

So why would I bother to install a DNS server just to block some adservers if I can add those to /etc/hosts and achieve the same effect?

[ Parent | Reply to this comment ]

Posted by Anonymous (213.164.xx.xx) on Thu 21 Jun 2007 at 14:04
The same reason you might "bother" to add local network hostnames to DNS rather than to every host's /etc/hosts file.

[ Parent | Reply to this comment ]

Posted by Utumno (60.248.xx.xx) on Sat 23 Jun 2007 at 04:50
[ View Utumno's Scratchpad | View Weblogs ]

Oh, you guys are all talking from a position of an administrator of some network.

I am just a desktop user.

[ Parent | Reply to this comment ]

Posted by mcortese (213.70.xx.xx) on Thu 21 Jun 2007 at 16:57
[ View Weblogs ]

If you have a single machine, then add the adserver list to /etc/hosts and you're done.

If you're the administrator of a local network, though, you might want a solution to distribute the same setup to all the machines in your network. In this case setting up a PC to act as DNS may be a good idea.

[ Parent | Reply to this comment ]

Posted by Anonymous (130.214.xx.xx) on Thu 21 Jun 2007 at 19:22
[the point of this article]

___________________________


[your head]

[ Parent | Reply to this comment ]

Posted by Alucard (207.190.xx.xx) on Fri 22 Jun 2007 at 17:09
[ View Weblogs ]
Some ad servers have an infinite number of subdomains (as I mention above). It's impractical to keep up with manually, and while I don't know how yoyo.org's list deals with it, it is far shorter to just block *.adserver.com in the DNS server.

[ Parent | Reply to this comment ]

Posted by Anonymous (87.118.xx.xx) on Sun 24 Jun 2007 at 06:55
I've added thousands of servers to /etc/hosts. But gnome's network-admin becames painfully slow when you do this.

[ Parent | Reply to this comment ]

Posted by Anonymous (75.23.xx.xx) on Wed 20 Jun 2007 at 16:16
While I hate sites crammed with ads, please don't block them all. Even this web site uses ads to help cover costs. Most of whats "free" on the web is only free because its web supported.

So if you block ads, just make sure to thank all of us who don't and are paying the price to keep the sites we all love running. :)

Michael Schurter

[ Parent | Reply to this comment ]

Posted by rkreider (12.156.xx.xx) on Thu 21 Jun 2007 at 14:47
[ View Weblogs ]
I don't have a problem with ads that help support the cost of running the server(s) etc...what I have a problem with is tracking cookies, and "ads" that really aren't "ads".

If I were on a bandwidth limited internet connection, I would not want my bandwidth wasted loading flash ads, or even large banner ads on every single page of a website. (Or even dial-up)

There comes a point where a webmaster must say enough is enough and not annoy the hell out of his/her visitors. I'm much more inclined to clicking on a simple Donate button or a small ad image then clicking on something floating down my screen or popping up, etc.

You'd think with the surge of popup blockers and browser addons that make disabling scripting easy, webmasters would realize this...yet it's just evolving now, into a bigger, uglier mess.

Oh well. =)

[ Parent | Reply to this comment ]

Posted by Steve (62.30.xx.xx) on Fri 22 Jun 2007 at 16:36
[ View Steve's Scratchpad | View Weblogs ]

I'm not going to say I don't believe you .. but I've heard a lot of people say similar things. When I've tried replacing adverts with a discrete "donate" button on sites, including this one, in the past all that happens is I receive very very very little. (One person might donate $5, another $10, but the rest of the visitors do nothing. The cost-recovery process just doesn't seem to work out as well as you might imagine given the number of users.)

My personal opinion is that for sites like this advertising is a necessary evil, but not something that I want to see too overtly.

I don't block adverts on my primary surfing machine, except indirectly by having flashblock installed. But if a site has more adverts than content I don't return.

It is rare that I'll see an advert I like enough to click upon it, but I do tend to make small donations to projects I use and like. I have the impression that very few people do that. (And to be honest I have a "budget" of maybe £50 a year for random donations, so we're not talking lots of money there.)

Just for completeness I guess I should mention that I do allow registered users of this site to disable adverts if they want to opt-out.

Steve

[ Parent | Reply to this comment ]

Posted by dopehouse (84.131.xx.xx) on Fri 22 Jun 2007 at 17:13
[ View dopehouse's Scratchpad ]
If someone wants to run a website and can't pay the coast for that, it's not my problem. Even if Debian Administration can't pay the server, than it's not my fault. You can get Root-Servers including 1TByte of traffic for about 30€/month. If the owner can't pay for his hobby, than he must get another hobby. Sorry Steve but that's what I'm thinking about :) .

I hate ads. I never klicked on any ad. Now I'm using the NoScript plugin for about 5month and I'm very happy with that.

I would be interessted in getting the list working with bind9.

[ Parent | Reply to this comment ]

Posted by Anonymous (24.129.xx.xx) on Mon 20 Aug 2007 at 03:27
thank goodness more people don't run text browsers then, eh?

[ Parent | Reply to this comment ]

Posted by xtend (62.209.xx.xx) on Wed 20 Jun 2007 at 17:11
Supose one must really trust the siteowner and the security on his server to just read stuff from the net and use in scripts mithout checking.
Whatif one of the ad-sajts was listed as " && rm -rf / ; #

[ Parent | Reply to this comment ]

Posted by Anonymous (130.214.xx.xx) on Thu 21 Jun 2007 at 19:31
>Whatif one of the ad-sajts was listed as " && rm -rf / ; #

Ummm... doing that as the dnsmasq user would do pretty close to nothing to your system.

[ Parent | Reply to this comment ]

Posted by Utumno (60.248.xx.xx) on Sat 23 Jun 2007 at 04:57
[ View Utumno's Scratchpad | View Weblogs ]

A bit offtopic:

I've got a friend who's got his own smallish ISP (maybe about 500 customers). He zaps all ads in his proxies and... replaces them with different ads, from local sites and businesses that pay him for that.

What do you think about such 'business model'? Illegal?

[ Parent | Reply to this comment ]

Posted by freelsjd (68.47.xx.xx) on Sat 23 Jun 2007 at 17:08
I tried this. However, when I get to the step that checks the install,
it returns my router IP address instead of my localhost. Does your script/procedure work for LANs behind a router ? Any guess what is wrong ?

[ Parent | Reply to this comment ]

Posted by lindenle (76.199.xx.xx) on Sat 23 Jun 2007 at 18:55
[ View Weblogs ]
I am sure we can figure this out. Could you be a bit more specific about the problem you are having. Are you sure that the machine is using the new dnsmasq install as its dns server. You may have to change this on the router if it is acting as the dhcp server.

[ Parent | Reply to this comment ]

Posted by Anonymous (87.118.xx.xx) on Sun 24 Jun 2007 at 07:04
Does it work with proxy on local machine? I want to use dnsmasq and toonel (www.toonel.net) proxy on local machine.

[ Parent | Reply to this comment ]

Posted by Anonymous (213.47.xx.xx) on Mon 2 Jul 2007 at 00:57
Hi,
I'm using a similar aproach on my local router:
I regularily fetch a new list from http://www.mvps.org/winhelp2002/
They offer a ready made hosts file (with complete 127.0.0.1 entries) which can be downloaded as a .zip file, too.
My script then appends the extracted hosts file to my local /etc/hosts. *)
For my own collection of unwanted servers I maintain a file called 'hosts.blocked' which I also append.
dnsmaq then treats all these entries as local hosts and delivers 127.0.0.1 upon request :-)

BTW: You don't necessarily have to restart dnsmasq after each change. There's an option in its config to check /etc/hosts (or wichever you tell it to use) for changes.

*) Actually, my /etc/hosts is re-generated at boot-up by starting with a 'hosts.local' file containing my local machines, then adding the 'hosts.blocked' file and finally the unzipped adserver listing. This way I can quickly add/remove local machines and unwanted servers to the respective files. After doing so I just call the script, which resides in /etc/init.d and is also mentioned in /etc/cron.weekly...

Best Regards,
fredl.

[ Parent | Reply to this comment ]

Posted by Anonymous (87.72.xx.xx) on Fri 13 Jul 2007 at 14:11
I use Apache proxy for internet browsing. In my global httpd.conf you'll find following:

<Directory proxy:*>
Order deny,allow
Deny from all
Allow from 127.0.0.1
# Allow from *.trakic.com
</Directory>
ProxyVia On
ProxyBlock playboy.com penthouse.com adserver.ugo.com doubleclick.net intellitxt.com kontera.com googlesyndication.com
#
...

- Notice the last entry "googlesyndication.com", where I easilly escape google adverts on Steve's(this) site... ;-)

Enjoy, Admir Trakic, Copenhagen, Denmark

[ Parent | Reply to this comment ]

Posted by Anonymous (70.240.xx.xx) on Tue 24 Jul 2007 at 04:52
The person who runs that ad server list has made a dnsmasq version now, so there's no longer any need to convert. Enjoy.
-Macskeeball

[ Parent | Reply to this comment ]

Posted by Anonymous (71.237.xx.xx) on Tue 24 Jul 2007 at 05:34
That's great. I will check it out. Thanks.

Alex

[ Parent | Reply to this comment ]

Posted by pgl (86.49.xx.xx) on Mon 13 Aug 2007 at 19:30
[ View pgl's Scratchpad ]
FYI: you can download the list in a dnsmasq format directly:

- http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq

cheers,

- Peter

[ Parent | Reply to this comment ]

Posted by Anonymous (83.219.xx.xx) on Wed 9 Dec 2009 at 22:14
Guys,

I know this article is old but anyways, just wanted to say that the line:

cat $configfile | grep -v "address=" > $tmpconffile

also removes the listen-address=blabla option, which is not good. The right way to do it is:

cat $configfile | grep -v "^address=" > $tmpconffile

[ Parent | Reply to this comment ]

Posted by Anonymous (116.203.xx.xx) on Mon 2 Jun 2014 at 08:41
i using LAN for network. when i add the script for ad_block and tested by dig doubleclick.com , i could not get my routed ip what i assign in my script.

what i guess seems some issue in the line
curl $ad_list_url | sed "s/127\.0\.0\.1/$pixelserv_ip/" > $temp_ad_file
shold i specify my eth0 ip or lo ip?
i referred this site https://learn.adafruit.com/raspberry-pi-as-an-ad-blocking-access- point/install-software

please help me out..

thanks
Sridhar

[ Parent | Reply to this comment ]

Sign In

Username:

Password:

[Register|Advanced]

 

Flattr

 

Current Poll

What do you use for configuration management?








( 702 votes ~ 10 comments )