Transparent proxies via Squid.

Posted by Steve on Tue 4 Jan 2005 at 16:12

If you've setup a Linux machine as a gateway, standing between you and the internet then you have a lot of options for tweaking it. One of the most common things to setup is transparent proxying via squid.

If you've followed the previous guide on setting up a Linux gateway you will have one machine with two network interfaces, one for internal use (eth0) and one which is publically connected to the internet (eth1).

This gateway machine has a collection of firewall rules which:

  • Allow machines "behind" it to make outgoing connections.
  • Prevent incoming connections from the internet.

If we wish to setup a transparent proxy server to cache web pages - which would speed up browsing for those machines behind the gateway we need to do two things:

  • Install a caching proxy server
  • Add some rules to our gateway to seemlessly allow our machines to use it.

The first is simple. As root run:

apt-get install squid

This will install the Squid caching proxy server. This is configured by the file /etc/squid/squid.conf and we will need to make several changes to it.

First of all we need to tell it that we only wish it to listen on the internal interface. Remember that this gateway machine has two networking interfaces, one for the internal LAN and one for the internet.

Pick the one which is internal and add it to the configuration file as follows:

http_port 127.0.0.1:8080
http_port 192.168.1.1:8080

(In my case the internal address is 192.168.1.1, allowing the server to listen on the "loopback" address of 127.0.0.1 is a good idea too, and will be required later).

As well as that we'll need to tell the server what it's hostname is, and which email address is in charge of it for error display, etc:

visible_hostname gateway.my.flat
cache_mgr        proxy@foo.com

We also need to add the support for the transparent proxying we will be using:

httpd_accel_host virtual
httpd_accel_port 80
httpd_accel_with_proxy on
httpd_accel_uses_host_header on

The only remaining thing we need to do is to tell Squid which networks are allowed to connect to our proxy server, without this it will refuse all incoming requests.

For a network which is 192.168.1.x internally the following will be fine:

# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS

# Example rule allowing access from your local networks. Adapt
# to list your (internal) IP networks from where browsing should
# be allowed
acl our_networks src 192.168.1.0/24
http_access allow our_networks
http_access allow localhost

# And finally deny all other access to this proxy
http_access deny all

If you're using a different network internally then you will need to adjust the addresses appropriately.

That's all the squid setup complete, so now we restart it:

/etc/init.d/squid restart

Now we have a caching proxy server - which you is listening on 192.168.1.1:8080. If you were to enter that into your browser you should see it working - but what we are going to do next is make it transparent.

Nobody behind the gateway should need to do anything, instead it should just magically work (tm ;)

The way we do that is to add a rule to the firewal, which will redirect outgoing requests to the web (port 80) to instead go via the proxy server we've setup on the gateway machine on port 8080.

Add the following towards the end of your firewall rules:

# Transparent proxying
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 
8080

This says that anything coming from the internal interface (eth0) which has a destination port of 80 (web) should be redirected to the new squid installation we've made on port 8080.

Reload your firewall and you should have it in place.

To test it you simply need to watch the squid logfile as you browse the web from another machine.

On the gateway machine you can run:

tail -f /var/log/squid/access.log

If you see something like the following when you surf the web on a machine upon your LAN you know it worked:

1104854410.086    159 192.168.1.50 TCP_MISS/302 469 GET http://www.google.com/ -
 DIRECT/216.239.59.104 text/html
1104854410.217    128 192.168.1.50 TCP_MISS/200 1459 GET http://www.google.co.uk
/ - DIRECT/216.239.59.99 text/html
1104854410.397    180 192.168.1.50 TCP_MISS/200 9022 GET http://www.google.co.uk
/intl/en_uk/images/logo.gif - DIRECT/216.239.59.99 image/gif
1104854415.196    200 192.168.1.50 TCP_MISS/200 1459 GET http://www.google.co.uk
/ - DIRECT/216.239.59.99 text/html
1104854415.271     74 192.168.1.50 TCP_REFRESH_HIT/304 235 GET http://www.google
.co.uk/intl/en_uk/images/logo.gif - DIRECT/216.239.59.99 text/html

 

 


Posted by Serge (213.224.xx.xx) on Thu 6 Jan 2005 at 22:36
[ View Serge's Scratchpad | View Weblogs ]

Tweakinq Squid's default listen port (tcp/3128 if I recall well) is not really necessary, as you can as well transparently forward whatever port to it.

An interesting extra might be to

apt-get install dansguardian

Which is a very good filtering engine and plugs to Squid (on tcp/3128) by default, and which listens himself on tcp/8080. Then configure transparent proxy to Dansguardian's port.

--

Serge van Ginderachter

[ Parent | Reply to this comment ]

Posted by Serge (213.224.xx.xx) on Thu 6 Jan 2005 at 22:37
[ View Serge's Scratchpad | View Weblogs ]
I really need a preview and edit comment tool :-s

--

Serge van Ginderachter

[ Parent | Reply to this comment ]

Posted by Steve (82.41.xx.xx) on Thu 6 Jan 2005 at 22:43
[ View Steve's Scratchpad | View Weblogs ]

I've patched up your comment, and I'll get working on the preview code again - it used to work but I broke it and couldn't understand why.

The ability to edit comments is very definately not going to be allowed, unless it will keep a diff against the previous one - it's too easy to get into arguments where people make each other look trolly by posting inflamatory comments and then editting them later..

Steve
-- Steve.org.uk

[ Parent | Reply to this comment ]

Posted by Steve (82.41.xx.xx) on Tue 11 Jan 2005 at 18:30
[ View Steve's Scratchpad | View Weblogs ]
Comment previews are back - along with the ability to choose the type of formatting which is used.

Steve
-- Steve.org.uk

[ Parent | Reply to this comment ]

Posted by Anonymous (201.252.xx.xx) on Tue 11 Jan 2005 at 01:14
[Steve: Not wanting to spoil the superb job you are doing with these nice articles. Just for the record and for people that may get interested on a little more advanced setup once they get the stuff you describe working]. Transparent HTTP proxying means a certain degree of messing with the network stack layers (network v. application) that can lead to some strange [and hard to diagnose] problems for the users. As an example we have the problem some adminstrators faced two or three weeks ago: The hotmail.com webmail login page suddenly (most probably after a web site upgrade) stopped working for users using M$ IE behind a transparent Squid setup, they entered their username/password and just got a blank page. Some admins had to disable Squid (and therefore caching) or apply some workarounds after trial-and-error. Even Squid developers recommend strongly not to use transparent proxying. You can find posts about this stuff on the Squid users mailing list. IMVHO it would be best to advise users to implement an explicit proxy solution and if end-user application configuration is an issue then a) WPAD and b) documentation about how users could setup their browsers to use the proxy/cache can help. It may be more work now bot would save some future headaches. PS: Not to mention that (by HTTP design) transparent proxying also is incompatible with user authentication at the proxy. Regards, -- R

[ Parent | Reply to this comment ]

Posted by Steve (82.41.xx.xx) on Tue 11 Jan 2005 at 18:29
[ View Steve's Scratchpad | View Weblogs ]
That's certainly an interesting comment, and thanks for the nice praise.

I use a transparent proxy on a gateway machine for 40+ users, and have never seen any problems, but I guess this is just a case of buyer-beware.

Testing is never a bad thing ;)

Steve
-- Steve.org.uk

[ Parent | Reply to this comment ]

Posted by Anonymous (81.74.xx.xx) on Fri 11 Feb 2005 at 23:04
I have a Transparent proxy too, It has been working for about three years for 30 users. Never had any such problem.

Should I be frightened ?

bye.

[ Parent | Reply to this comment ]

Posted by Anonymous (128.109.xx.xx) on Sun 6 Sep 2009 at 09:38
that depends on how you configured your firewall.:):):)

[ Parent | Reply to this comment ]

Posted by Anonymous (80.97.xx.xx) on Mon 18 Jul 2005 at 06:35
i've managed to set up the squid server and to modify the firewall.. but what happens is that when a network computer tries to access the apache server from the machine which also runs the squid this request goes through the squid also, and i would like to avoid this.
do you know of any way of doing this?

[ Parent | Reply to this comment ]

Posted by Anonymous (208.181.xx.xx) on Thu 10 Nov 2005 at 02:35
How would you configure a transparent proxy on a bridge ?
--internet---<eth0---eth1>----internal network
assuming that you have a working transparent proxy according to the tutorial?
to clarify, using the transparent proxy and turning it into a bridge.
firewalling in not neccessary as there is a firewall, dhcp server between internet and eth0

[ Parent | Reply to this comment ]

Posted by sunu (202.83.xx.xx) on Mon 17 Apr 2006 at 08:17
hai
try this
iptable -A FORWARD -s 192.168.0.0/24 -j ACCEPT
iptable -t nat -A POSTROUTING -s 192.168.0.0/24 -j MASQUERADE
echo 1 > /proc/sys/net/ipv4/ip_forward

or

iptables -t nat -A PREROUTING -s 192.168.0.0/24 -p tcp --dport 80 -j REDIRECT --to-port 3128
echo 1 > /proc/sys/net/ipv4/ip_forward

or

iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 3128
echo 1 > /proc/sys/net/ipv4/ip_forward

assuming
httpd_accel_host virtual
httpd_accel_port 80
httpd_accel_with_proxy on
httpd_accel_uses_host_header on
is your squid.conf file entries)

sunu

[ Parent | Reply to this comment ]

Posted by Anonymous (222.154.xx.xx) on Thu 12 Jan 2006 at 08:34
Is it possible to still have transparent proxying, but the proxy server be another phsyical machine on the network?

For example, my gateway is 192.168.1.1, and my proxy server is 192.168.1.2, I want to be able to redirect requests on port 80 to 192.168.1.2:8080

[ Parent | Reply to this comment ]

Posted by GecKo (222.154.xx.xx) on Thu 12 Jan 2006 at 08:42
I got an account, and should have googled before I asked:

iptables -t nat -A PREROUTING -i eth0 -s ! squid-box -p tcp --dport 80 -j DNAT --to squid-box:3128
iptables -t nat -A POSTROUTING -o eth0 -s local-network -d squid-box -j SNAT --to iptables-box
iptables -A FORWARD -s local-network -d squid-box -i eth0 -o eth0 -p tcp --dport 3128 -j ACCEPT

From http://www.faqs.org/docs/Linux-mini/TransparentProxy.html#s6

Sorted :)

[ Parent | Reply to this comment ]

Posted by simms (216.46.xx.xx) on Thu 26 Jan 2006 at 19:22
Squid is indeed a robust and easy-to-use proxy solution. Some of its strength lies in the fact that it is expandable and customizable through the use of all kinds of "plugin" applications.
One such plugin -- and honestly I'm surprised it's gone unmentioned here -- is adzapper, a simple filtering add-on designed to match and block advertisements in all web content that passes through Squid (it apparently works with Apache, too).

Installing it on a sarge machine is as simple as invoking
apt-get install adzapper
and then adding the following line to your /etc/squid/squid.conf :
redirect_program /usr/bin/adzapper.wrapper

Once this is done, simply restart your proxy by issuing a /etc/init.d/squid restart, and voilà! You have a basic ad-blocking setup. Any website you visit (via your squid proxy of course) will now be filtered on the fly so that ad images are replaced with an unvarying placeholder, and ad text is replaced with the words This ad zapped.
Over time this will save you tons of bandwidth, not to mention time spent waiting pages to load.

[ Parent | Reply to this comment ]

Posted by Anonymous (66.82.xx.xx) on Wed 15 Feb 2006 at 21:11
Just a note great quick how too setup squid
you forgot one thing though took me hours
to figure out why it was not working. since i'm a noob
at this.
First, you must create the swap directories. Do this by running Squid with the -z option:

Squid -z

[ Parent | Reply to this comment ]

Posted by Steve (82.41.xx.xx) on Wed 15 Feb 2006 at 21:12
[ View Steve's Scratchpad | View Weblogs ]

That should happen automatically since the startup script /etc/init.d/squid will create them if they are missing ...

Steve

[ Parent | Reply to this comment ]

Posted by Anonymous (87.251.xx.xx) on Tue 21 Feb 2006 at 10:53
I suggest turn off option

forwarded_for off

if not your private and public IP is visible

--
Best regard

[ Parent | Reply to this comment ]

Posted by Hercynium (64.69.xx.xx) on Fri 16 Jun 2006 at 16:33
Here's a question I have: does this implementation work with a dial-up connection?

I've got ipmasq/dnsmasq running for my parents' business. (they're in the boonies w/NO broadband) I could use dial-on-demand, but for their convenience I've built a little perl app that allows them to browse to http://192.168.1.1:8000 and click a button to connect and disconnect. They like that level of control.

For anybody who's curious: The script gets launched from an init script like a daemon, forks like apache and then listens on the configured port for HTTP requests. When somebody clicks the connect or disconnect button, the script just issues an ifup ppp0 or ifdown ppp0 command. It's been working wonderfully for several years. (Though sometimes ifupdown freaks out over dropped connections and has to be reset - that's done through another button)


Anyhow, I'd love to use a transparent proxy to speed up their browsing and things like windows updates, but I've found nothing that's 'seamless' with dial-up.

If anybody reading this can give me some pointers, please reply! Maybe there's something I can do for you as well! :)

[ Parent | Reply to this comment ]

Posted by Steve (62.30.xx.xx) on Fri 16 Jun 2006 at 16:35
[ View Steve's Scratchpad | View Weblogs ]

Yes this will work with a dialup link, you'd just need to adjust the firewall rule to use ppp0/whatever instead of eth0.

Steve

[ Parent | Reply to this comment ]

Posted by Hercynium (64.69.xx.xx) on Fri 16 Jun 2006 at 16:44
sweet!

I'll have to try it now.

[ Parent | Reply to this comment ]

Posted by Dino (213.6.xx.xx) on Mon 26 Jun 2006 at 14:21
Hello ..

I have freeradius & mysql & pppoe-server installed
I'd install squid and it's only working when i put the proxy and port in browser
Im trying to make it work in transperant way i'd set the iptables rule and the ip forward too
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080
echo 1 > /proc/sys/net/ipv4/ip_forward

But still it's not working in transperant way.. here is my route -n settings
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.67.15.30 0.0.0.0 255.255.255.255 UH 0 0 0 ppp1
10.67.15.31 0.0.0.0 255.255.255.255 UH 0 0 0 ppp2
10.67.15.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 eth1

eth1 is connecting to the router
eth0 is for the network

when i open http://ipaddress:8080/ it works but when i browse it didn't show me the browsing proccess in the access.log

waiting ur response

Dino

[ Parent | Reply to this comment ]

Posted by yousuf87 (202.83.xx.xx) on Sun 17 Sep 2006 at 02:13
Gateway: 192.168.1.1 - outside my administrative privileges
Proxy1: 192.168.100.13:8080 - outside my administrative privileges

Station1: 192.168.1.30 - connected to the Internet via Proxy1
Station2: 192.168.1.31 - not allowed to use Proxy1

I want to configure Station1 to act as a proxy server through which Station2 can get access to the Internet. Something like Proxy1 -> Station1 (intermediate proxy) -> Station2.

Two problems:

1 - Looks like the httpd_accel_* TAGS aren't recognized when squid.conf is parsed:

ltsp-server:~# /etc/init.d/squid restart
Restarting Squid HTTP proxy: squid Waiting.................................done.
2006/09/17 05:42:13| parseConfigFile: line 2982 unrecognized: 'httpd_accel_host virtual'
2006/09/17 05:42:13| parseConfigFile: line 2983 unrecognized: 'httpd_accel_port 80'
2006/09/17 05:42:13| parseConfigFile: line 2984 unrecognized: 'httpd_accel_with_proxy on'
2006/09/17 05:42:13| parseConfigFile: line 2985 unrecognized: 'httpd_accel_uses_host_header on'
.
ltsp-server:~# 

2 - Ignoring these errors, squid starts on the server (Station1). The client (Station2) connects to Station1:8080, sends an HTTP request, and the proxy server returns "ERROR 503: Service Unavailable":

ultra1:~# wget http://www.google.com
--06:02:09--  http://www.google.com/
           => `index.html'
Connecting to 192.168.1.30:8080... connected.
Proxy request sent, awaiting response... 503 Service Unavailable
06:03:45 ERROR 503: Service Unavailable.

ultra1:~#

This is logged on the proxy server machine too:

ltsp-server:~# tail -f /var/log/squid/access.log
1158452477.733  49432 192.168.1.31 TCP_MISS/503 1542 GET http://www.google.com/ - DIRECT/www.google.com text/html
1158452477.733 155009 192.168.1.31 TCP_MISS/503 1542 GET http://www.google.com/ - DIRECT/www.google.com text/html
1158452477.733  34808 192.168.1.31 TCP_MISS/503 1542 GET http://www.google.com/ - DIRECT/www.google.com text/html
1158452707.379 155012 192.168.1.31 TCP_MISS/503 1542 GET http://www.google.com/ - DIRECT/www.google.com text/html

ltsp-server:~#



What am I doing wrong? Or is such a setup not possible?

[ Parent | Reply to this comment ]

Posted by trailmug (68.58.xx.xx) on Wed 27 Sep 2006 at 18:30
The accel directives are deprecated for transparent proxy with Squid 2.6. See http://mirrors.dotsrc.org/squid/squid-2/STABLE/squid-2.6.STABLE1- RELEASENOTES.html#s2 .. grab a new squid.conf.default and go from there.

Basically, the accel directives go away, and http_port becomes
http_port <port> transparent

[ Parent | Reply to this comment ]

Posted by yousuf87 (202.83.xx.xx) on Thu 28 Sep 2006 at 19:26
Problem 2 solved.

I need to add a 'parent' 'cache_peer':
cache_peer 192.168.100.13         parent    8080  0     proxy-only no-query 


Additionally, I need to forward all cache requests to my 'parent':
never_direct allow all


Though it's not always necessary, I should also specify which pages never to cache:
acl QUERY urlpath_regex cgi-bin \? asp jsp php 
cache deny QUERY


Works beautifully for me. Hope this helps.

[ Parent | Reply to this comment ]

Posted by Anonymous (211.68.xx.xx) on Fri 10 Nov 2006 at 02:43
Hello yousuf87,would you plz contact to me? xmwang2005@lzu.cn
I have the same problems with you.

[ Parent | Reply to this comment ]

Posted by Anonymous (213.42.xx.xx) on Fri 12 Jan 2007 at 09:24
I have squid in my gentoo server and the proxy working well, and the output in the access.log is the same as the mentioned before. But Why the TCP is always MISS and how would i know that my cache is working or not >>?

[ Parent | Reply to this comment ]

Posted by Anonymous (84.45.xx.xx) on Sun 21 Jan 2007 at 10:36
Steve,

Another great article, that got me up and running in a few minutes. When I googled for those magic words, squid, transparent, proxy, iptables, debian I kinda knew your site would be near the top. Glad I hit this first, it Just Works (tm).

Cheers,
popey

[ Parent | Reply to this comment ]

Posted by Anonymous (195.78.xx.xx) on Tue 30 Jan 2007 at 06:08
I'm having trouble with access.log. After it reaches a certain size, access.log is automatically renamed to access.log.0 and a new access.log is created. The problem is that squid won't run unless I reset the owner of access.log to 'proxy' (the default owner of the new access.log is always 'root').

How do I set the default owner to 'proxy' for the new access.log, or how do I get squid to accept an access.log owned by 'root'?

Regards,
Yousuf

[ Parent | Reply to this comment ]

Posted by yousuf87 (195.78.xx.xx) on Tue 30 Jan 2007 at 06:12
I'm having trouble with access.log. After it reaches a certain size, access.log is automatically renamed to access.log.0 and a new access.log is created. The problem is that squid won't run unless I reset the owner of access.log to 'proxy' (the default owner of the new access.log is always 'root').

How do I set the default owner to 'proxy' for the new access.log, or how do I get squid to accept an access.log owned by 'root'?

Regards,
Yousuf

"The Network IS The Computer"

[ Parent | Reply to this comment ]

Posted by Steve (62.30.xx.xx) on Tue 30 Jan 2007 at 09:33
[ View Steve's Scratchpad | View Weblogs ]

Take a look at /etc/logrotate.d/squid - that should be the file that does the rotation.

Steve

[ Parent | Reply to this comment ]

Posted by Anonymous (86.87.xx.xx) on Thu 22 Mar 2007 at 11:37
Is this also possible for Squid 3?

The following directives are not available any more in the squid.conf file in the third version of squid

httpd_accel_host virtual
httpd_accel_port 80
httpd_accel_with_proxy on
httpd_accel_uses_host_header on

[ Parent | Reply to this comment ]

Posted by Anonymous (220.233.xx.xx) on Mon 9 Apr 2007 at 14:59
Apparently they changed things around to keep us guessing (and worrying as we do our late night upgrades).

New way to enable transparent proxying is to make your http_port line read similar to this:
http_port 192.168.1.1:8080 transparent
Note the word "transparent at the end... Seems this is the only part required. I have mine simply read "http_port 3128 transparent".

[ Parent | Reply to this comment ]

Posted by Anonymous (62.1.xx.xx) on Wed 1 Aug 2007 at 19:15
One word (transparent) made me lose lots of minutes until i found that post :p
Thanks :)

[ Parent | Reply to this comment ]

Posted by Anonymous (58.69.xx.xx) on Tue 4 Sep 2007 at 18:36

QUESTION: I tried this stuffs on my proxy server. But why am i getting the following errors:

parseConfigFile: line 131 unrecognized: 'iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080'
parseConfigFile: line 132 unrecognized: 'iptables -A PREROUTING -t nat -i eth1 -p tcp --dport 80 -j DNAT --to 192.168.2.10:80'
parseConfigFile: line 133 unrecognized: 'iptables -A FORWARD -p tcp -m state --state NEW -d 192.168.2.10 --dport 80 -j ACCEPT'

I hope someone out there could help me. Thanks!

[ Parent | Reply to this comment ]

Posted by Steve (82.32.xx.xx) on Tue 4 Sep 2007 at 19:51
[ View Steve's Scratchpad | View Weblogs ]

These are lines to be added to the end of your firewall - or entered interactively. They should not be added to the squid configuration file.

Steve

[ Parent | Reply to this comment ]

Posted by Anonymous (81.210.xx.xx) on Sun 4 Jan 2009 at 23:34
Oh man, you saved my butt with this command:
# Transparent proxying
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port
8080

Been searching looong for a solution for transparently filtering content on LAN hosts.
THANK YOU VERY MUCH !

[ Parent | Reply to this comment ]

Posted by Anonymous (137.164.xx.xx) on Tue 19 May 2009 at 17:21
What if the school blocks downloads of any kind? then what and im not sure what else to do :(

[ Parent | Reply to this comment ]

Posted by Anonymous (196.0.xx.xx) on Fri 3 Jun 2011 at 12:53
hi, can anyone help out with a written configured transparent script of squid version 3, the whole of it written down to easy the job because when it divided in bits, it confuses, please. thank you. if you can you may help me send it to vascojoseph66@yahoo.com

[ Parent | Reply to this comment ]

Sign In

Username:

Password:

[Register|Advanced]

 

Flattr

 

Current Poll

What do you use for configuration management?








( 676 votes ~ 10 comments )